System Failure

Databases & Article searching is the most heavily used link on the Library Front Door. As such, that means we should invoke the failover whenever MetaLib on exlib1 is unavailable by modifying the Front Door page, redirecting the alias for extended downtime, and communicating broadly about the failure.

In the event of a failure:

• Inform ILTS staff of off hours. Call until someone is reached in this order: Melissa Wisner, Audrey Novak, Este Pope, Kalee Sprague, Karen Kupiec who will then contact additional staff as needed.
• Post to YULIB to report the failure; to report the invocation of failover; and to report an alias change. Also contact the Medical library directly (Call in this order: Hongbin Liu, Lei Wang, Mark Gentry), contact Katie Bauer and send a copy of YULIB message to smlref@yale.edu. Include the following:
  • Indicate that Databases & Article searching (MetaLib) is down, we are working on it and will update the Front Door to point to the failover system within 30 minutes. Make clear that access to MetaLib from the Library Front Door will be restored, but deeplinks and MetaLib feeds to other webpages will not work. Staff should not use the Management module to modify MetaLib IRD records.
    • sample email contents for MetaLib to YULIB: <MetaLib is down (or describe nature of problem, what features aren't working) as of (date/time) and members of the ILTS/ITS Library group are currently investigating.  We will give a report back in about 30 minutes.Make note of any specifics that are still working.>
    • sample email contents for FAILOVER to YULIB: <We are still troubleshooting the problem with our production MetaLib server, and so we have invoked our failover server, sfx1, temporarily until the problem is resolved.  This means that the link to MetaLib from the Library Front Door is now working as expected, but that any links from other webpages to MetaLib will not work.  Also, please do not work in the MetaLib management module until we have resolved this problem.  We will send an update once we have more information.>
  • If the problem is not resolved within 3 hours or we know it will last more than 3 hours ask DNO (ITS Data Network Services) to re-direct the alias. Inform staff through YULIB and let them know that when the alias is redirected, the Databases & Article searching page, deeplinks and MetaLib feeds will work. Staff should NOT modify MetaLib IRD records in the Management module.
    • sample email contents for DNO alias change to YULIB: <We are still troubleshooting the problem with our production MetaLib server and will now change the MetaLib alias to point to our failover server.  This means that all links from other webpages to MetaLib will work once the alias change is complete.  We will notify you when the alias change is complete, and will then send an update once the problem is resolved.>
• Manually change the redirect in sfx1 (Failover server) to make sure sfx1 doesn't redirect to exlib1. The file is /exlibris/metalib/[version]/ins01/www_v_eng/home. The refresh line in the header is causing the problem:
  <meta http-equiv="refresh" content="0; url= http://exlib1.library.yale.edu:8331/V?func=find-db-1">. Update this to point to sfx1 instead of exlib1.
• Invoke the failover system as indicated below.

Invoking Failover

Change the Databases & Article Searching link from the Library Front Door (home page) to the failover server. Notify karen.kupiec@yale.edu, kathleen.bauer@yale.edu and melissa.wisner@yale.edu about the change. During normal business hours contact Karen Kupiec, Ernie Marinko, or George Ouellette in WWWDCS to make this change and notify Katie Bauer and Melissa Wisner. Instructions follow.

• The Library's homepage is \usr\web\htmldocs\yulhome.htm on dagda.library.yale.edu
• Modifying the Page:
  • 1. E-mail karen.kupiec@yale.edu, kathleen.bauer@yale.edu, prodsys@mailman.yale.edu
  • 2. Create a copy of the page in the same directory - cp yulhome.htm yulhomeDATE.htm . (DATE = date of modification)
  • 3. vi yulhome.htm
  • 4. Locate the text 'Articles' .
  • 5. Change the link for MetaLib to: http://sfx1.library.yale.edu:8331/V
  • 6. Save the page
  • 7. Test the link and check the page for correct display!.
  • Note: Do not FTP the page off of the web server to make this change - you risk the display being modified - use VI on the server.

Redirect the alias
If the downtime is expect to last for 3 hours or more, contact DNO to have the alias databases.library.yale.edu redirected to sfx1.library.yale.edu. Note that the Library's homepage should be updated to point back to databases.library.yale.edu rather than to sfx1.library.yale.edu

Tomcat application failures:
If there are problems either with either the PDS/CAS login (MetaLibRemoteLogin) or the MLAccess scripts (used in the Front Door's "Search this Site" combined search), ITS/LSG staff will troubleshoot the problem to determine whether it is at the server, Tomcat or application level and will resolve the problem or contact ILTS staff as necessary.

• Post to YULIB informing staff that login in MetaLib or the MetaLib feeds to other webpages (MLAccess functions) are not working and the problem is being addressed.

Sample Searches:
Sample searches to use to check that MetaLib is working.
URL: http://databases.library.yale.edu:8331/V

• Select Find Database. Search by title for Oxford. You should retreive about 7 titles
• Select QuickSearch. Click Books. Enter Didion, Joan as an author. You should see each database return Found and Retrieved record counts or give an error message.

Sample searches to use to check that MetaLib-related TomCat applications are working follow. In the event of an unresolved failure, please contact Melissa Wisner (or Audrey Novak) and Karen Kupiec.

• To test that the Front Door combined search is working, go to
  http://www.library.yale.edu/htmldocs/search.html
  • Search "Film". You should see results listed under Article Databases. Select details. You should see a more complete record.
• To test the TomCat apps directly invoke these URL's:
  • http://columbus.library.yale.edu:8085/MLAccess/ML_Search_Locate.jsp?wrd=film (Executes a keyword search for film and returns approximately 25 XML encoded full records.
  • http://columbus.library.yale.edu:8085/MLAccess/ML_Search_Locate.jsp?wrd=film&mode=medium (Same search as above, but fewer fields. Includes title, desc, url and status.)
  • http://columbus.library.yale.edu:8085/MLAccess/ML_Search_Locate.jsp?wrd=film&mode=brief (Same search as above, but even fewer fields returned. Includes title, url and status.
  • http://columbus.library.yale.edu:8085/MLAccess/ML_Retrieve_By_Category.jsp?CatID=31 (Same output as MLAccess/ML_Search_Locate, but by category id)
  • http://columbus.library.yale.edu:8085/MLAccess/ML_Retrieve_One_Record.jsp?RecNum=YUL03663 (Retrieve one record for 20th Century English poetry).
  • http://columbus.library.yale.edu:8085/MLAccess/ML_Retrieve_By_Category.jsp?CatID=31&mode=brief (Retrieves all records in a category. Records include same data elements as mode=brief)

Testing PDS/CAS authentication:

URL: http://databases.library.yale.edu:8331/V Login in using your netid and password. In the event of an unresolved failure, please contact Melissa Wisner (or Audrey Novak).

Server Admin Document

Sync cron (nightly) jobs:

1, when exlib1 is production and sfx1 is failover (crons are active as exlib1 is in production now)

• On exlib 1:
  • # exporting VIR00 library
    • 01 06 ***/exlibris/metalib/local_bin/exlib1_export_vir00.csh
  • # MPSync : exporting DAT01
    • 30 06 * * * /exlibris/MetaLib/local_bin/exlib1_mpsync_export.csh
• On sfx1:
  • # sync user interface HTML or ins01 from exlib1
    • 10 06 * * * rsync --stats -auvz -e ssh m_ftp@exlib1:/exlibris/MetaLib/m3_1/ins01/ /exlibris/MetaLib/m3_1/ins01
    • 15 06 * * * rsync --stats -auvz -e ssh m_ftp@exlib1:/exlibris/MetaLib/m3_1/dat01/www_m_eng/ /exlibris/MetaLib/m3_1/dat01/www_m_eng
  • # importing VIR00 library from sfx1
    • 10 07 * * * /exlibris/MetaLib/local_bin/sfx1_import_vir00.csh
  • # MPSync importing export of sfx1
    • 31 07 * * * /exlibris/MetaLib/local_bin/sfx1_mpsync_import.csh

2, when sfx1 is production and exlib1 is failover (crons are NOT active as sfx1 is not in production now, to activate simply remove # from the command lines)

• On exlib1:
  • # MPSync importing export of sfx1
    • #01 08 * * * /exlibris/MetaLib/local_bin/mpsync_import.csh
  • #importing VIR00 library from sfx1
    • #10 07 * * * /exlibris/MetaLib/local_bin/import_vir00.csh
• On sfx1:
  • # sync user interface HTML or ins01 to exlib1
    • #10 06 * * * rsync --stats -auvz -e ssh /exlibris/MetaLib/m3_1/ins01/ m_ftp@exlib1://exlibris/MetaLib/m3_1/ins01
  • # MPSync : exporting DAT01
    • #30 06 * * * /exlibris/MetaLib/local_bin/mpsync_export.csh
  • # exporting VIR00 library
    • #01 06 * * * /exlibris/MetaLib/local_bin/export_vir00.csh

Logs:
/exlibris/MetaLib/m3_1/apache/logs/
access_log
error_log

Start and stop scripts:
1, stop
/etc/init.d/httpd_m3_1 stop
/etc/init.d/start_MetaLib_m3_1 stop

2, start
/etc/init.d/httpd_m3_1 start
/etc/init.d/start_MetaLib_m3_1 start

Ex Libris Contact Information

• Submit an incident report: http://support.exlibrisgroup.com/
• Boston office contact information: http://www.exlibrisgroup.com/offices.htm

© 2007 Yale University Library
This file last modified Monday, 12:00 PM 4/30/2007
Send comments to Integrated Systems and Programming