Updated by MG July 15, 2004
|
Integrated Library Technology Services Web, Workstation & Digital Consulting Services Database Loads Schedule |
ITPRO Loads SS Adds Configuration File | SS Duplicate Detection Profiles | SS Bulk Import Rules NEW RECORDS / UPDATED RECORDS / DELETES
Complete files of ITPro records are available at the end of quarter. We initially loaded the file of complete records as of 12-31-03 (2,842 records). Twice each month, ITPro makes available updates (records added). We pick up these records and concatenate them for three months. At the end of three months, we send this file of new records to MARS for processing. Once the file is returned we add these new records to our database using the following process. Twice a month the Catalog Librarian for Networked Information Resources ftp's to the ITPro server and retrieves records that were added as new between the 1st and 15th of the month and between the 16 through the end of the month. He loads the records onto the Clark server into the /export/home/eresource/24by7 directory. He concatenates three months of these records. At the end of the quarter, the Authority Control Librarian ftp's the records to MARS for authority control. MARS will send an e-mail notification once the records have been processed and our ITS department will pick them up and place them on the Deleon server in the MARS-AUTH/ER directory. when update files are ready to be picked up from their server. At this point the records are ready to be preprocessed by DPA and loaded to production. Once we receive the quarterly file back from MARS processing, DPA downloads it to a PC to edit the file via the MarcEdit program. The following edits are required for the ITPro records: 1) Add 945 tag - $byulint $zOnline Resource 2) Delete tag 856 |z 3) Add tag 856 |zOnline book 4)
Add tag 506 \\$aAccess restricted by licensing agreement. 5)
Add tag 590 \\$aAccess is available to the Yale Community through use
of a networked computer with a Yale IP address. We create mfhds for the new records via Prebulk and then add them to our database unconditionally using the 035 as a match point. Even if a match is found for an incoming record, it will not be overlaid, the record will simply be added to the database as a duplicate record. There should not be any of these among the SS new records. However, after the load is complete, we will run an AWK program against the output stats to see if any matches had been found. In order to create an interleaved file of bibs and mfhds for loading, we run a Prebulk script from clark: /m1/voyager/yaledb/sbin as follows: $ Pprebulk -i /m1/bulk/MARS-AUTHS/d031117.ss.new.bib -o ../local/SS/ssnew.d031117.pre -c ../local/SCRIPTS/ssadd.cfg.
After Prebulk, the records are run through the addmfhdindicators.jar program. This is a Java program developed by our Programming Group to add a second indicator to the mfhd. To run this program, the AddMFHDIndicators.ini file found on clark:/m1/voyager/yaledb/local/MARChandlers must be changed to reflect the input path and output path for the file of records we want changed. Input_File=/m1/voyager/yaledb/local/SS/ssnew.d031117.pre Output_File=/m1/voyager/yaledb/local/SS/ssnew.d031117.mfhdind The program is then run from the same directory on the server (m1/voyager/yaledb/local/MARChandlers) with the following command: runaddmfhdind Once the preprocessing is done, the files are ready to be loaded to production via Endeavor's Bulk Import program. DPA sends an e-mail to ITS letting them know that the files have been preprocessed and are ready to be copied to degama and loaded using the following Bulk Import script: From degama: /m1/voyager/yaledb/sbin: $ Pbulkimport -f /m1/voyager/yaledb/local/SS/ssnew.d031117.mfhdind -i ERADD -o ERADD -m After the files are loaded to production, ITS will notify DPA that the job has run and will include the names of the log files that contain the job output. DPA then checks the output stats for the load to see that it has loaded as expected and checks some records via the ProdOrbis cataloging module to assure the records have loaded correctly. DPA also runs the following AWK program against the output log to see if any incoming records found a match in the database: from degama:/m1/voyager/yaledb/rpt:awk -f /m1/voyager/yaledb/local/AWK/VENDORREPLACE log.imp.20031120.0940(file name) This program will display both the bib id of the incoming record and the bib id of the database record that was found to match on the Duplicate Detection match point (035). If any of these are found, send both bib ids to the Electronic Records cataloger for clean-up. Updated records do not need to have mfhds created; we simply want to overlay bib records already in the database. We do run the file through Prebulk, however, in order to strip the 945 tags. The Prebulk script is as follows: From clark:/m1/voyager/yaledb/sbin: $Pprebulk -i /m1/bulk/MARS-AUTH/d031117.ss.chg.bib -o ../local/SS/sschg.d031117.pre -c ../local/SCRIPTS/ssupdate.cfg Once the records have been run through Prebulk, DPA sens an e-mail to ITS letting them know the records have been preprocessed and are ready to be copied to degama and loaded to production using the following Bulk Import script: From degama: /m1/voyager/yaledb/sbin: $Pbulkimport -f /m1/voyager/yaledb/local/SS/sschg.d031117.pre -i ERUPDT -o ERUPDT All of these records should load as "replaces." This should be reflected in the output stats. If the stats show that some records were "added" instead of replaced, run the following grep statement against the output log file: From degama:/m1/voyager/yaledb/rpt: $fgrep 'Adding Bib' log.imp.20031120.1030(log name) This will pull out a list of bib ids from the log file for records that were added as new. These bib ids should be sent to the electronic records librarian for resolution. The files of deletes we have received from Serials Solutions thus far is small so we send it to the Electronic Records cataloger to take care of them manually. Once DPA is satisfied that both the new and changes records have loaded correctly, the output stats are sent to the Electronic Resources librarian along with the file of records to be deleted.
|