SERIALS SOLUTION LOADS

SS Adds Configuration File | SS Duplicate Detection Profiles | SS Bulk Import Rules

NEW RECORDS / UPDATED RECORDS / DELETES

We receive full MARC records, enhanced with specific Yale information, from Serials Solutions on a bi-monthly basis. These records are first sent through MARS processing for authority control (see Handling below). Once we receive the records back from MARS, we are ready to load them to production. These records constitute three different types of loads, records added as new, records to replace previously loaded records, and records that need to be deleted. We handle these three different record loads in the following manner:

Handling:

We will receive notification from Serials Solutions when update files are ready to be picked up from their server. The Authority Control Librarian will then notify MARS that our files are ready to be picked up from the SS server for MARS to process for authority control. Once we receive notification from MARS that the records are have been processed are ready to be picked up from their server, The Authority Control Librarian will pick them up from the OCLC (MARS) server and put them in the SS folder in the eresources directory on our Clark server. At this point the records will be ready to be preprocessed and loaded to Production.

New records:

We create mfhds for the new records via Prebulk and then add them to our database unconditionally using the 035 as a match point. Even if a match is found for an incoming record, it will not be overlaid, the record will simply be added to the database as a duplicate record. There should not be any of these among the SS new records. However, after the load is complete, we will run an AWK program against the output stats to see if any matches had been found.

In order to create an interleaved file of bibs and mfhds for loading, we run a Prebulk script from clark: /m1/voyager/yaledb/sbin as follows:

$ Pprebulk -i /m1/bulk/MARS-AUTHS/d031117.ss.new.bib -o ../local/SS/ssnew.d031117.pre -c ../local/SCRIPTS/ssadd.cfg.

• The -i flag signifies where the records are stored before Prebulk.
• The -o flag signifies where the records will be stored after Prebulk (add the .pre extension to avoid confustion)
• The -c flag signifies where to find the correct configuration file for processing the Serials Solutions adds.

After Prebulk, the records are run through the addmfhdindicators.jar program. This is a Java program developed by our Programming Group to add a second indicator to the mfhd. To run this program, the AddMFHDIndicators.ini file found on clark:/m1/voyager/yaledb/local/MARChandlers must be changed to reflect the input path and output path for the file of records we want changed.

Input_File=/m1/voyager/yaledb/local/SS/ssnew.d031117.pre Output_File=/m1/voyager/yaledb/local/SS/ssnew.d031117.mfhdind

The program is then run from the same directory on the server (m1/voyager/yaledb/local/MARChandlers) with the following command: runaddmfhdind

Once the preprocessing is done, the files are ready to be loaded to production via Endeavor's Bulk Import program. DPA sends an e-mail to ITS letting them know that the files have been preprocessed and are ready to be copied to degama and loaded using the following Bulk Import script:

From degama: /m1/voyager/yaledb/sbin:

$ Pbulkimport -f /m1/voyager/yaledb/local/SS/ssnew.d031117.mfhdind -i ERADD -o ERADD -m

After the files are loaded to production, ITS will notify DPA that the job has run and will include the names of the log files that contain the job output. DPA then checks the output stats for the load to see that it has loaded as expected and checks some records via the ProdOrbis cataloging module to assure the records have loaded correctly. DPA also runs the following AWK program against the output log to see if any incoming records found a match in the database:

from degama:/m1/voyager/yaledb/rpt:awk -f /m1/voyager/yaledb/local/AWK/VENDORREPLACE log.imp.20031120.0940(file name)

This program will display both the bib id of the incoming record and the bib id of the database record that was found to match on the Duplicate Detection match point (035). If any of these are found, send both bib ids to the Electronic Records cataloger for clean-up.

Updated records:

Updated records do not need to have mfhds created; we simply want to overlay bib records already in the database. We do run the file through Prebulk, however, in order to strip the 945 tags. The Prebulk script is as follows:

From clark:/m1/voyager/yaledb/sbin:

$Pprebulk -i /m1/bulk/MARS-AUTH/d031117.ss.chg.bib -o ../local/SS/sschg.d031117.pre -c ../local/SCRIPTS/ssupdate.cfg

Once the records have been run through Prebulk, DPA sens an e-mail to ITS letting them know the records have been preprocessed and are ready to be copied to degama and loaded to production using the following Bulk Import script:

From degama: /m1/voyager/yaledb/sbin:

$Pbulkimport -f /m1/voyager/yaledb/local/SS/sschg.d031117.pre -i ERUPDT -o ERUPDT

All of these records should load as "replaces." This should be reflected in the output stats. If the stats show that some records were "added" instead of replaced, run the following grep statement against the output log file:

From degama:/m1/voyager/yaledb/rpt:

$fgrep 'Adding Bib' log.imp.20031120.1030(log name)

This will pull out a list of bib ids from the log file for records that were added as new. These bib ids should be sent to the electronic records librarian for resolution.

Deleted records:

The files of deletes we have received from Serials Solutions thus far is small so we send it to the Electronic Records cataloger to take care of them manually.

Once DPA is satisfied that both the new and changes records have loaded correctly, the output stats are sent to the Electronic Resources librarian along with the file of records to be deleted..

Return to Database Loads Page