EMBEDDED ORDER DATA LOADS

Each week the Acquisitions Department sends ISP e-mail notifications and concatenated MARC record files from vendors that contain EOD information. These enhanced Marc records contain 9xx fields with price and Yale fund code information that make it possible to create P.O. records along with bibs and mfhds. The e-mail notifications contain the file names and number of records in each file.

After DPA records the file name and record counts in the log book, the files are uploaded to deleon in order to run the files through Prebulk to strip fields from records and verify record counts. We do not create mfhds via Prebulk for EOD records. It is not possible to load an interleaved file of bibs and mfhds when creating purchase orders. Instead mfhds are created upon Bulk Import from information contained in the 9xx fields and defining a call number hierarchy in the Bulk Import Rules.

We use four different configuration files for running Prebulk for EOD records, EOD.cfg, BWUSeod.cfg, CASAeod.cfg, and IBEReod.cfg. The BWUSeod.cfg is used solely for the Blackwell's BNA (US) records because we strip less fields from these records then the other vendor's records. The CASAeod.cfg strips out specific 9XX tags that may conflict with holdings creation. Similarly, the IBEReod.cfg file strips specific tags for its records. The EOD.cfg file is used with Prebulk for all other EOD loads.

The Prebulk scripts are as follows:

From deleon: /m1/voyager/yaledb/sbin:

$ Pprebulk -i /export/home/mgarman/EOD/(Blackwell's US file name) -o ../local/EOD/(Blackwell's US file name.pre) -c ../local/SCRIPTS/BWUSeod.cfg

$ Pprebulk -i /export/home/mgarman/EOD/(Casalini's file name) -o ../local/EOD/(Casalini's file name.pre) -c ../local/SCRIPTS/CASAeod.cfg

$ Pprebulk -i /export/home/mgarman/EOD/(Iberbook's file name) -o ../local/EOD/(Iberbook's file name.pre) -c ../local/SCRIPTS/IBEReod.cfg

$ Pprebulk -i /export/home/mgarman/EOD/(file name) -o ../local/EOD/(file name).pre -c ../local/SCRIPTS/EOD.cfg

Once these files have been run through Prebulk, DPA sends an e-mail to ITS letting them know the records have been preprocessed and are ready to be copied to degama and loaded using the following Bulk Import script:

From degama: /m1/voyager/yaledb/sbin:

$ Pbulkimport -f /m1/voyager/yaledb/local/EOD/(file name) -i EODBWUS -o EODBWUS

$ Pbulkimport -f /m1/voyager/yaledb/local/EOD/(file name) -i EODBWUK -o EODBWUK

$ Pbulkimport -f /m1/voyager/yaledb/local/EOD/(file name) -i EODTZT -o EODTZT

$ Pbulkimport -f /m1/voyager/yaledb/local/EOD/(file name) -i EODHAR -o EODHARR

$ Pbulkimport -f /m1/voyager/yaledb/local/EOD/(file name) -i EODIBER -o EODIBER

$ Pbulkimport -f /m1/voyager/yaledb/local/EOD/(file name) -i EODCASA -o EODCASA

Notice that each file loaded has a separate Bulk Import Rule (flag -i). This is because each file must be mapped to a specific vendor code to create purchase orders. We can only concatenate files if they are from the same vendor and fall under the same vendor code.

Once the files have been loaded, ITS will notify DPA that the jobs have completed and will include the names of the log files that contain the output stats. DPA then checks the stats for each load to see that the records have loaded as expected and checks some records via the ProdOrbis cataloging module to assure the records have loaded correctly.

Although a Duplication Dectection profile is used during Bulk Import, these records are loaded to production unconditionally. Therefore, even if a duplicate record is detected in the database, the incoming record is loaded anyway. The ouput log stats record how many records were detected as "replaces" (even though they were not replaced) and gives the bib id for the record that would have been replaced as well as the bib id for the record that was loaded unconditionally. DPA runs an AWK program, VENDORREPLACE, against the log file to pull out these ids to send to Acquisitions to resolve.

If the stats indicate that some line items were rejected during a load, it ususally means that the vendor codes were invalid. DPA runs another AWK program, NOTVALIDFUND, against the log file which will pull out all the bib ids for records with invalid funds. These are sent to Acquisitions along with the stats and list of dups for each load.

Return to Database Loads Page