GPO Changes Load

GPO CHANGES LOADS

GPO Configuration File | GPO Duplication Detection Profile | GPO Bulk Import Rules

GPO CHANGES ONLINE / GPO CHANGES USGS / GPO CHANGES OTHER
/ GPO CHANGES ONLINE NO LONGER AVAILABLE

Once a month we receive an e-mail notification from MARCIVE letting us know that a set of GPO Changes records is ready for us to pick up from their server. We receive four separate files of GPO Changes records, GPO CHANGES ONLINE, GPO CHANGES USGS, and GPO CHANGES OTHER (everything that doesn't fall into one of the other three categories), and GPO CHANGES ONLINE NO LONGER AVAILABLE. We requested that these records be broken out into separate files because of the manner in which we want to load them. Below is the documentation for how we handle each of the four files.

GPO CHANGES ONLINE

GPO Changes Online records are loaded as an interleaved bib and mfhd file, using a replace Duplicate Detection profile, and a Bulk Import rule of load bib/auth only. We want these records to overlay records already in our database and add a mfhd for the yulint location with a public note of Online Resource. In order to do this, the file must be run through four preprocessing programs.

The first program is a Java program written by one of our programmers to replace the prefix tmp which precedes the source's system id number in the 035 |a field with a (GPO)tmp prefix.

The Java program is stored on clark: /m1/voyager/yaledb/local/GPO/tagchangernew.jar. The script for running this program is as follows:

From clark:/m1/voyager/yaledb/local/GPO:

$ java -jar tagchangernew.jar (input) (output)

(input) - where the input file resides, typically, /m1/bulk/GPO/YACCO039

(output) - where the output should be stored - store in the local/GPO/CHANGES directory as chgsonline039.tag

After the tagchangernew.jar program is run, the file should be downloaded to a local PC in order to use MarcBreaker to add a 945 tag for holdings information as follows:

Add Tag: 945 $byulint$zOnline Resource

After the tag has been added via MarcBreaker, the file must be run through MarcMaker to reformat it into a Marc file suitable for loading. Once the file is in the correct format, upload it to your home directory on clark.

The next preprocessing program is Prebulk. Prebulk will use a gpoonline.cfg file to point to the 945 tag for holdings information. Here is the script for running Prebulk:

From clark: /m1/voyager/yaledb/sbin:

$Pprebulk -i /export/home/mgarman/chgsonline039marc -o ../local/GPO/CHANGES/chgsonline039.pre -c ../local/SCRIPTS/gpoonline.cfg

The last preprocessing program needed for the GPO Changes Online load is the addmfhdindicators.jar program created by one of our programmers to add a second indicator to the mfhd. To run this program, the AddMFHDIndicators.ini file found in clark:/m1/voyager/yaledb/local/GPO must be changed to reflect the input path and output path for the file of records we want changed.

Input_File=/m1/voyager/yaledb/local/GPO/CHANGES/chgsonline039.pre Output_File=/m1/voyager/yaledb/local/GPO/CHANGES/chgsonline039.mfhdind

The program is then run from the same directory on the server (m1/voyager/yaledb/local/GPO) with the following command: runaddmfhdind

Once the preprocessing is done, the files are ready to be loaded to production via Endeavor's Bulk Import program. DPA sends an e-mail to ITS letting them know that the files have been preprocessed and are ready to be copied to degama and loaded using the following Bulk Import script:

From degama: /m1/voyager/yaledb/sbin:

$ Pbulkimport -f ../local/GPO/CHANGES/chgsonline039.mfhdind -i GPOUPDT -o CHGSONLN039 -m

The -f flag signifies where the preprocessed records are stored.
The -i flag signifies which Bulk Import Rule will be used.
The -o flag signifies the name of the load under which the record was added. This will be the "operator" found in the bibliographic record history tab when viewing the record in the cataloging module.
The -m flag signifies that both bibs and mfhds will be loaded.

Once the files have been loaded, ITS will notify DPA that the job has run and will include the names of the log files that contain the job output. DPA then checks the output stats for the load to see that it has loaded as expected and checks some records via the ProdOrbis cataloging module to assure the records have loaded correctly.

If any record were added as new, you will need to run a grep statement against the output log to find the bib ids for those records.

fgrep 'Adding Bib' log.imp.20031017.1600> /m1/voyager/yaledb/local/GPO/CHANGES/chgsAdds039

This file can then be downloaded to your PC and the bib ids included with the output stats that are sent to Government Documents.

Also, if there were any discards during this load they will need to be sent to Government Documents as well. The discard file will be in the format discard.imp.20031017.1600 and can be found on degama:/m1/voyager/yaledb/rpt with the other output stats.

GPO CHANGES ONLINE NO LONGER AVAILABLE

This file has always been small, so we send it to Government Documents to pull the records into Voyager through the cataloging client to determine if we have a matching record in the database so they can either delete or suppress it.

In order to match correctly with records in our database, this file need to be run through one preprocessing program to add the (GPO) descriptor in front of the tmp number in the 035. To do this we run the file through the Java program, tagchangernew.jar, outlined below.

The Java program is stored on clark: /m1/voyager/yaledb/local/GPO/tagchangernew.jar. The script for running this program is as follows:

From the above directory:

$ java -jar tagchangernew.jar (input) (output)

(input) - where the input file resides, typically, /m1/bulk/GPO/YACCU039

(output) - where the output should be stored - store in the local/GPO/CHANGES directory as chgsnolonger039.tag

After the tagchangernew.jar program is run, the file should be downloaded to a local PC and sent to Government Documents along with the stats from the rest of the files in this load.

GPO CHANGES USGS

GPO Changes USGS tend to contain relatively few records. If there are under ten records, the file should be run through the tagchangernew.jar program to add the (GPO) descriptor and sent to Government Documents to load manually. A larger file can be loaded via the Bulk Import process. To load these records through Bulk Import, only the tagchangernew.jar preprocessing program is necessary as these records should all be overlays of Titles records that already exist in our database and that already have mfhds.

The tagchangernew.jar program is stored on clark: /m1/voyager/yaledb/local/GPO/tagchangernew.jar. The script for running this program is as follows:

From clark: /m1/voyager/yaledb/local/GPO:

$ java -jar tagchangernew.jar (input) (output)

(input) - where the input file resides, typically, /m1/bulk/GPO/YACCU039

(output) - where the output should be stored - store in the local/GPO/CHANGES directory as chgsusgs039.tag

Once the file has been run through the tagchangernew.jar program it is ready to be loaded to production via Endeavor's Bulk Import program. DPA sends an e-mail to ITS letting them know that the file has been preprocessed and is ready to be copied to degama and loaded using the following Bulk Import script:

From degama: /m1/voyager/yaledb/sbin:

$Pbulkimport -f /m1/voyager/yaledb/local/GPO/chgsusgs039.tag -i GPOUPDT -o CHGSUSGS039

The -f flag signifies where the preprocessed records are stored.
The -i flag signifies which Bulk Import Rule will be used.
The -o flag signifies the name of the load under which the record was added. This will be the "operator" found in the bibliographic record history tab when viewing the record in the cataloging module.

A grep statement is run against the output log to isolate the "replaced" record's bib ids. If there are fewer than ten, the history tab in the staff cataloging module should be checked to see if any of these records were formerly suppressed. If they were, they should be resuppressed. If there are more than ten replaced records, follow the procedures for REPLACES below.

$fgrep - log.imp.20031017.1555(output log name) The "-" is the flag to pull out replaces from a log.

If any "Adds" are recorded in the output stats, a grep command must be run against the output log to pull out the bib ids

$ fgrep 'Adding Bib' log.imp.20031017.1555(output log name)

Once DPA is satisfied that the records have loaded correctly, the output stats and list of bib ids that were added as new are sent to Government Documents to review.

GPO CHANGES OTHER (*this load must be run early enough during the day to have time to get the stats and run an Access query for suppressed records against DeLeon before it is updated the next day)

The GPO Changes Other records are all supposed to overlay Titles records in our database. However, we have found this is not what is happening. Because the file is large and the number of records that are added as new is usually over 100 for a Changes Other load, we do some post-processing after the initial load to extract and then reload the records to create mfhds and reload the bibs and mfhds suppressed so that Government Documents can decide if they are needed. Here are the procedures for the GPO Changes Other records:

The GPO Titles Other file must be run through one preprocessing program, tagchangenew.jar, to add the (GPO) descriptor to the tmp number in the 035 |a if it exists.

The tagchangernew.jar program is stored on clark: /m1/voyager/yaledb/local/GPO/tagchangernew.jar. The script for running this program is as follows:

From clark: /m1/voyager/yaledb/local/GPO:

$ java -jar tagchangernew.jar (input) (output)

(input) - where the input file resides, typically, /m1/bulk/GPO/YACC039

(output) - where the output should be stored - store in the local/GPO/CHANGES directory as chgsother039.tag

From degama: /m1/voyager/yaledb/sbin:

$Pbulkimport -f /m1/voyager/yaledb/local/GPO/CHANGES/chgsother039.tag -i GPOUPDT -o CHGSOTHR039

The -f flag signifies where the preprocessed records are stored.
The -i flag signifies which Bulk Import Rule will be used.
The -o flag signifies the name of the load under which the record was added. This will be the "operator" found in the bibliographic record history tab when viewing the record in the cataloging module.

Immediately following this load, two grep statements should be run against the output log. The first one isolates the bib ids for records that were added as new and the second one isolates records that were added as replaces.

fgrep 'Adding Bib' log.imp.20031017.1600> /m1/voyager/yaledb/local/GPO/CHANGES/chgsAdds039

fgrep - log.imp.20031017.1600> /m1/voyager/yaledb/local/GPO/CHANGES/chgsRepl039

These two files are then downloaded to a PC and saved as .txt files.

ADDS:

Import the "Adds" file to Excel deleting all columns except the bib ids. Then resave this file as a .txt file (chgsAdds039.txt). This file is then uploaded to your home directory on degama to use as the input file for running Pmarcexport to extract the bibs that were added as new. Send the follwing script to ITS for running Pmarcexport:

from degama: m1/voyager/yaledb/sbin:

$Pmarcexport -o /m1/voyager/yaledb/local/GPO/CHANGES/chgsAdds039.out -rB -mM -t /export/home/mgarman/chgsAdds039.txt

After the Pmarcexport job is done, the output file must be moved to clark and then run through Prebulk to create an interleaved file of bibs and mfhds, and addmfhdindicators.jar to add the second indicator to the mfhd before it can be reloaded to production in order to overlay the previously loaded records.

Prebulk Script:

from degama: /m1/voyager/yaledb/sbin:

Pprebulk -i /m1/voyager/yaledb/local/GPO/CHANGES/chgsAdds039.d031109(based on date) -o ../local/GPO/CHANGES/chgsAdds039.pre -c ../local/SCRIPTS/gpoship.cfg

The output from Prebulk is then used as the input for the addmfhdindicators.jar program. To run this program the AddMFHDIndicators.ini file found in clark:/m1/voyager/yaledb/local/GPO must be changed to reflect the input path and output path for the file of records we want changed.

Input_File=/m1/voyager/yaledb/local/GPO/CHANGES/chgsAdds039.pre Output_File=/m1/voyager/yaledb/local/GPO/CHANGES/chgsAdds039.mfhdind

The program is then run from the same directory on the server where the program exists (m1/voyager/yaledb/local/GPO) with the following command: runaddmfhdind

Once the preprocessing is done, the files are ready to be reloaded to production via Endeavor's Bulk Import program as suppressed bibs and mfhds. Therefore the GPOSHIP Bulk Import rule is used. DPA sends an e-mail to ITS letting them know that the files have been preprocessed and are ready to be copied to degama and loaded using the following Bulk Import script:

From degama: /m1/voyager/yaledb/sbin:

$ Pbulkimport -f ../local/GPO/CHANGES/chgsAdds039.mfhdind -i GPOSHIP -o CHGSADDS039 -m

The -f flag signifies where the preprocessed records are stored.
The -i flag signifies which Bulk Import Rule will be used.
The -o flag signifies the name of the load under which the record was added. This will be the "operator" found in the bibliographic record history tab when viewing the record in the cataloging module.
The -m flag signifies that both bibs and mfhds will be loaded.

REPLACES:

Import the "Replaces" file to the Access DeLeon reports database formatting the import as a new table for the bib ids only. Name the bib id column BIB_ID, data type: text, not indexed, and with no primary key. Name the new table MGGPOreplacestbl (you will be promted as to whether you want to overwrite the old table - choose yes). Next, run the MGGPOsuppressqry. This will update a table named MGGPOsuppressedtbl with bibs that had been suppressed before these Titles records overlaid them with unsuppressed records. You will need to go into the ProdOrbis cataloging module and resuppress these records.

Once DPA is satisfied that the records have loaded correctly, the output stats, list of bib ids that were added as new, and any records that need to be loaded manually are sent to Government Documents for processing.

Return to Database Loads Page