LMS Evaluation Work Group

Thursday, November 9, 2000

9:00 a.m. to 10:00 a.m.

SML Spoon

All members of the work group introduced themselves.

Holly explained that the large Evaluation Work Group is actually composed of two groups:

• Will write a summary report as a group to weigh pros and cons of each system and raise issues of concern.

Audrey Novak, the LMS Migration Project Manager, summarized some recent decisions of the Migration Management Group (MMG)

The MMG has been gathering and reviewing a great deal of information about the candidate systems. They are reviewing systems information, hardware configuration options and costs, the results of telephone surveys of other sites, and the vendors' replies to our Criteria documents. The MMG decided that based on this information it would attempt to identify only two systems for exhaustive return demonstration sessions in January. The MMG was motivated in this by the realization that each of the demonstration sessions is likely to require, library-wide, something like 300 days of full-time equivalent staff effort. Their decision is expected Dec 13th.

Audrey is preparing a systematic, comparative analysis of the vendor responses to the criteria document. These reports will be available for consultation. They vary in length and complexity.

Overall the systems appear to be competitive. Our evaluation will determine the areas where there is better capability in one system.

A Site Visit Work Group has been created to visit libraries where the two vendors have installed systems. They will begin work after December 13.

Audrey emphasized that

• This Evaluation WG is a key effort that summarizes earlier work on the migration
• All members of the Evaluation Work Group were involved in either Demo Situation or Criteria Work Group
• This is not a lifetime commitment

The Group discussed the actual evaluation process.

• Holly summarized the difficulty of evaluating the vendors by purely numeric scoring.
• We will use checklists based on the demo situations with room for notes.
• Holly and Audrey will score them to minimize variation across the vendor system.
• Consistency across the vendors by each evaluator will be important.
• All staff who attend demos will be invited to complete short evaluations or provide comments after each demo.
• Focus groups, facilitated by Holly, for all staff who want to discuss a particular module will be held after the demos. The discussion will be based on the summary documents that the Evaluation Work Group prepares and any comments submitted by staff after the demos.

Holly outlined the timeline of the Group’s work

• November 9 – Large Group meeting for orientation
• November-December – Specialized Evaluation Groups begin to formulate questions to ask vendors
• December 13 – Final two vendors are announced. Demo situations sent to them for revisions.
• Mid-January – Meeting of the six Functional Evaluation Groups to review demo situations and clarify evaluation process
• Mid-January – Meeting of the Specialized Evaluation Groups to review questions and clarify evaluation process.
• Late January – Two weeks of demos
• Later January – Small groups meet to discuss pros and cons and submit reports
• Later January – Holly and Audrey score evaluation sheets of all groups
• Early February – Focus groups based on Functional Evaluation Groups meet to discuss vendors.
• Mid February – Final Evaluation Work Group report submitted to MMG

Holly asked each small group to nominate a point person. These contacts are:

Marybeth Bean

Martha Conway

Marcia Romanansky

Steven Arakawa

Mary Angelotti

Karen Reardon

Rich Richie

Fred Martz

Kalee Sprague

Scott Bennett