LMS Evaluation Work Group
Thursday, November 9, 2000
9:00 a.m. to 10:00 a.m.
All members of the work group introduced themselves.
Holly explained that the large Evaluation Work Group is actually composed of two groups:
Functional Evaluation Groups
Six groups focused on particular modules (see below)
Groups will use slightly modified versions of demonstration situations to evaluate the vendors.
Will attend vendor-led demos with all staff.
Will have private hands-on session with vendor.
Will write a summary report as a group to weigh pros and cons of each system and raise issues of concern
Specialized Evaluation Groups
Five groups with specific assignments that are best covered in small, focused discussion.
Groups will compose questions that can be evaluated to ask the vendor
- Will write a summary report as a group to weigh pros and cons of each system and raise issues of concern.
Overview to LMS Project
Audrey Novak, the LMS Migration Project Manager, summarized some recent decisions of the Migration Management Group (MMG)
The MMG has been gathering and reviewing a great deal of information about the candidate systems. They are reviewing systems information, hardware configuration options and costs, the results of telephone surveys of other sites, and the vendors' replies to our Criteria documents. The MMG decided that based on this information it would attempt to identify only two systems for exhaustive return demonstration sessions in January. The MMG was motivated in this by the realization that each of the demonstration sessions is likely to require, library-wide, something like 300 days of full-time equivalent staff effort. Their decision is expected Dec 13th.
Audrey is preparing a systematic, comparative analysis of the vendor responses to the criteria document. These reports will be available for consultation. They vary in length and complexity.
Overall the systems appear to be competitive. Our evaluation will determine the areas where there is better capability in one system.
A Site Visit Work Group has been created to visit libraries where the two vendors have installed systems. They will begin work after December 13.
Audrey emphasized that
- This Evaluation WG is a key effort that summarizes earlier work on the migration
- All members of the Evaluation Work Group were involved in either Demo Situation or Criteria Work Group
- This is not a lifetime commitment
The Group discussed the actual evaluation process.
- Holly summarized the difficulty of evaluating the vendors by purely numeric scoring.
- We will use checklists based on the demo situations with room for notes.
- Holly and Audrey will score them to minimize variation across the vendor system.
- Consistency across the vendors by each evaluator will be important.
- All staff who attend demos will be invited to complete short evaluations or provide comments after each demo.
- Focus groups, facilitated by Holly, for all staff who want to discuss a particular module will be held after the demos. The discussion will be based on the summary documents that the Evaluation Work Group prepares and any comments submitted by staff after the demos.
Timeline for WG
Holly outlined the timeline of the Groups work
- November 9 Large Group meeting for orientation
- November-December Specialized Evaluation Groups begin to formulate questions to ask vendors
- December 13 Final two vendors are announced. Demo situations sent to them for revisions.
- Mid-January Meeting of the six Functional Evaluation Groups to review demo situations and clarify evaluation process
- Mid-January Meeting of the Specialized Evaluation Groups to review questions and clarify evaluation process.
- Late January Two weeks of demos
- Later January Small groups meet to discuss pros and cons and submit reports
- Later January Holly and Audrey score evaluation sheets of all groups
- Early February Focus groups based on Functional Evaluation Groups meet to discuss vendors.
- Mid February Final Evaluation Work Group report submitted to MMG
Point Person for Each Group
Holly asked each small group to nominate a point person. These contacts are: