Metalib Test of Yale Undergraduates
September 2005
This was a test of six undergraduates, five males and 1 female. There were 2 freshman, 3 sophomores and 1 senior. Three indicated low use of library resources (less than once per month), and no one indicated heavy use of the library. Four of the students were freshman or sophomores, who basically have used Orbis to look for books. Two students clearly had a little more experience in searching for articles in library research databases. It began to be clear through this test and a previous test done with graduate students at the School of Medicine that we may think of student Metalib users in distinct groups:
Summary
There is confusion about which results come from books and which results come from articles. There is sometimes confusion about where to search for these different items.
Subject searching leads people to trouble. Surprisingly 3 of 6 undergraduates tested went out of their way to select subject search, usually because they wanted to be more focused in their approach.
Almost uniformly (5 of 6) undergraduates were not able to find book holdings in the Metalib full record display, and mistook the LC call number for a Yale call number.
Similarly with articles, 3 of 6 undergraduates did not find it obvious how to get full text or how to check for print. The one Metalib feature that helped was the prevalence of full text brought into the Metalib record from some databases. By chance, several undergraduates hit records containing full text in Metalib, and it was apparent to them that they had found full text.
Multidatabase searching is problematic. The most common response from people when asked to find a set of databases in a topic area was to go to MultiDatabase and then search for the category (that is, they entered "history" in the center search box). 3 of 6 undergraduates after a great amount of fumbling are able to correct for this problem, but it is not intuitive. Although the configuration of Multidatabase search bothered people, it was encouraging that half were able to correct their mistakes without being shown what to do.
Undergraduates (4 of 6 failed to do this) were often unable to make a connection between the process of clicking on "Add to My Research" and then going to the "My Research" section of Metalib.
Results:
Question 1.a. If you
were looking for a book about Greek culture please show me from this page where
you would first go.
Success=3
Success after initial failure= 3
Failure to complete=0
All chose QuickStart.
"I'm a student in a hurry. I'm thinking 'Quick' is a good thing."
Question 1.b Ok, now
please do the search for a book about the Greek culture.
Three chose Subject Search (got no results) and three chose Keyword (or did not change the default). Five chose the Local search, one chose Books Anywhere. (It was not clear to this person that Local Catalogs search looked for books. )
After choosing Subject Search and getting no results all thought, with no prompting, to redo the search. With this in mind (3 of 6 needed to redo the search) all went on to successfully complete this task.
It was already shown in tests of graduate students that leaving subject search in is problematic, and this option should be removed.
Students who made the mistake of searching by subject
were able to correct by going back and searching with keyword. However, this impediment should be
removed.
1c.Let's take a moment to talk about this display of
results.
·
How would you go about looking through the results
to find a book that interests you?
·
How many pages would you be willing to scroll
through?
·
Is anything about the display confusing?
Most subjects (5) indicated they would look through 3 to 5 pages of results. One indicated he only scans one page, and he felt slightly embarrassed by this. Many mentioned that it really would depend on the situation/importance/assignment, etc.
No one talked about issues of discrepancy between the
number of actual results (384) versus displayed results (57). This was after being given time to look at
the screen, prodded to discuss things that confused them, etc. It is probable that users will not notice
that they don't see all results, and with very large sets they will also never
get to the last screen that says "Get more."
Question 1.d Pick
any one of the books and tell me
·
where it is located
·
the call number
·
is it available to be checked out
Success=0
Success after initial failure= 1
Failure to complete=5
This question brought unexpectedly bad results. Only one
of five successfully completed this question, and the one who successfully
completed it only found holdings secondarily: after looking at the full record,
he went back to title display and found the Orbis link, and said he would go
there. Most people assumed the LC
record was the Yale call number. This
is a significant problem. They did not
follow the Orbis link, holdings link, or Yale Links, even with prompting to see
if they could determine if Yale had the volume, if it was available to be
checked out, etc.
One person commented that he didn't want to follow the
Orbis link (after I noted that he kept passing the cursor over it) because he
thought that he would need to redo the search, and he only wanted this specific
record.
2a. Please show
me how you would go about finding some general background information in
articles about living wills.
Success=3
Success after initial failure= 2
Failure to complete=1
5 people went to QuickStart. Two of those continued to search books, one was able to correct that mistake, and one never did. This person did the search and from the displayed results couldn't tell what was an article and what was not.
1 person went to MultiDatabase search and stumbled with finding a set, and eventually found the Quickset (so basically was doing a QuickStart search from the MultiDatabase interface).
Five people of six were eventually able to successfully run
a search for articles. However, four
of the six expressed a level of uncertainty about what is in Orbis versus what
is in research databases.
Question 2b Please
pick any article that looks interesting and show me how you would find
·
more about that article
·
full text or
·
print copy of the journal
Success=3
Failure=3
Three were able to do this, mainly when the fulltext was pulled into the Metalib record. Three could not find any way indicated to check for full text. Problems included the indication that "No holdings available" for a database record, a lack of knowledge that Yale Links would lead to this type of information, and an inability to navigate the native interface links.
Question 3: 3a. Let's say a professor asks you to use a
particular database to research a topic.
Could you show me how you'd use this tool to find a particular database,
in this case the Biography Resource Center?
Success=5
Failure=1
Five chose to use Find and were successful. One went to MultiDatabase search and could not complete the task.
3. b Assume you
don't know anything about this resource.
Can you show me from this display how you would find out more about
it?
3.c. Do you ever use this type of information on the
library's web site, and do you find it useful?
When prompted to find out more about the database, four
went directly to the native interface.
When prompted, they did not find the "I" button. When shown the information available, all
indicated it was useful. When the
"I" was pointed out, some people were puzzled by it, and perhaps
"I" not yet universally recognized as the symbol for information.
Others recognized it when they saw it, but still the "I" is quite
far from the title, and the eye may not be drawn to it.
Question 4: Please show me how you'd identify a group of
research databases that you could use to find articles about American History.
Success=0
Success after initial failure= 3
Failure to complete=3
The use of MultiDatabase to identify a set of databases by topic is problematic. Half the people who tried failed completely. Others struggled but were able to complete the task.
Most people at the MultiDatabase page immediately try to search by the broad topic of interest. That is, they go to the search box in the center of the page and type in "history." They typically do not see the search options (pick a category) until the original search they enter does not produce the results they expect.
Once they find the options, many people still have difficulty choosing the right combination of options so that they see the listing of categories and then subcategories.
One person used the categories function of Find a Database to try to complete this question. The navigation and layout of this page worked very well for this one person, however it is not possible with our current configuration to select databases and search from the Find a Database section.
Question 4b: Show
me how you'd use this tool to search these databases simultaneously for
information about the United States Civil War.
Success=3
Success after initial failure= 2
Failure to complete=1
Once people understood and had a set of databases in front of them they were able to search. Only one person was really defeated by the whole MultiDatabase display. Two people at first neglected to selected databases, but they easily self-corrected when they saw no results.
The way that this question was phrased it was not easy to determine how well people understood the way results came back. There seemed to be an assumption that they were seeing results from all the databases. One person was very confused by the link to view records in JSTOR (a search and link). He at first wouldn't follow the link and thought something wrong had happened, he went back and reran the search. When he saw the same result he was willing to follow the link. This is the one person who then went back and looked at "results in other databases" and it seemed to be partly because being pushed into the different JSTOR interface alerted him that he needed to do something to see results in other databases. No one else looked at results in other databases.
Question 5: Let's assume that you would like to go back
and review the first search you did today and look at the results again. Please show me how you would do that.
Success=5
Success after initial failure=0
Failure=1
Results are somewhat skewed--at this point in the test Previous search was obviously displayed. The one failure was a student who just couldn't seem to grasp what I was asking. Several students said that in reality what they would do would be to retype the first search. This was not a priority for these students; no one expressed any feeling that this was particularly helpful.
Question 6: For the rest of the questions I'm going to
ask you to login to the service. Please
login now.
Success 6 out of 6
No one had any problem at all finding or completing the login procedure.
Question 7: Please search simultaneously in two
databases, Literature Online with MLA
and ABELL, Annual Bibliography of English Language and Literature, for
articles about Graham Greene.
Success=2
Success after initial failure=1
Failure=3
This broke down into two categories of people: the first successful category consisted of people who immediately looked for the English subcategory (3 out of 3 easily identified Lang&Lit and then went to English). One of these people initially tried to search by subject, which did not produce results, then switched to keyword. The second category of people who could not complete this tried to search for the databases by name, and then couldn't figure out how to search them together.
Question 8: 8. a.
From the search results, pick 5 citations that interest you. Can you save these citations and email them
to a colleague?
Success=1
Success after initial failure=1
Failure=4
Majority could not do this. Often people's first instinct was to click on Add to My Research, but with the exception of one person, no one could easily figure out how to get from the Add to My Research to My Research. People clearly were looking for something to display when they clicked on Add to My Research, but the fact that it turned Red, changed color asn't enough. They were looking for some indication of "where do I go now." Of the four failures, all four did go into the full record and found the option that would let them send one record at a time.
There was not an obvious connection between Add to My
Research and My Research.
Debriefing
All (6 of 6) said they would use this again. One of the most inexperienced users did indicate that this was too complex for most of the simple searches he needs to do in Orbis. People were happy that it let you search across databases. Students who have had experiences searching already (juniors and seniors) showed more enthusiasm for the timesaving aspects of Metalib. Also mentioned as a positive was that a student is able to identify databases in a given subject area. One person was especially pleased that he could search for books and articles at the same time, because he was aware that he couldn't do that now in Orbis.
General negative comments centered around the difficulty in multidatabase searching and the lack of clarity in the full record when they were trying to find where a book was or how to get the article. There were also technical problems that one student indicated would discourage him from using Metalib if he encountered them again.
Actions taken since this test:
Based on these results, some changes have already been made, or will be made on the test server for further exploration:
Done:
For testing: