Appendix D: notes from semi-structured interviews, Group A1 All of the notes which follow were made by the interviewers during the semi-structured interviews and are necessarily rough, since note-taking was the only method by which these interviews were recorded. The notes have been edited for privacy and intelligibility. Except where enclosed in quotation marks, the notes do not represent direct quotes but represent the interviewer’s understanding of what was said. Notes in square brackets are interviewer’s comments where indicated; elsewhere, they are additions for intelligibility or privacy, made during transcription and editing. COHSE-NeLI interviews ===================== Long format ----------- COHSE user 1 Job Title and Responsibilities: Senior Lecturer What made you decide to do the evaluation? Interesting What did you think? Hard, not intuitive, hard to find information What would make you want to use the Sealife browser regularly? No Further discussion: Hard – both [control and intervention], link box not relevant, […] info not relevant. Interface – not clear, lots of stuff on Home Page - Took a long time searching for stuff not relevant - Popup box not useful - Found search more precisely, not using [COHSE] which took her to stuff not relevant Not encouraged to use. Task: Not difficult, some brought too many papers – not sure were relevant - [need to] show [participants] how to use it - “this is how you should use it” would make all the difference I wouldn’t use it. Demonstrate how to use it, there are quicker and easier ways of answering the questions. [usually uses] Google, Google Scholar COHSE user 2 Job Title and Responsibilities: Librarian: trains students how to do literature searching. This is what she primarily did before this job. Time in the profession: 20-odd years What made you decide to do the evaluation? Wanted to see what it was What did you think? Did not like at all What would make you want to use the Sealife browser regularly? [extensive feedback about the NeLI DL, and not about COHSE] Further discussion: [extensive feedback about the NeLI DL, and not about COHSE] Clicked at bottom of page to go to HPA when couldn’t find on NeLI, went to another webpage. [Interviewer asks – prompted by the fact that she left the test site to go to the HPA – why didn’t she also leave the site and use Google instead?] I wouldn’t use [Google], I would go straight to NLH. We spend our lives trying to stop people using Google. [Usually uses] databases: OVID, Medline, HMIC, Cinahl, Amed, Evidence-Based Medicine, BNI, Global Health (which would have the statistics) [Interviewer asks: did you make any use of link boxes?] “Those awful little boxes? They were really distracting, I didn’t really understand what they were.” Did click on the things sometimes. [Interviewer asks: so you avoided them?] “No – if it looked helpful I clicked on it” but it may have been the look of them. - If you wanted to click on the home page it would just come up with that box again [Interviewer opens a link box to show interviewee and asks explicitly whether interviewee used them] Didn’t look at link boxes properly because they looked “really heavy” - Maybe it’s about being aware what the link boxes are for. - “Didn’t know what the blue things were” - Instead of using the link box for “prison” clicked on HPA logo at bottom of page and entered “prison” - If she’d realised purpose of link boxes, her experience might not have been worse - It’s quite helpful if it was clearer what it was and how you got at it – not clear that now you’re going to get some dropdown boxes that are going to refine your search. If so, would have paid more attention instead of ignoring them. - Highlighting obliterated original link, stopped her going Home – maybe that’s why she started ignoring them - but there are other links [to Home] on the page. COHSE user 3 What did you think? Find it difficult. Students [would find it] even worse – tried to do it from their perspective (Students: First-year, Diploma for Nursing, Mature Students, [these students] struggle with Google) What would make you want to use the Sealife browser regularly? [feedback about the NeLI DL and not about COHSE] Further discussion: [Feedback about NeLI DL and external sites, and not about COHSE] COHSE user 4 Job Title and Responsibilities: Surveillance Nurse/Infection Control Team Time in the profession: 2 years What made you decide to do the evaluation? Doing Master’s in Health Informatics – related to studies/courses What did you think? Good system. COHSE resources can expand search Further discussion: Rigid People used to Google Lots of good information: could be made easier 5 mins too short, time not appropriate Very useful Highlights [terms] Blue box Depends on what you are doing Depends on tasks you are doing Types of questions: some specific/not specific, different kind of search Too rigid, have to know resources Liked design Liked text Blue box discreet – […] thought it was a pop-up; clicked away to begin with; define what it is; used to pop-ups – ignore them More options: more resources needed; 2-3 websites only seen [in link boxes]/pop-up Wouldn’t use it as first choice for now Use BMJ for now: not first choice Short format ------------ COHSE user 5 Job Title and Responsibilities: Teaches research skills Time with [organisation] and/or in the profession: 7 years/5 years What made you decide to do the evaluation? Professional relevance Further discussion: Didn’t know the ins and outs. Needs to sit and explore and play with it to become comfortable for a while – not one you’d sit on and immediately love – complex. Goes into different things, depends on what you’re looking for – whereas with PubMed [GoPubMed???] you don’t have a lot of choices. COHSE/NeLI very busy on the senses – too much going on on home page and you don’t know where to go. Ended up on eBug, for navigational purposes needs to look more like GoPubMed – some of her students would get lost in 5 seconds. Better for advanced students who understand the terms. Main search bit needs to be more focussed. Could get lost in unrelated stuff. Busy-ness of home page. eBug [website related to NeLI] came up: thought “what have I done?” COHSE user 7 Job Title and Responsibilities: Student (Nursing) What made you decide to do the evaluation? Curiosity – hoped to use it for research What did you think? Quite helpful – refs to books. What would make you want to use the Sealife browser regularly? Would like to use it regularly. COHSE user 6 Job Title and Responsibilities: Information Assistant/Librarian What made you decide to do the evaluation? Education in the system What did you think? Not much difference What would make you want to use the Sealife browser regularly? Didn’t offer anything he wouldn’t have had, had it not been there Further discussion: Links: one-word links stick you with link boxes instead of the page linked to. Can’t move link box around. [At least 2 people noted this, not only the current subject] COHSE various users Participant’s name: Various user comments noted throughout session Further discussion: If you have multiple subjects and can’t limit the results (i.e. search within results) you get too much info Needed more introduction Not much difference between basic/advanced search If very familiar with COHSE, could speed up, very useful to have extra links. Familiarise with new system. Would have helped to have more demonstration before. Which to ignore/which to press? Not necessarily easier to find the answers through this than through any other service, e.g., Ovid “novice searcher” (expects answers to be in the link titles themselves) Got bored/grumpy and left CORESE-NeLI interviews ====================== CORESE user 1 Job Title and Responsibilities: Healthcare Scientist/Microbiology What made you decide to do the evaluation? Free lunch What did you think? All right – 4th question was easier than 3rd. NeLI alone is easier than CORESE. Keyword-sensitive. What would make you want to use the Sealife browser regularly? Not really [keen to use it regularly] – wasn’t quite clear [in communicating to the user] that it didn’t find any results Further discussion: All right – a little bit harder with CORESE (the 3rd task more so). 4th and 2nd task were easier. Liked the graph – takes the monotony out when you’re looking for something and was “distracting” from the stress (not the task :-)) “enjoyed that, actually” CORESE user 2 Job Title and Responsibilities: Post-Doctoral Scientist What made you decide to do the evaluation? Teabreak – someone gave me a leaflet. Searches on Medline & Pubmed. What did you think? NeLI more useful than CORESE. What would make you want to use the Sealife browser regularly? Because they have to work with guidelines, a lot of documents from NeLI were useful, but he wouldn’t use CORESE because “prion” is classified as a “microorganism” which [he considers] not correct information. [Interviewer later ascertained by Google search quickly shows that experts in the field disagree as to whether a prion is a microorganism or not.] Further discussion: [He considers that] prions are not a microorganism – not correct information. Couldn’t navigate diagram, doesn’t understand what graph is trying to tell us. CORESE user 3 Job Title and Responsibilities: [deleted: medical] What made you decide to do the evaluation? Saw poster. What did you think? Don’t know/not clear what it’s looking for: “care home” finds results, “care homes” does not. What would make you want to use the Sealife browser regularly? Get to know them as you use them. Usually start broader and then narrow it down. Not clear at all, don’t understand what terms to put in. Further discussion: Seems designed to make you find the documents using only certain terms; the words “care home” are the only ones that will find the document that answers one of the questions, therefore test seems contrived/rigged. [Interviewer’s note: this and other comments about vocabulary sensitivity emphasised the need for user tutorials emphasising that CORESE is a graphical navigation tool rather than a keyword search. The need to enrich the NeLI vocabulary, making more terms available for keyword search, was also apparent.] [Typing in] all the likely terms brings nothing whereas using fewer search terms brings more results, which doesn’t make sense. [Interviewer’s note: Actually it does, but several people during this session expressed similar expectations.] CORESE user 4 Job Title and Responsibilities: Information Specialist Time with [organisation] and/or in the profession: [deleted] (previous career in communications and information delivery, not involving search engines) What made you decide to do the evaluation? Something to learn from the other side of the fence What did you think? Didn’t like it What would make you want to use the Sealife browser regularly? Not pretty, where “pretty” => ease of use. E.g. Apple computers are prettier, therefore more useable. Further discussion: Handled misspellings very poorly, if at all. Errs on side of caution – if it had been a human librarian it would have been saying “I haven’t got anything on that” [instead of looking for what they do have that’s similar] FAST search engines give you stuff [including irrelevant stuff], and it’s frustrating, but at least gives you something to work with. Would rather have more peripherally relevant results. Default window is Related Concepts. Related Documents should be the default because [he thinks that] the graph is what uses memory and slows the rest of the search. [Interviewer’s note: the workstations had only 512mb of RAM.] However, the graph does give alternative directions to the info being sought. Graph reminded [him] of French search engine [he really likes], which gives you a mind map, can’t remember the name of it. Has nothing particularly *harsh* to say about it; no strong negative feelings; wouldn’t refuse ever to use it again. Lists delivered in a less appealing way; search box in top left as well as bottom right. [Interviewer’s note: he may not be remembering the layout correctly.] It doesn’t tell him when search is finished. CORESE user 5 Job Title and Responsibilities: Project Manager What made you decide to do the evaluation? Interested in search engines. Wanted to see how intuitive it was. What did you think? Couldn’t find it [the answers]. Not intuitive at all. [Feedback follows about the NeLI DL and not about CORESE] What would make you want to use the Sealife browser regularly? Use same parameters as Google – needs more help in terms of “did you mean”. [I.e. “There were no results, did you mean: “ and then suggesting synonyms, alternative spellings, etc.] Further discussion: No “did you mean” feature, didn’t try to help you, was quite difficult. Put phrases in expecting it to work the same way as you would with Google. Didn’t know how to phrase what he was looking for. Potentially quite useful. CORESE user 6 What made you decide to do the evaluation? To find out more What did you think? More useful than expected; easy to navigate; impressed. What would make you want to use the Sealife browser regularly? More links to it so s/he can find it. A link to [her organisation’s] intranet. Further discussion: Last [task] – not a scientist, so didn’t know a lot about the domain. Found the answers OK, easy to navigate. CORESE user 7 Job Title and Responsibilities: Health Care Scientist What made you decide to do the evaluation? Free voucher/incentives What did you think? Easy to use/a lack of feedback when no documents found. What would make you want to use the Sealife browser regularly? More feedback to the user. Further discussion: Useful for papers/quick/easy. Good. (No delay in displaying results.) CORESE user 8 Job Title and Responsibilities: Data Manager Time with [organisation] and/or in the profession: >5 years What made you decide to do the evaluation? Need to search for information What did you think? Not impressed What would make you want to use the Sealife browser regularly? Same general comments as CORESE user 9. CORESE user 9 Job Title and Responsibilities: Health Care Scientist Time with [organisation] and/or in the profession: ~3 years What made you decide to do the evaluation? Incentives/flyers What did you think? Problems loading Java Further discussion: - Did not work. - No resources came up when searching. - [feedback about NeLI DL and not about CORESE] - Restrictions on usability. - Impression that questions were not tested prior to evaluation [Interviewer’s note: the tasks and questions were, in fact, tested] & documents come up with no answers (frustrating). - Java icon appears, and then nothing. - Documents on left should be sorted in descending order of recency of use. CORESE user 10 Job Title and Responsibilities: Health Care Scientist What made you decide to do the evaluation? Interest in scientific literature. What did you think? Easy to use, but not helpful enough. Not enough user support. What would make you want to use the Sealife browser regularly? [make it] More efficient. Impression that the search did not produce any results or that computer is not working properly.