Appendix B – questionnaires ======================================================================================================================= Pre-questionnaires ================== COHSE-NeLI and CORESE-NeLI --------------------------- Question 1 Please enter your current profession (e.g. Nurse, Microbiologist) Answer choices Free text 2 How many years’ experience do you have in your current profession? (Not just your current post) Answer choices 0 or 1 of: - under 5 years - 5 years or more 3 (Optional) Where do you go to find the best available evidence for your work? Answer choices 0 or more of: - BMJ Clinical Evidence - PubMed - Cochrane - Government Sources - Other (please specify) [free text field follows] 4 On average, how often do you use the National Electronic Library of Infection (NeLI) or National Resource for Infection Control (NRIC)? Answer choices 0 or 1 of: - Daily - More than once a week but not as often as daily - Once a week - More than once a month but not as often as once a week 5 For what reasons have you used NeLI/NRIC? Please select all that apply. Answer choices 0 or more of: - General research - Personal study/research/continuing education - Seeking information to help decision-making for a specific patient - I have not used NeLI/NRIC before this study 6 How useful do you find NeLI/NRIC? Answer choices 0 or 1 of: - Very useful - Somewhat useful - Useful - Not particularly useful - Not useful 7 How has NeLI/NRIC helped you in your work? Please select all that apply. Answer choices 0 or more of: - Made it quicker to find the best available evidence - Made it easier to find the best available evidence - Contributed directly to decisions about patient care - Helped to improve my professional knowledge - Influenced my attitudes to professional practice - I have not used NeLI/NRIC before this study GoPubMed/GoGene ------------------ Question 1 My main university degree is: Answer choices 0 or 1 of: - Biology - Physics - Chemistry - Engineering - Other 2 My experience in this field is: Answer choices 0 or 1 of: - under 5 years - 5 years or more 3 I use PubMed: Answer choices 0 or 1 of: - Daily - More than once a week but not as often as daily - Once a week - More than once a month but not as often as once a week 4 I have used for my research (please select all that apply): Answer choices 0 or more of: - PubMed - Google Scholar - HighWire - Google or other web search - Scopus - Scirus - Other specialist 5 On a 1-10 scale, how useful do you find PubMed (1 not useful, 10 extremely useful)? Answer choices Likert scale from 1 (not useful) to 10 (extremely useful) 6 What do you search for in PubMed? Answer choices 0 or more of: - I look for specific articles - I search for reviews for an overview - I look for papers in a specific journal - I look for papers of specific authors - I look for papers for specific diseases, genes, etc. - I look for the most recent papers ============================================================================================================================= Post-task questions: all evaluations ==================================== In every evaluation of every SWB, the task question page appeared after every task page. The task questions, and their answer choices, were always the same. Question 1 How well did the information you found answer the question? Answer choices 0 or 1 of: - Not at all - Partially - Fully 2 Was finding the answer in the information returned by the search engine: Answer choices 0 or 1 of: - Hard - Neither Hard nor Easy - Easy ======================================================================================================================= Post-questionnaire ================== COHSE-NeLI and CORESE-NeLI -------------------------- Question 1 I think that I would like to use this system frequently. Answer choices: Unmodified system (NeLI alone): Likert scale from 1 (Strongly disagree) to 5 (strongly agree) Modified system (NeLI + [SWB]): Likert scale from 1 (Strongly disagree) to 5 (strongly agree) 2 I found the system unnecessarily complex. Answer choices: Unmodified system (NeLI alone): Likert scale from 1 (Strongly disagree) to 5 (strongly agree) Modified system (NeLI + [SWB]): Likert scale from 1 (Strongly disagree) to 5 (strongly agree) 3 I thought the system was easy to use. Answer choices: Unmodified system (NeLI alone): Likert scale from 1 (Strongly disagree) to 5 (strongly agree) Modified system (NeLI + [SWB]): Likert scale from 1 (Strongly disagree) to 5 (strongly agree) 4 The system found a lot of information that was irrelevant to the tasks. Answer choices: Unmodified system (NeLI alone): Likert scale from 1 (Strongly disagree) to 5 (strongly agree) Modified system (NeLI + [SWB]): Likert scale from 1 (Strongly disagree) to 5 (strongly agree) 5 Finding the answers to the tasks in the information returned by the system was: Answer choices: Unmodified system (NeLI alone): Likert scale from 1 (Hard) to 5 (Easy) Modified system (NeLI + [SWB]): Likert scale from 1 (Hard) to 5 (Easy) 6 Most of the information returned by the system was: Answer choices: Unmodified system (NeLI alone): Likert scale from 1 (Irrelevant) to 5 (Relevant) Modified system (NeLI + [SWB]): Likert scale from 1 (Irrelevant) to 5 (Relevant) 7 I was able to find the answers to the tasks in the information found by the system: Answer choices: Unmodified system (NeLI alone): Likert scale from 1 (Slowly) to 5 (Quickly) Modified system (NeLI + [SWB]): Likert scale from 1 (Slowly) to 5 (Quickly) 8 The amount of relevant information I found was: Answer choices: Unmodified system (NeLI alone): 0 or 1 of: - Less than I expected - Same as I expected - More than I expected Modified system (NeLI + [SWB]): 0 or 1 of: - Less than I expected - Same as I expected - More than I expected 9 Were the modifications mostly relevant or mostly irrelevant to you? Answer choices: Modified system (NeLI + [SWB]): Likert scale from 1 (Irrelevant) to 5 (Relevant) 10 I thought the overall system was: a) Unmodified system (NeLI alone): Likert scale from 1 (Terrible) to 5 (Wonderful)) Modified system (NeLI + [SWB]): Likert scale from 1 (Terrible) to 5 (Wonderful)) b) Unmodified system (NeLI alone): Likert scale from 1 (Frustrating) to 5 (Satisfying) Modified system (NeLI + [SWB]): Likert scale from 1 (Frustrating) to 5 (Satisfying) c) Unmodified system (NeLI alone): Likert scale from 1 (Dull) to 5 (Stimulating) Modified system (NeLI + [SWB]): Likert scale from 1 (Dull) to 5 (Stimulating) d) Unmodified system (NeLI alone): Likert scale from 1 (Difficult) to 5 (Easy) Modified system (NeLI + [SWB]): Likert scale from 1 (Difficult) to 5 (Easy) e) Unmodified system (NeLI alone): Likert scale from 1 (Too slow) to 5 (Fast enough) Modified system (NeLI + [SWB]): Likert scale from 1 (Too slow) to 5 (Fast enough) f) Unmodified system (NeLI alone): Likert scale from 1 (Rigid) to 5 (Flexible) Modified system (NeLI + [SWB]): Likert scale from 1 (Rigid) to 5 (Flexible) GoPubMed/GoGene --------------- Question 1 I would like to use this system frequently. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 2 The system was easy to use. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 3 The system was too complex. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 4 The user interface was easy to understand. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 5 The system responded fast. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 6 The system provided enough help information and examples. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 7 Finding the answers to the tasks with the system was easy. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 8 Finding the answers to the tasks with the system was fast. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 9 A lot of information the system found was irrelevant to the tasks. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 10 Most of the information returned by the system was relevant. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 11 The amount of relevant information I found was less than expected. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 12 The amount of relevant information I found was same as expected. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 13 The amount of relevant information I found was more than expected. Answer choices: Unmodified system (PubMed): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) 14 The modifications were mostly relevant to me. Answer choices: Modified system (GoPubMed (short format)) OR (GoGene/GoPubMed (long format)): Likert scale from 1 (Strongly disagree) to 10 (strongly agree) Functionality ------------- 15 Did you find the highlighting of ontology terms helpful? Answer choices 0 or 1 of: - Yes - No 16 Did you get an overview over your search results from the tree on the left? Answer choices 0 or 1 of: - Yes - No 17 Did you manage to navigate efficiently through the tree? Answer choices 0 or 1 of: - Yes - No 18 Did you find any papers you would probably have missed with PubMed? Answer choices 0 or 1 of: - Yes - No 19 What do you like/dislike about using the tree to explore your search results? Answer choices Free text Suggestions ------------ 20 Please write down any other suggestions you have for GoPubMed (short format) OR (GoGene/GoPubMed (long format)). Answer choicesFree text