Saint Louis University Libraries Website Usability Study—Fall 2014 Submitted April 13, 2015 Patricia Gregory (chair), Assistant Dean for Library Assessment, Pius Library Conducted by the Pius/Medical Center Libraries Assessment Committee: Bram Boettge, Access Services Coordinator, Medical Center Library Deborah Cribbs (Ad Hoc Member), Technology Coordinator, Web Developer, Pius Library Ronald Crown, Research and Instruction Librarian, Pius Library Sam Deeljore, Technology Manager, Pius Library Jonathan Harms, Library Associate, Research & Instructional Services, Pius Library Haroon Iqbal, Technology Coordinator, Pius Library Miriam Joseph, Research and Instruction Librarian, Pius Library Lori Kupsky, Catalog Librarian, Pius Library Erica Lauriello, Library Associate, Special Collections, Pius Library Donghua Tao, Medical Reference Librarian, Medical Center Library Saint Louis University Libraries Website Usability Study—Fall 2014 Introduction and Overview A functional, appealing website is vital to the success of organizations of all kinds, including academic institutions. However, redesign is occasionally needed in order to adapt to user needs and current Web design standards. The designs of the websites for Pius XII Memorial Library and the Medical Center Library were last updated in Spring 2010. Thus in December 2013, SLU Libraries began designing new websites for Pius and MCL, and selected Web forms for the Madrid Library. The purpose of the redesign was to create Pius and MCL websites that: • Reflect the needs and interests of library users • Inform users about library resources and services • Help users carry out their research • Showcase unique resources • Brand individual libraries as part of the SLU library system, while ensuring they maintain their unique identity • Situate/contextualize departments and the libraries in relationship to each other • Provide ready access to internal information for library staff, campus partners, and potential employees Among the key components of current Web design standards incorporated into the redesigned websites: Graphics and Layout • Present a simple, clean design • Provide a uniform and distinct organizational identity • Offer a good balance between functionality and aesthetics • Emphasize graphic and interactive navigation elements over long lists of text links • Whenever possible, avoid long drop-down menus and limit the need to scroll Structure • Consolidate and/or unify site functionality and avoid unnecessary duplication of information across the site • Make it easy to find basic information about the library and its services and resources • Provide ready assistance to novice/first-time users, but without disadvantaging more-experienced users Content • Display generally understood words and phrases in all labeling • Use simple language. Organize information logically so it is easy to find things on the site Upon completion of the new websites’ design in September 2014, the Pius-MCL Assessment Committee conducted a usability study to get user input. The results were promising: 88% (101/115) of participants rated their overall experience of the new Pius website positively (“Good” or “Very Good”), and 79% (23/29) of participants rated their overall experience of the new MCL site positively (“Good” or “Very Good”). The overall Pius response averaged 4.22 on a Likert scale of 1 (“Very Poor”) to 5 (“Very Good”). The overall MCL response averaged 3.86 on the same scale. In the Pius study, 90% (103/114) of participants rated their experience of accomplishing key tasks as “Mostly Successful” or “Completely Successful.” In the MCL study, 69% (20/29) of participants rated their overall experience of accomplishing key tasks as “Mostly Successful” to “Completely Successful.” The Pius response averaged 4.31 on a Likert scale of 1 (“Very Poor”) to 5 (“Very Good”). The MCL response averaged 3.83 on the same scale. 2 Saint Louis University Libraries Website Usability Study—Fall 2014 Suggestions from survey participants were analyzed and specific suggestions for improvement were incorporated into the ongoing site redesign. See Design Changes Based on Participant Suggestions (Pius website) and Design Changes Based on Participant Suggestions (MCL website) on pages 5-6 in the Findings section. The new sites launched on March 10, 2015. In Spring 2016 users will be surveyed again to determine their degree of satisfaction. Method Participants Students, staff, and faculty in the University’s St. Louis schools, colleges, and centers were recruited to participate in this study. Students received a message distributed via a University listserv advising them of the purpose of the study and inviting them to register via an embedded link for one of 15 sessions scheduled from October 28 through November 5, 2014. Faculty were contacted by their respective subject librarians, who emailed a similar invitation. All students, faculty, and staff who used public computers at Pius Library or the Medical Center Library from October 3, 2014, through November 5, 2014, viewed a pop-up invitation that displayed soon after login. Ultimately, 152 SLU faculty, students, and staff, representing nine of the eleven St. Louis-based colleges and schools, the libraries, degree-granting centers, and two University office divisions, participated in and completed the study. Participants were asked to identify all status categories applicable to them; their primary affiliations were identified through the University’s online directory. See Appendix 1 for demographic details. Materials and Procedure This usability study involved participant review of draft redesigns of both the Pius Library website and the Medical Center Library website. Participants who attended sessions at the Medical Center Library viewed that site; all other participants viewed the Pius Library website. Regardless of location, each participant worked individually at a computer set up with links to the pertinent website and a Qualtrics questionnaire, as well as CamStudio software, which recorded a video of screen activity. Each participant was assigned and required to enter a unique identifying code, which allowed pairing each questionnaire with the video of that session; however, participants themselves were not identifiable by their responses. Qualtrics also tracked the time spent on each question, which included reading the question, finding the answer on the site and recording the answer in the questionnaire. Sessions typically ran 30–45 minutes. Library faculty and staff served as on-site moderators to answer general questions and assist with technology problems. Participants viewing each draft website were asked to complete the following tasks, identified by library faculty and staff as reflecting some of the most common basic reference queries: • Find the library hours for the current day and a weekend day • Get directions to the library • Renew a book • Find a specific book in the SLU Libraries Catalog • Find a specific e-book in the SLU Libraries Catalog • Describe how to get help from a librarian • Find a journal article • Find an e-journal 3 Saint Louis University Libraries Website Usability Study—Fall 2014 Participants used the draft websites to try to find the information and typed their responses into the Qualtrics form. Simultaneously, the tracking software captured the actual path taken on the websites to accomplish the task. This not only provided more exact information about how tasks were completed, but later allowed library staff to address graphics and layout, structure, and content (e.g., number of clicks required to locate primary information). In addition, participants were asked to (1) rate the new websites on the following criteria: overall experience, ease of navigation, ease of accomplishing task, organization, visual appeal, and amount of content; (2) self-assess their success in accomplishing the search tasks; and (3) compare the draft sites with other library websites they had previously visited. Lastly, participants were asked to suggest changes or additional features and also were invited to provide any additional comments about their visits to the draft websites during this study. Findings Participants’ Overall Experience and Self-Assessment of Success For the Pius website, 88% (101/115) of participants rated their overall experience either “Good” or “Very Good.” The overall Pius response averaged 4.22 on a Likert scale of 1 (“Very Poor”) to 5 (“Very Good”). Ninety percent (103/114) of participants using the Pius website rated their experience of accomplishing the tasks “Mostly Successful” or “Completely Successful.” The Pius response averaged 4.31 on a Likert scale of 1 (“Not at All Successful”) to 5 (“Completely Successful”). For the MCL website, 79% (23/29) of participants rated their overall experience either “Good” or “Very Good.” The overall MCL response averaged 3.86 on a Likert scale of 1 (“Very Poor”) to 5 (“Very Good”). Sixty-nine percent (20/29) of participants using the MCL website rated their overall experience of accomplishing the tasks “Mostly Successful” to “Completely Successful.” The MCL response averaged 3.83 on a Likert scale of 1 (“Not at All Successful”) to 5 (“Completely Successful”). Selected Findings Finding Library Hours (see Task 1 in Appendix 1) Random sampling of the CamStudio videos showed that the majority of participants were able to find the hours for the current day within 10 seconds and with one or two clicks. Most participants simply scrolled to the bottom of the site, where the hours for the current day were displayed. Renewing a Book (see Task 3 in Appendix 1) Random sampling of the CamStudio videos showed that the majority of participants were able to find how to renew a book. Pius website participants averaged 57 seconds to record their answer, and MCL participants averaged 101 seconds. MCL participants appear to have taken longer because the MCL site had a “My Account” icon/button, whereas the Pius site had a “Renew Books” icon/button. Finding Books (see Tasks 4-5 in Appendix 1) Almost all participants were able to find the location of a physical book using the Pius website. A smaller percentage were successful using the MCL website, but most of the incorrect responses simply found the wrong edition of the book. It also took participants longer to find the physical book using the MCL website. Participants using either website seemed to struggle more with finding an e-book. There were several comments and suggestions to make searching for e-books easier. Of those participants who found the e-book, the majority 4 Saint Louis University Libraries Website Usability Study—Fall 2014 used the search box on the websites’ homepages. Finding an e-book took less time than finding a physical book using either website. Getting help from a librarian (see Task 6 in Appendix 1) Participants were asked to list as many ways as they could find to get help from a librarian. Most participants were able to list between three to six different ways within 45 to 60 seconds. Finding Electronic Articles and Journals (see Tasks 7-8 in Appendix 1) Almost all participants were able to find electronic articles and journals successfully using either website, and most used the search boxes on the homepages. Several of the different tabs within the search box on the Pius homepage were used, but the “Journals” tab was used the most. Within the search box on the MCL site, the “Journal Articles” tab was used the most by far. Some used the “Quick Links” at the bottom of the MCL homepage as well. Average time was 54 seconds for Pius website users and 85 seconds for MCL website users. Design Changes Based on Participant Suggestions (Pius website) Participant responses to the questions on rating, self-assessment of task success, and comments were reviewed by the committee. The following suggestions were later incorporated into the Pius website design: Graphics & Layout • Changed landing page so the user doesn’t have to scroll (not visible in “Before/After” below) • Changed wording on landing page for column headings and listed links (see “Before/After” below) • Removed Quick Links heading from footer (participants said it wasn’t useful there) • Increased space around “Ask Us” icon (several participants said it didn’t stand out enough) • Enlarged icons and increased contrast of icons and text in 3x3 grid at middle right of homepage Structure • Simplified sub-items under the “About” menu tab Content • No changes “Before” (Pius website used for survey) “After” (Pius website incorporating suggestions) 5 Saint Louis University Libraries Website Usability Study—Fall 2014 Design Changes Based on Participant Suggestions (MCL website) As with the Pius site, participant responses to the questions on rating, self-assessment of task success, and comments were reviewed by the committee. Among the main suggestions: • Making icons stand out more. They found the icons unnoticeable or unneeded due to the color used, size, and placement of the icons. • Adding more contact information, e.g., adding more visible link to Research Librarians, making “Ask Us” button more noticeable, including location of librarians, etc. • Adding the “Renew Books” link and more quick links to specific resources. Some users reported that they had trouble renewing books and finding book (both physical and electronic) information. The following changes based on participant suggestions were later incorporated into the MCL website design: Graphics and Layout • Changed wording on landing page for column headings and listed links—Contact information in first column was replaced with Resources containing links for: Interlibrary Loan, Renew a Book, and eBooks • Changed My Account graphic on landing page, upper left corner, to Renew a Book • Added tab graphic to search box, making the active column more visible • Removed Quick Links heading from footer • Changed links in footer to include Renew Books (changed from My Account), mySLU, Email Feedback, and Ask a Librarian. Removed Contact, LibAnswers, Café, and Research Guides • Changed Interlibrary Loan icon to Liaison Librarians • Databases search—made drop down search box longer Structural • Changed “Core Electronic Text Books” under “Research” to “eBooks” • Books search—added eBooks parameter for catalog search Content • No changes “Before” (MCL website used for survey) “After” (MCL website incorporating suggestions) 6 Saint Louis University Libraries Website Usability Study—Fall 2014 Appendix 1: Survey Results and Responses In addition to recording participants’ answers, the Qualtrics software recorded several other metrics for each task: time spent; number of clicks, time of first and last click. Specifically, it measured the amount of time that passed from the moment the page with the question was loaded until the participant’s last interaction with the Qualtrics survey (not including their click to move to the next page). We included time for both correct and incorrect answers. Participants’ task answers, suggestions, and comments for Pius and MCL website were coded separately by two different committee members for each website. Note: Participants who attempted to go past a question without answering it were prompted to answer it by a Qualtrics pop-up box, but they were allowed to skip questions if they wished. Thus, the number of participants who answered each question varies. Task 1: What are the hours of [name of library] for today? For this Saturday? Pius: 90% (103/114) of participants found the correct hours for the current day, while 6% (7/114) wrote “23hrs,” which could possibly be considered typing error. 91% (89/98) found the correct hours for the upcoming Saturday. Participants averaged 51 seconds to find both answers. MCL: 93% (28/30) of participants found the correct hours for the current day, and 93% (26/28) found the correct hours for Saturday. The average amount of time it took participants to find answers for both questions was 61 seconds. Results for Pius: Results for MCL: Today’s hours—121 answered correctly Today’s hours—31 answered correctly Saturday’s hours—115 answered correctly Saturday’s hours—29 answered correctly Total Responses—123 Total Responses—33 Task 2: How can you find directions to the library? Results for Pius: Results for MCL: “Directions” link at bottom of the homepage—59 “Directions” link at bottom of the homepage—23 Click on Address at bottom of the homepage—16 Click on Address at bottom of the homepage—1 “About” menu item > “Contact” > “Directions and “About” menu item > “Contact” > “Directions and Parking”—15 Parking”—5 Other response—6 Not found—3 Not completed / no data—25 No response—1 Total Responses—121 Total Responses—33 Appendix 1, Page 1 Saint Louis University Libraries Website Usability Study—Fall 2014 Task 3: How can you renew a book online? Pius: Average time to answer: 57 seconds MCL: Average time to answer: 101 seconds Results for Pius: Results for MCL: “Renew Books” blue button/icon—70 “Services” menu > “My Library Account”—2 “Renew Books” link under For Students—29 “My Account” link location undetermined—10 “My Account” link at the bottom of the homepage—7 “My Account” link at the bottom of the homepage—7 Other—13 Couldn’t find—3 Answer Found but path cannot be determined—5 Used “Search SLU” box in upper right corner—2 “LibAnswers” link at the bottom of the homepage”—1 Total Responses—119 Total Responses—30 Task 4 (Pius): Where is the book Quantum Man: Richard Feynman’s Life in Science located in the library? Task 4 (MCL): What is the call number of the book Mosby’s Guide to Physical Examination/ Henry M. Seidel, c2011? Pius: 97% (29/30)* of participants found the correct location. Average answer time was 99 seconds. (*Note: 84 responses to this question were lost during data transfer from survey to spreadsheet.) MCL: 85% (25/30) of participants gave the correct call number for the specified book, while 10% (3/30) gave the call number for the previous edition of the book. Average answer time was 154 seconds. Results for Pius: Results for MCL: QC16.F49 K73 2011—15 WB 205 M894 2011—25 Level 5 South—11 WB 205 M894 1995—3 Pius Library—3 Answer not found—1 Other—1 Other—1 Total responses—30 Total responses—30 Task 5 (Pius): Where would you find the e-book Republic of Noise: The Loss of Solitude in Schools and Culture by Diana Senechal? Task 5 (MCL): Where would you find the e-book Andreoli and Carpenter’s Cecil Essentials of Medicine / editor-in-chief, Thomas E. Andreoli, c2010? Pius: 82% (83/101) of participants found the e-book, and at least 51% (42/83) of those used the search box on the home page. Average answer time 69 seconds. MCL: 73% (22/30) of participants found the e-book, and at least 55% (12/22) of those used the search box on the home page. Average answer time was 87 seconds. Appendix 1, Page 2 Saint Louis University Libraries Website Usability Study—Fall 2014 Results for Pius: Results for MCL: Answer Found but path cannot be determined—28 “Books” tab in search box > “eBooks”—12 “Books” tab in search box > “e-books”—22 “Books” tab in search box w/out specifying “eBooks”—4 “SLUth” tab in search box—13 MCL Reserve—2 SLU Library Catalog (not specified)—8 “Research” menu item > “E-Books” > Search—2 Answer not found—7 Answer found but path cannot be determined—6 Search box (not specified)—6 Answer not found—6 Other—17 Other—1 Total responses—101 Total responses—30 Task 6: How many ways can you use the website to get help from a librarian? Results for Pius: Results for MCL: Average number of ways—4.52 Average number of ways—3.93 “Contact Us” tab—14 “Ask a Librarian” link (Pius website)- 10 “Subject Librarian” icon—13 “Contact Us” icon—9 “Ask a Librarian” link—11 “Questions? Ask Us” link (Pius website)- 7 “Questions? Ask Us” link—10 “Contact Us” icon—(Pius website)—3 “Help” link in the middle—9 Other—5 “Directory” icon—5 Quick Links—5 For Students—5 For Faculty/Staff Resource page—5 Other—6 Not completed / no data—18 Total Responses—101 Total responses—29 Task 7 (Pius): You are writing a research paper on American history and need to find journal articles. Where on the SLU Libraries website would you begin? Task 7 (MCL): You are writing a research paper on the public health aspects of the Ebola virus. Where on the site would you begin to find relevant journal articles? Pius: At least 84% (100/118) of participants used the search box on the homepage to find relevant journal articles, and of those participants, 37% (37/100) used the “Journals” tab, 26% (26/100) used the “Databases” tab, 21% (21/100) used the “SLUth” tab, and 16% (16/100) used the “Research Guides” tab. Average time to find an answer was 54 seconds. MCL: All participants used an efficient path to find relevant journal articles. More than 86% (25/29) used the search box on the homepage, and 76% (19/25) of those used the “Journal Articles” tab specifically. 20% (6/29) used the “Quick Links” on the bottom of the homepage. Average time to find an answer was 85 seconds. Appendix 1, Page 3 Saint Louis University Libraries Website Usability Study—Fall 2014 Results for Pius: Results for MCL: “SLUth” tab search box—21 “SLUth” tab search box—2 “Journals” tab search box—37 “Journal Articles” tab in search box—19 “Research Guides” tab in search box—16 “Quick Links” on homepage > PubMed—5 “Databases” tab in search box—26 “Databases” tab in search box—5 Other—3 “Research” drop down—1 Not completed / no data—15 Total Responses—118 Total Responses—29 Task 8 (Pius): Where would you find International Journal of Accounting in electronic format? Task 8 (MCL): Where would you find Journal of Advanced Nursing in electronic format? Pius: At least 55% (65/118) of participants used the search box on the homepage to find an e-journal, and 77% (50/65) of those used the “Journals” tab specifically. The average time to find an answer was 53 seconds. MCL: All participants used an efficient path to find an e-journal. About 69% (20/29) used the “eJournals” search option in the “Journal Articles” tab of the search box on the homepage, while another 24.1% (7/29) used the “E- Journal Portal” under “Quick Links” on the homepage. The average time to find an answer was 107 seconds. Results for Pius: Results for MCL: “SLUth” tab in search box—13 “Quick Links” on homepage > “E-Journal Portal”—7 “Journals” tab in search box—50 “Journal Articles” tab in search box > “eJournals”—20 Science Direct Journals—12 “Research” menu item > “E-Journals”—2 “Databases” tab in search box—2 “Books” tab—2 Other—6 Not completed / no data—35 Total Responses—118 Total Responses—29 Task 9 (Pius): Please find the name and the email address for the subject librarian for Theology. Task 9 (MCL): Please find the name and the email address for the subject librarian for Public Health. Results for Pius: Results for MCL: “Ron Crown”—116 answered correctly “Donghua Tao”—23 answered correctly “[email protected]”—113 answered correctly “[email protected]”—23 answered correctly 1 answered incorrectly 6 answered incorrectly Total Responses—117 Total Responses—29 Appendix 1, Page 4