Research Guides Usability Test

UNC Charlotte

Research Methods: Usability Testing

2018

The Problem

Research guides are organized collections of resources (books, articles, videos, relevant databases, etc.) on a subject or for a specific class. They're part of a vendor platform called SpringShare used by most libraries in the U.S. At university libraries, undergraduate students use them as a kind of shortcut to locate the best sources for their research. Freshman and Sophomore students in particular use research guides because they are often new to scholarly research, and may not trust Google to return sources that are Professor-approved. Research guides are curated by subject librarians, so students know they contain trustworthy sources (so they won't lose points on assignments for using non-academic sources they found in the dark recesses of the internet.)

Even though the research guides tend to be incredibly valuable for the students who knew about them, only students who already knew about them tended to use them for two reasons:

They were nearly unfindable: research guides were buried on the library website and there was weak information scent leading students to them.
Students new to research guides weren't clear on the purpose of the guides. That made it difficult for them to assess why they would even want to look for sources in a guide instead of doing a basic search on Google or on the library homepage.

We needed to understand how students located research guides from the library homepage, navigated content within guides, and assessed the purpose and value of guides so that the library could make them a more useful resource.

When we started, the research guides homepage looked like this:

Yikes, this

can't be the

right place

My Role

I planned and executed the usability study including recruiting participants and moderating test sessions with users. I held a kickoff session with the project team working on updating the research guides to determine our research questions, timeline, and top tasks for usability testing. I conducted 45-minute in-person usability test sessions with undergraduate and graduate students. Because it's important to me to bring key stakeholders along for the research, I invited interested stakeholders including research librarians and library leadership to observe and take notes on the usability test sessions in a separate drop-in "observer lab" (read: a conference room down the hall connected to our room via Google Hangouts - resource constraints often lead to creative solutions).

After completing the study, I presented findings and recommendations to the project team first, then in a larger meeting of Research and Instructional Librarians who were responsible for creating the research guides. After the research findings presentations, I handed off the project to the UX Designer to continue working with the project team to implement the study recommendations.

Work

Project Kickoff and Current State Journey Mapping

In the kickoff meeting with the project team, I led the team through an exercise to map the journey of an undergraduate student trying to use a research guide to find the right resource for a paper. We used sticky notes and a whiteboard to build a basic journey map that included what the user was doing, thinking, saying, and feeling at each step along the way.

After journey mapping, we identified the top 2-3 users of research guides to build out our recruitment screener, and identified the top tasks for each of these users to provide me with direction as I built the usability test script. Finally, we worked together to identify the team's research questions and call out the specific decisions they needed to make using data from this usability study.

Recruitment

After the initial kickoff meeting with the project team, I developed a research plan and began recruiting participants. I designed recruitment materials and collaborated with campus communications to post the digital flyer on monitors located in the library and in student common spaces around the campus like the dining hall and student union. The goal was to recruit undergraduate and graduate students representing a range of majors, identities, and level of experience using the library website. I managed communications and scheduling of all participants. I recruited 5 undergraduate students and 2 graduate students.

Research Questions

With the project team, I aligned on the following research questions to guide the study:

Behavioral

How do users locate information within the guides?
How do users find and access the guides initially? What is the user flow from the homepage to their final action (locating a resource)?
How do users interact with the tabbed navigation within guides?
Are users able to return to the research guides after navigating away from them to complete tasks?
Where do students turn for help if they encounter a question or problem?

Attitudinal

What do users think of the layout and design of the research guides?
What improvements can we make to the overall layout and look and feel of the guides?

Study Design

I conducted in-person, moderated usability tests in scheduled 45 minute appointments with each participant over the course of 2 days. The appointments began with open-ended interview questions to understand the participant's experience using the library website to conduct research, and their familiarity with research guides. Next, we moved into task-based usability testing using a think-aloud protocol to capture user reactions, reasoning, decision-making, and understanding.

Along with critical qualitative data captured as participants thought aloud, I also captured quantitative data including task success rates, time on task, and asked participants to rate the ease or difficulty of completing each task.

To ensure that I would be able to answer the team's attitudinal research questions about student reactions to the design of the page, I also included an X's and O's activity after the task-based usability test. During this portion of the study, I handed the student printed-out screenshots of three different research guides and asked them to circle anything on the page they found useful, and draw an X over anything on the page they found confusing or unhelpful, and then walk me through their reasoning. At the conclusion of the test, I thanked the participants and provided them with a gift card incentive.

Observer Lab

One of my goals as the UX Researcher at UNC Charlotte's Atkins Library was to build a UX and design thinking culture among my colleagues in the library. To do this, I always looked for opportunities to involve stakeholders and co-workers in my research, even if they weren't directly involved in the project. As part of this larger goal, and with the permission of the project team, I set up an observer lab in a separate conference room where stakeholders could observe usability test sessions throughout the 2 days of testing. I sent out a library-wide invitation to drop-in to the observation lab. I created an observer note-taking guide and printed out stacks of them to give observers a structured format for recording observations. After each test session, I returned to the observer lab to debrief on what we saw, what we heard, and to whiteboard potential usability improvements while they were fresh in our minds.

Reporting and Presentations

I wrote a report for the study, created a slide deck, and compiled and edited video highlight reels to share some of the most critical usability issues observed during testing. I presented the findings and recommendations with the project team as well as a larger group of stakeholders who were responsible for updating and editing content within the research guides.

Outcome

As a result of the usability study, the project team aligned on making minor updates immediately. This included disabling the search box within research guides that tended to confuse students because it didn't work the way they expected. We also prioritized and built an action plan for tackling some of the more major recommendations, including updating and standardizing the format of research guides, matching the formatting to the rest of the website to give students confidence that they are using a library resource, and to redesign the research guides homepage to more clearly convey the value of research guides and improve the information scent. After completing the study, the library's UX Designer continued to work with the project team on making these and additional updates. The updated research guides launched in August 2019.