Rapid usability testing: Developing a process
Ginkgo Bioworks is a Bio Tech company that also makes in-house software to support their in-house scientists.
But lots of that in-house software went out untested with users,
so many software issues went undiscovered...
The Challenge
Product Design / UX at Ginkgo Bioworks was only a few years old when I joined, and still a growing team. At that point, software engineers weren't used to engaging with product design or UX research, and a lot of software was going out untested by users.
The Product Design team wasn't yet large enough to cover all the software teams & products, so I was tasked with ensuring that usability issues were discovered and could be fixed early, across 8+ software & content teams, and around 20 products.
The Solution
I developed and iterated on a process of week-long usability testing sprints, rotating through the different software teams throughout each quarter. The process I developed was efficient but also engaging & highly collaborative.
Read below for more about how I thought through the challenge, iterated on the process, and the results & impacts of the process I built...
How I tackled the challenge
I worked with my manager to identify all the user-facing software teams and the software they supported, and set a rotating schedule. I picked one team to be my pilot session, and started with an initial process.
Here's an outline of the process and my reasoning behind it:
| About the method: | Why: |
|---|---|
| One week running sessions for a scrum team, while planning for the next week's testing | This felt like the shortest amount of time I could reasonably do a round of research. |
| Qualitative usability testing | Qualitative usability testing is great for identifying usability issues and needs ~5 users, which was doable within a single week. |
| Collaborate with the scrum team members to have them attend sessions and take notes | This gets the team members directly witnessing users using their software (which has proven benefits) and saves time because I don't have to rewatch a session to take notes before the synthesis session. |
| Provide a note-taking template | This helps scrum team members know what to look for and take notes on during sessions. |
| Facilitate group synthesis / analysis session at the end of the week | Again, beneficial to have the whole team involved in reviewing the findings and building consensus, while also having more hands on deck to analyze the findings from the week. |
Iterating over time
As I continued to run these research sprints, I also reflected on the effectiveness of the process, and made small changes over time as I noticed issues or based on feedback I solicited from the scrum teams.
Some examples:
- "Webinar" style Zoom, so lots of scrum team members could observe without the participant feeling self-conscious about the number of people (or who) was observing. This also prevented engineers from jumping in to "help" users overcome issues during the session.
- Quantitative metrics (SUS, SEQ), to provide data around severity of user struggle.
- Concrete prioritization exercise in the synthesis session, so pain points could be more easily compared across usability testing rounds, to help the teams prioritize as time went on.
Results & Impacts
The process I developed resulted in 7 to 40 issues uncovered per round, and teams addressing 2-59% of the issues and opportunities uncovered (except for extenuating circumstances). Smaller issues were typically fixed within a sprint or two, and larger issues & findings informed or initiated larger research and design efforts.
Some teams would even implement small fixes before the end of the testing week!
Here are some outputs from a recent round of this usability testing:
Some excerpts from the research report
Mural of pain points by impact to users and effort to address
Recordings of usability testing sessions with users
Praise about the process
At this time, I've run over 55 research sprints like this, engaging with around 200 users per year.
People from every role on scrum teams have given me consistent, positive feedback about the process and the value they've gotten from it.
I love providing this type of value to the teams, and seeing how much people appreciate it!
Here are a few examples:
