The best way to validate design and requirement decisions with
your users is via usability testing. The general idea is to gather feedback on
early concepts, understand workflows & user pain points and allow designers
and product managers to document findings and create/maintain personas, roles
and system workflows.
Agile methodology lends
itself extremely well for iterative usability testing. In this article I’ll
review my own team’s process for how we plan to integrate usability testing
into our product development cycle.
Test Subjects
As a general rule, the best test subjects are real product users,
but it is better to test anyonerather than not to test at all.
We’ve created 3 tiers of participants:
·
Tier 1: Customers/Partners –
Ideally, we only test real users from our customers and partners in the field.
We are establishing a charter customer program to make real users readily
available for such testing.
·
Tier 2: Recruited
Participants – When needed, we may supplement our testing with recruited
participants – either paid, gifted or volunteer.
·
Tier 3: Internal Employees –
As a last resort if no other participants are available, we’ll test our ideas
on Sales, Marketing, Business Development, and/or any other customer-facing
groups. This approach still provides valuable feedback and validation, but we
need to keep in mind that results may be slightly skewed.
Planning, Preparation and Facilitation
To prepare for the test, our design team collaborates closely with
product management to define the usability tasks to test and the interview
questions concerning value. We try to concentrate on the key tasks that users
will do most of the time.
How we test users is determined by multiple factors, including
what type of test environments are needed (i.e., Web v. Mobile), as well
as when and where we are testing. Over the
years I’ve tested in-house, remotely (using a video conferencing tool
like Adobe Connect, JoinMe orSkype) and on-site at a
customer’s office. The latter is usually the best case since you can get a feel
for the user’s actual workspace limitations, workflows and peer interactions –
but it is not always possible, so take what you can get.
We also consult with our product team to decide the most efficient
approach, based on the available time and resources needed to:
·
Create a wireframe or build an interactive prototype
·
Present working code in a test environment
·
Use card sorting, paper prototypes and/or taxonomy studies.
When conducting the test, it’s important to make sure to provide
the user with a proper orientation to the environment and ask for permission to
record. It also helps to mention that that they won’t be hurting your feelings
by giving an honest opinion, and to continually remind them to “think aloud” to
provide feedback and context. Watch what they do, and observe both verbal and
non-verbal clues about why they fail or become confused.
The testing environment and equipment (laptop, phone, tablet,
software) may vary, depending on the product type (desktop v. mobile), but
should be a quiet, closed space that can minimally accommodate a tester, a
facilitator and a note taker. Recording audio and video is highly recommended;
using cameras to capture both screen and the user’s face provides emotion and
context to any issues that arise. A laptop equipped with a webcam and screen
recording software is ideal — I typically use Silverback on a MacBook Pro for
this. Remember that face-to-face testing is ideal, but any testing is better
than no testing.
When to Test
Our recommendation is to test users early and often, during all
phases of the product lifecycle. This includes:
Conceptual/Prototyping Stages – This
happens earlier in the project, during the requirements/planning stage
(iteration -1, no code or design specifications). Try to recruit 8-10 testers
to participate in 45-60 minute individual sessions covering multiple features
or a new application. It’s critical to get input from existing customers for
validation and understanding real workflows via wireframes and/or an
interactive prototype. This type of testing usually requires a detailed script
and covers a new product or multiple user stories. In the past, I’ve
successfully used Flash, HTML, Axure and PowerPoint to create
mid-to-high fidelity interactive prototypes. I’ve found that the choice of tool
is less important than its ability to simulate an experience that mimics the
desired end product, or that it can be delivered within the chosen test
platform – and you absolutely must be able to work efficiently with the tool.
Development Stages – During product
development, testing is done during each sprint. Note that the duration of the
sprint isn’t important, as long as at least one round of testing is conducted
per sprint. Usability testing is much lighter and informal during this stage,
so try to recruit about 5 participants for 15-30 minute individual sessions
that focus on a specific feature. You’ll be able to quickly spot trends and
patterns in usage, which will allow you to iterate on your design during the
sprint, if needed. Tests can either use live code (from a QA environment) or a
quick wireframe/mockup if testing items for a future sprint. Allocate a set
amount of hours for usability testing activity in the sprint backlog – to be
burned down – making sure that all UT activities (planning, testing and
analysis) fit within your time estimate.
After testing is completed, we generate a Usability
Observation List that will be shared with the team (via verbal review
at scrum, and entered into our wiki within JIRA).
Product Management will prioritize these results into one of two categories:
Bugs or Enhancement Requests. Bugs are entered into the tracking system (we
use JIRA, but TFS, or even
something as simple as Excelworks
as well) and should be fixed before the end of the sprint. See figure 1 for a
visual representation of our process.
Post Launch – After your product
is launched, be sure to have an internal retrospect with the product/feature
team to discuss what was successful versus what was a problem and consider how
to streamline and improve your UT process. In the past, I’ve used surveys,
email discussions, message boards and field-testing to gather feedback from
end-users. It’s also helpful to have the product team speak directly with
external product champions about adoption rates and to ensure reference-ability
for marketing and future UT sessions. Lastly, but most importantly, reach out
to your charter customers to follow up on the end results now that they are
using features in their daily workflows.
Final Deliverables
Our deliverables vary depending on what type of testing has been
done. For conceptual testing, deliverables typically include a Summary
& Recommendations document that may include a deep dive to
identify what the core root causes were for failures based on actual
observations, conversations and concrete recommendations to improve the user
experience. Recommendations are categorized into Urgent, Important or Nice-to-Have to
help the product team accurately prioritize, and the overall scores, statistics
and notes are presented. This document is uploaded into the wiki, along
with any audio/video recordings to be archived for reference. If you have the
resources to convert the recordings into a transcript, it can be quite helpful
for easily searching and quickly scanning.
Deliverables during sprint testing usually include the Usability
Observation List that quickly identifies the points of failure and
provides recommendations to improve the user experience. These findings are
communicated in the daily scrum and uploaded to the wiki for future reference
(along with any audio/video recordings).
Finally, the post-launch deliverables can include a retrospect
meeting, updating the feature enhancement list based on customer feedback, and
identifying reference-able customers. This last deliverable can lead to
highlighting a customer in a case study, or inviting interested customers to
participate in future UT sessions.
Conclusion
As you can see, user testing is an invaluable part of the agile
methodology that can help you better understand your customer’s needs and make
your products and apps more useful to users. No matter how or when you
test, you’ll reap many benefits such as early problem detection, increased user
satisfaction, reduced support costs and increased efficiency for users. And
you’ll constantly be amazed by the innovative ideas for enhancements and new
features that come directly from your customers.
The bottom line is that if you are developing in agile, then you
should be incorporating some form of usability testing into your iterative
process. It does not need to be a formal or expensive process – it just needs
to capture user feedback in some way so that you can ensure a usable product.
As Admiral Grace Hopper once
said, “One accurate measurement is worth more than a thousand expert opinions.”
