Product Thinking Playbook – Wizard of Oz Testing
July 14, 2022
Estimated Reading Time: 5 minutes
Continuous discovery and product testing are precursors to building better products. As a rule, the sooner you start validating or invalidating your ideas, the faster you can begin to mitigate risk and save time and money while identifying new opportunities. And while there are numerous ways to test products and gain user insights, few are as effective at simulating the intended experience as a prototype.
Undoubtedly, the contextual data and user responses you can elicit from prototyping are second to none. Unfortunately, the challenge is that it inherently requires significant time and money, primarily the two things you’re hoping to mitigate through testing in the first place.
But wouldn’t it be nice if you could test and validate an idea, and assess desirability and usability without actually investing the time and cost to develop the product or features?
You can – with Wizard of Oz Testing. Heel clicking optional
What is Wizard of Oz Testing?
Like its namesake, Wizard of Oz (WoZ) Testing is a prototyping technique where participants interact with a simulated product experience that they believe to be functioning and autonomous. Much like the man behind the curtain was the “wizard,” to imitate an immersive experience, this technique relies on an unseen operator completing the more complex or responsive tasks that haven’t been developed yet.
Why would product teams do it?
When product teams can simulate experiences with users, they can more accurately and thoroughly validate critical hypotheses related to the desirability and usability of product concepts without having to commit time and resources to build the product or feature.
WoZ is most useful when the product needs to process incoming contextual data (e.g. the user’s time or location), manual inputs that impact the flow, and/or output custom responses (e.g. responding to a request). Voice experiences are a common candidate for this method, as it would be difficult for participants to imagine the value of a voice experience if faced with unsuccessful interactions. This allows you to meet the participant’s baseline expectations and ensure more fruitful data is obtained.
When should product teams use it?
Prototyping: Typically designed for testing with participants, this technique can also be used to demonstrate a product idea. Testing is conducted to understand the desirability better and/or usability of an experience before building. Desirability is the primary motivator for conducting Wizard of Oz Testing. Usability testing using Wizard of Oz prototypes should be focused on identifying high-level principles for how users want to interact with the product such as validating the type of interaction that should be used.
Concept Evaluation: The need to simulate, evaluate, and prioritize a list of potential features is of course valuable, but also, quite costly. When assessing the desirability of one or multiple product concepts or critical features, WoZ can be incredibly helpful to mitigate that cost and still get the data you want.
Demand Validation: Testing a product idea in the participant’s context/environment helps determine problem/solution fit while amassing qualitative data to help validate or invalidate your desirability hypothesis.
Who is required?
How do I do it? (Best Practices)
- Let the fidelity of your prototype be determined by the research objectives and build feasibility
Low-fidelity WoZ Prototypes typically are perceived as early-stage, and most of the functionality might not be present; the participant should expect some imperfections across the experience. Only feedback on the concept, not the design, should be invited. The intent isn’t to be used to assess usability.
High-fidelity WoZ Prototypes require considerable time and effort, impacting the overall timeline but providing a better experience, increasing the chances of the participant interacting naturally with it. In addition, testing with a high-fidelity prototype can yield more feedback related to usability and design decisions. The flip side of using high-fidelity prototypes is that participants may hesitate to give critical feedback on the idea at a high level if it is perceived that a great deal of time and effort has already been invested.
- When it comes to Wizard of Oz Testing, test the test!
Due to the nature of Wizard of Oz Testing, it is critical to pilot these testing sessions to ensure that the prototype and control room is functioning as expected. Identify risks or problem areas where prototypes or participants might get stuck, and make sure you have a backup plan for getting back on track.
- Don’t lie; just don’t disclose everything
Wizard of Oz Testing is not about lying to participants that the product they’re testing is fully functional; it would be unethical to lie outright about the experience in which someone is participating. Instead, the researcher presents limited information and relies on participants to make assumptions about the product they are interacting with. If a participant asks questions about the prototype, use your best judgment regarding how much to reveal, but it is highly discouraged to fabricate information explicitly.
The Product Thinking Playbook is filled with tactics and techniques that help product teams build better products. Click here to download your copy of the complete playbook, and stay tuned as we share more from it in the coming weeks.
Subscribe to Our Newsletter
Join the Thoughtworks newsletter list to receive curated content that exemplifies our Product thinking approach.
Thu Dec 1
Global Day of Coderetreat Toronto 2022
Earlier this month, we hosted Global Day of Coderetreat (GDCR) at our Toronto office. After a three-year hiatus, we wanted to re-awaken some of the community enthusiasm when we hosted the event for the first time back in 2019. This year we planned for a larger event which turned out to be a really good thing as we nearly doubled our attendance. And in case you were wodering, this is how we did it.
Wed Nov 9
You’re Wrong & Don’t Know It: Process Biases
Process biases occur when you process information based on cognitive factors instead of concrete evidence, skewing your perception of reality, even if all the pertinent and necessary data is right in front of you. And in our third installment of You’re Wrong & Don’t Know It, discover some of the different types of process biases, their impact, and most importantly, how they can be avoided.