Encouraging Negative Feedback During User Testing

Have you ever sat in on a user testing session, watching a user really struggle with the task at hand, only to have them tell you at the end that everything was easy and straight forward? How do you encourage these participants to be negative? I’ve discovered a few techniques that might be able to help.

A colleague and I were running some usability tests on a registration form last week. The form was far from perfect and we could see our participants struggling through the process. However, when it came to befriending them at the end of the tests, one by one they would respond positively, informing us that the form was very simple and straight forward to use. Shocked and confused, we started to discuss different techniques to coax some negativity out of them.

The first thing we did was try to understand the reasons why our participants might feel pushed to be positive. Understanding the situation we were putting them in highlighted three areas which might lead them away from negative feedback.

People don’t want to insult someone’s work

Firstly, we realized asking someone to critique a stranger’s work is especially difficult if the user thinks the stranger who designed it is sitting right next to them. So, we started beginning our sessions by informing the participants we had nothing to do with the company who built the form being tested. This puts the users at ease by assuring them that we have no personal attachment to the work being tested and any criticism would not offend us at all.

We’re not testing them

The next point we noted was that some of the less confident people being tested seemed very apologetic when making “mistakes.” It suddenly occurred to us that our participants could be thinking that the troubles they were encountering was down to their human error and not attributing them to the poor design of the form. To solve this issue, we clearly stated at the start of each session that we were testing the system, not them, and that we knew the system had issues. We expected them to encounter problems along the way. Giving them these pieces of information really seemed to set some of our testers free, allowing them to blame every mistake or confusion on the system.

Users focus on the end goal

In a forced situation, where you are paying someone to complete a pre-arranged process, participants are more likely to focus on completing the task at hand. It seemed that our testers didn’t remember the small issues along the way as long as they completed what was being asked of them. To combat this, we ran the users back through the form, one stage at a time, and quizzed them on issues that we noted during the initial task. This worked to remind the users of all the little problems they had encountered and allowed them to take us through their thought processes, identifying exactly what it was that caused it to become a problem.

You’re the expert

At the end of the day, the user can only tell you so much. However, the things they can show you are limitless. At the end of the session, it’s a good idea to ask the participants how they would improve things because it gives them a greater feeling that they are helping you out and providing value. After all, if they could give you all the answers, it would be them doing the testing. The main role of the participants is to highlight all of the problems with the process, and it doesn’t matter if at the end of the session they walk away telling you the form was a breeze and they loved it, as long as you have been able to note all of the areas where they stumbled.

With this in mind, it’s always a good idea to have a second person in on a user testing session, making notes on everything the user is doing. Or if you have the budget and facilities, record/stream the session to be reviewed at a later date. In this particular set of tests, I sat a few meters behind the participant and facilitator, watching and making notes on everything that was going on. It’s a good idea to ask the user to think out loud when completing the task because it gives the person taking notes extra information to interpret.

Having someone there taking notes means that it really doesn’t matter what the users final opinion is of the process. As long as you have all the problems that tripped them up written down, you have what you came for.

To summarize

• Assure the participants that you did not design the process being tested.
• Inform the individuals that it’s the system being tested and not them.
• Run them back through the process step-by-step and bring up any issues that you noted.
• Have someone else on hand to note everything the user is doing.
• At then end of the day, it’s the problems you want to find out, not what your participants felt about the process.

Your thoughts

Those are just a few very small things that you can do to try to encourage more feedback from your users. What techniques do you use to draw out feedback from your user testing sessions? I’d love to hear them.

Further reading

• Microsoft desirability toolkit
• Rocket Surgery Made Easy
• Practical Usability Testing
• Quick Turnaround Usability Testing, Part II