A story about guerrilla usability testing, melting chocolate, a satanic cult, and stakeholders always being right.
We're in the heart of Manhattan, working on a project within a 10 person team.
Things are not going well.
For the past few months, we've been contributing remotely — as design consultants, but only once we arrived in NY (to help with the "final touches"), it became clear how fucked up this whole thing actually is. The team is completely misaligned and priorities are changing every hour. No one is even mentioning customer needs, and we don't really know what is it that we're building, but our sales team is in full throttle — selling the shit out of... well nothing.
No decisions are being made. We're stuck! Torn between all these potential directions we could be taking.
So we try to take the initiative — "Guys, listen - we've got all these design concepts lying around. Let's show it to people — do a couple of usability sessions — It will help us make some calls."
"Usability?" - everyone looks at us in shock! "There's no money for usability sessions!"
"Oh come on! Just a small guerrilla test. We can do it after working hours!"
"After working hours you say?" — And just like that, we get a green light (in true capitalism there's hardly anything harder than rejecting a fool willing to do free work).
Little do they know that this is all part of our plan — our Trojan horse. Cause once they see how eye-opening the results of a usability study can be, they will beg us to do more!
We roll up our sleeves, and in a day filled with other "more important" distractions, we manage to prepare a simple AB type of test. It's all very improvised, but this is our only chance.
With our official working hours over, It's time for our real work to begin. Armed with one iPad, a phone, and a bag of Reese's, we hit the streets of Manhattan looking for our first victim.
It takes us good 20 minutes to collect our courage and make the first approach.
"Sorry... do you have a minute for a quick question? We're doing this study and we could really use your help."
"Umm, sure..." — a girl in her 20s responds.
I shove my iPad in her face, and she quickly goes through our prototypes — giving us the exact kind of feedback we were hoping to get.
We say our thanks, we give her some candy, and we part our ways with smiles on our faces.
We did forget to record the audio though, but the transcript will do.
With our courage on the rise, we quickly move on to our next victim. This time it's a woman in her 30s. Super nice. We go through the same scenario, and she even lets us record.
"Hah this is easy!" — I say out loud... and I regret it immediately. The next 5 people we approach ignore us before we even get to say hello, shattering our confidence into pieces. And just as we're about to lose faith in humanity, we come by this girl who turns out to be the friendliest person on earth — and we manage to get a great interview.
Our skin slowly starts thickening and the rejections stop hurting (so much). Within the first hour, we end up with 5 recordings. And the feedback we are getting? Fantastic! Addressing major points of our team's misalignment, and filled with great insights.
The second hour begins and the Reese's are starting to melt. We're getting tired, and maybe it's our smell, but all over again, nobody want's to talk to us. And the rejections are getting harsh. One of the worst ones still haunts me to this day — with this lady screaming her lungs out — "Don't talk to them!!! They're trying to scam you!" — and the entire street looking at us like we're some fanatics begging for donations for our satanic cult (which we usually do on Fridays only).
But we push through, knowing that our luck will turn at any moment. And it does!
After 3 hours, we are exhausted. But we have what we came for! Recorded interviews with almost 20 people roughly fitting our target audience.
We spend the entire night marking the best bits and highlighting patterns — putting it all together into an impressive sequence of slides that points in a clear direction.
The next morning.
7 am team meeting, and our presentation begins. The group's grim mood starts brightening up. Halfway through, the excitement is in the air.
My partner reveals the winning concept while playing the best quotes from our interviews, and for the first time the entire team sings in harmony — "I knew it! This was my favorite from the beginning!"
And I feel great! Not only we will finally be able to start making some progress, but after this, we are guaranteed to get more time to do real usability studies. It was all worth it!
There's only one opinion left to be heard — The owner’s opinion.
He's not in the room with the rest of us, but he joined via video call, and finally, he speaks — "Guys, this is great! So many fantastic insights. But..." — the owner pauses, and my heart skips a beat. "...I have to say, I'm not the biggest fan of the winning concept".
The time slows as he points his finger to one of the other designs. Without even looking at the numbers he says — "I like that one!"
"But Nick, have you seen...? It's... I mean..."
"Yeah, I just don't like it" — Nick cuts me off. "I have a feeling about this other one. Let's go with that"
"Great work team!" — the meeting ends.
The room goes silent, and then slowly everyone starts circling around the design that Nick chose. "Hm... I think he's right, there is something about this one!" — Anna says. "Yeah it does seem more inviting!" — John says. "More accessible!" — Brian from sales adds.
I stand there in disbelief, thinking about all those people we interviewed the other day. I look at the numbers we've spent collecting all night, and everything points the same way. Then I look at Nick's choice for a few minutes, and I finally realize what's so special about it — it's the only one that wasn't even noticed by the folks we interviewed.
Soon after, we left the project. It was clear that this was not the environment that could give birth to anything meaningful. It was sure as hell not the way to build successful products. We realized that, as long as random decisions of those in power are being valued more than hard work, as long as the CEO's "feeling" has more weight than data... oh who am I kidding?
The guy actually had a feeling, and few months after we left, his company became a multi-million dollar making machine.