Via @UserVoice: Everyone’s Customers Are Wrong & Their Data Is Lying

I’ve written before about the dangers of bad experiment design, this presentation from everyone’s favorite customer support tool UserVoice nails it on the head:

Whether you’re in product management, customer service, community management, or even an entrepreneur, I highly suggest taking a moment to absorb these stories.

Everyone’s Customers Are Wrong and Their Data Is Lying from Evan Hamilton

Quote: What’s Actually Wrong with Yahoo!’s Purchase of Summly

Scathing critique from a well-informed source.. I don’t know what Summly had behind the Kimono, but I will say this: in startups -unlike in academia- often the product that is out there visible to the public is already outdated internally, with learnings from its use already being incorporated into a new, better build. One explanation I can see is that Summly was able to parlay the data from its ~1 million users into some kind of IP that Yahoo! believes it can develop and defend. That, or Yahoo!’s desperate enough for talent to pay a cool $15 million per engineer. 

via @el33th4xor

But it’s critical to keep tabs on the ratio known as “glue versus thought.” Sure, both imply progress and both are necessary. But the former is eminently mundane, replaceable, and outsource-able. The latter is typically what gives a company its edge, what is generally regarded as a competitive advantage.

So, what is Yahoo signaling to the world? “We value glue more than thought.”

If Summly is an innovative company worth purchasing, I have some news for Yahoo: my AI colleagues have tricks up their sleeves that will blow your minds!

On experiment design (and animal intelligence)

Much has been written about the lean startup movement, which advocates data-driven decision-making within a principled, customer-driven framework to both picking a startup idea and growing the fledgling company through its initial lifecycle. 

There’s no question that overall the philosophy is a strong net positive to the ecosystem, but data can also lead to some bone-headed conclusions. If you ask the wrong question, data doesn’t say “sorry try again” - it often gives you back a perfectly reasonable but utterly incorrect answer. VC is just as prone to this as any startup.

It’s surprisingly rare to see real vision borne out of deep domain knowledge paired with good old common sense (in both VC and startups!) - but those are the guys you want to hang out with. Not enough attention is paid to asking the right questions, gathering the right data, even in choosing to play in the right ballpark. 

I don’t have all the answers. What I am doing (with @AxiomZenTeam) is running experiments. Lots of them. And bit by bit, we learn and get better. What we try to do is continually challenge ourselves to ask our questions in different ways, to upend assumptions and get back to basics.

I feel like my biology background helps me have a more critical eye on experiment design, which is why I chuckled when reading the recent (absolutely fascinating) article from WSJ on animal intelligence. Namely, that we may have been underestimating our furry friends’ cognitive abilities for centuries precisely because of terrible experiment design by scientists that were likely too smart for their own good. 

It’s a great article, so I’ll just post an excerpt and you should read the whole thing:

Experiments with animals have long been handicapped by our anthropocentric attitude: We often test them in ways that work fine with humans but not so well with other species. Scientists are now finally meeting animals on their own terms instead of treating them like furry (or feathery) humans, and this shift is fundamentally reshaping our understanding.

Elephants are a perfect example. For years, scientists believed them incapable of using tools. At most, an elephant might pick up a stick to scratch its itchy behind. In earlier studies, the pachyderms were offered a long stick while food was placed outside their reach to see if they would use the stick to retrieve it. This setup worked well with primates, but elephants left the stick alone. From this, researchers concluded that the elephants didn’t understand the problem. It occurred to no one that perhaps we, the investigators, didn’t understand the elephants.

Think about the test from the animal’s perspective. Unlike the primate hand, the elephant’s grasping organ is also its nose. Elephants use their trunks not only to reach food but also to sniff and touch it. With their unparalleled sense of smell, the animals know exactly what they are going for. Vision is secondary.

But as soon as an elephant picks up a stick, its nasal passages are blocked. Even when the stick is close to the food, it impedes feeling and smelling. It is like sending a blindfolded child on an Easter egg hunt.

More experiment design #fails:

Another failed experiment with elephants involved the mirror test—a classic evaluation of whether an animal recognizes its own reflection. In the early going, scientists placed a mirror on the ground outside the elephant’s cage, but the mirror was (unsurprisingly) much smaller than the largest of land animals. All that the elephant could possibly see was four legs behind two layers of bars (since the mirror doubled them). When the animal received a mark on its body visible only with the assistance of the mirror, it failed to notice or touch the mark. The verdict was that the species lacked self-awareness.

(hint: elephants can in fact recognize themselves)

and more:

A similar experimental problem was behind the mistaken belief, prevalent until two decades ago, that our species has a unique system of facial recognition, since we are so much better at identifying faces than any other primate. Other primates had been tested, but they had been tested on human faces—based on the assumption that ours are the easiest to tell apart.

that one’s a real face plant. 

Read the whole WSJ article here. 

I can’t help but feel that none of these scientists were deep domain experts, rather that they looked at animals as foreign objects to be studied. My academic advisor was Robert Sapolsky, among the world’s foremost experts in neurobiology (and really a fantastic guy). When he talked about baboon behavioral psychology, he knew his subject matter deeply. He had lived in Africa, immersed himself in the baboons’ day to day lives, their social structure and -dare i say it- their culture. He would not have given them human faces to tell apart. 

What does this have to do with startups? I’m personally disappointed in the number of smart people making bone headed mistakes. What mistakes? Groupon. Trainwreck most of us saw coming for years before the IPO. Selling shitty goods from shitty merchants at 1/4 their traditional value to shitty customers. *But the data* they said, and the growth! Did anyone bother to actually keep using the product? 

not just Groupon, lots of companies. don’t be one of them. I’ll leave you with one of my favorite Sapolsky quotes:

I am a reasonably emotional person, and I see no reason why that’s incompatible with being a scientist. Even if we learn about how everything works, that doesn’t mean anything at all. You can reduce how an impala leaps to a bunch of biomechanical equations. You can turn Bach into contrapuntal equations, and that doesn’t reduce in the slightest our capacity to be moved by a gazelle leaping or Bach thundering. There is no reason to be less moved by nature around us simply because it’s revealed to have more layers of complexity than we first observed.

The more important reason why people shouldn’t be afraid is, we’re never going to inadvertently go and explain everything. We may learn everything about something, and we may learn something about everything, but we’re never going to learn everything about everything. When you study science, and especially these realms of the biology of what makes us human, what’s clear is that every time you find out something, that brings up ten new questions, and half of those are better questions than you started with.

Web businesses and user trust

I’ve avoided my own diatribe about the @googlereader debacle..  here @calbucci gets it largely right in a biting and accurate post. His third point is one we’ve thought about at length for our own products:

Third, and lastly, Google is sending a strong signal to the market that it will have no mercy of killing whatever product it doesn’t think it’s going well. It just told users, professionals and enterprises that we all should not use any product from Google if requires long term commitment (not business-type commitment, but data and emotional commitment) unless we have a sense that’s going to succeed. Now, I have to be in the business of evaluating Google’s product long-term viability before I can commit. Sure, I’m pretty sure Gmail is not going away, but what about Google Talk and Google Wallet? What about Picassa or YouTube?

One solution we like is easy data export from day one. Turns out when customers know they can easily get their data out, they’re much more likely to come play in your sandbox in the first place.