Nobody on Apple's task force judged a product from a spec sheet. They took it home and used it. The 1-week version for your line... and the "3 percent" answer it exists to catch.

Nobody on the Stop Crap Products Task Force graded a product from a spec sheet. Each member took one home and used it.

Then they came back and said what happened. If the answer was crap, the instruction went out: pull every unit off the shelf.

When did you last use your own product the way a stranger does... from the ad to the first 10 minutes, without the login you've had for 3 years? Most founders judge their product from a dashboard, and a dashboard can't tell you the manual is 30 pages long. "It's selling a bit" passes for a reason to keep something. So does a clean spec sheet. The Task Force accepted neither.

Take one home

Jay Elliot lays out the method in a few lines of Leading Apple with Steve Jobs. The engineers presented, which is Part 2. Then "members of the task force would each take one of the products, try it out, and come back with their evaluation." If it was determined to be crap, every unit came off the shelf.

That's the whole evaluation. Not a scorecard. A person, a product, and what happened when they used it.

The standard behind it was Steve's, and Jay quotes a version of it that Steve gave the New York Times years later, reworded slightly, he says, so it would be suitable for framing: the starting point isn't a chip or a design. The starting point is the question, "What's the user experience?"

Steve ran that test on himself before he ran it on anyone else. Jay's description is that Steve built products he wanted to use, and wouldn't call one ready until it met his own standard. He was, in Jay's phrase, his own ultimate consumer, asking at every step what was still missing to make this a product he'd want to own.

"We got 3 percent"

This is what the test is built to catch.

After the iPod, SanDisk, a company known for storage, shipped an MP3 player. Asked some time later why, the CEO gave an answer Jay records as along the lines of, "Well, we got 3 percent market share."

Jay's response in the book is the question SanDisk didn't ask: how can we give buyers a great mobile music player? Share isn't a reason for a product to exist. It's a description of what happened after it did. A reason is what the product does for the person holding it, and if nobody in the company can say what that is, 3 percent is what the product is worth.

You'll recognize the SanDisk answer when it shows up in your own meetings. It sounds like "it's bringing in some revenue." It sounds like "a few customers would be upset."

The test cut both ways

The Macintosh wouldn't have passed it on day one, and Jay says so.

Steve had decided the Mac didn't need a hard drive, partly because the price he'd set early wouldn't cover one, and that price was part of the dream of a machine nearly anyone could afford. The result was the disk-swap problem. Jay remembers an Apple executive, Betsy Pace, writing a last-minute speech on one of the first Macs the night before she gave it. Every couple of paragraphs the machine needed the program disk, so her text disk popped out and the screen asked for the other one. A few paragraphs later it happened again. All night.

"He had to face that he had been ignoring the realities," Jay writes. He calls it a lesson Steve learned well, and says that from then on Steve was more open to input from everyone around him when he was forming a vision for a product.

Jay got the same lesson himself, decades later, and he puts it in the book without flinching. In 2012 he launched Data Tunnel, a tool for moving very large files fast. "In the design of the product, instead of keeping the user in mind, I listened to my technology experts." They built an interface any technology guru could understand, for a product that wasn't for gurus. Buyers pushed back hard. "They were right," Jay writes. "The user interface was terrible." They stopped and redesigned it.

The man who ran the Task Force failed his own test the moment he judged from the expert's chair.

What Jay doesn't claim

Jay doesn't say how long members used each product. "A week" below is this article's version, not Apple's. The SanDisk quote is "along the lines of," which is exactly how he gives it. And the New York Times line is his rewording of Steve's remark, which he says himself.

The turn

You can't grade a product from a description of it. The verdict comes from use, by someone who can act on it... and it has to be allowed to come back "crap."

A dashboard tells you what happened after people found the product. Use tells you what happened to them inside it. Every honest take-home test carries a risk, which is that your own product fails it, the way the first Mac would have and the way Data Tunnel did. That isn't a flaw in the test. It's the only reason to run it.

Day 2 of the free course is this test pointed at your own answers: whether you guessed on question 3, what do they actually need to be successful, and how to check it with 3 real users. Start the 5-day course →

Run it on your product

1. Assign every item on your Part 2 page to one of your three. Nobody gets the product they own.
2. Use it the way a stranger would, this week. The ad, the page, the checkout, the first 10 minutes, the help doc. A new email address, and none of the shortcuts you know.
3. 1 page back: what it promised, what happened, keep / fix / pull. 3 words for the verdict, not a paragraph.
4. If nobody on the team can stand to use one of your products for a week, that's the verdict. Don't argue with it.

What this doesn't do: it finds the crap. It doesn't tell you what the user needed instead. That's question 3, and it's the one most people guess on.