Cheap Tests Beat Forecasts

The Core Insight

Marc Randolph tested the idea that became Netflix for under two dollars. He bought a used Patsy Cline Greatest Hits CD, put it inside a greeting card envelope, and mailed it to Reed Hastings. The disc arrived undamaged in less than twenty-four hours. The whole test ran in under an hour.

The line under that test comes from William Goldman. Nobody knows anything. Heaven's Gate had an Oscar-winning director, an Oscar-winning lead, and a budget of 50 million dollars, and it flopped. The Blair Witch Project had a first-time director, no script, and a budget under 50,000 dollars. It grossed more than 250 million dollars.

Most founders treat founding as a prediction problem. Pick the right idea, write the plan that proves it, raise against the plan. Randolph argues that no business plan survives a collision with a real customer. So the work is to arrange that collision early and cheap.

Randolph converged on Netflix rather than conceiving it. A merger left him and Hastings with about four months of paid idleness. They carpooled over Highway 17, Randolph pitched an idea a day, and Hastings kept delivering the verdict that named the book.

The Framework

The method runs in two stages. A structural filter kills ideas on their unit economics before any test exists. A cheap test then puts the survivor in front of physical reality.

Hastings supplied the filter. Customizing a product per customer never gets easier, because making a dozen takes twelve times the effort of making one. He wanted the effort to sell a dozen to match the effort to sell one, plus repeat purchase. That pointed at an existing physical product to move online.

Everything after the filter is a substitute for forecasting.

• Business plans are near-worthless, so put the idea on a collision course with a customer instead.
• Tell everyone the idea, because telling produces feedback, prior failures, and recruits.
• Test sloppily, because broken links and misspellings do not stop a person who wants the thing.
• Test ten bad ideas rather than engineer one perfect one, because overplanning is procrastination.
• Trust your gut, and then go test it.

He rejects the cliche that every idea has merit. Bad ideas exist, and you cannot tell which ones they are before you run them.

Key Ideas

The Filter Runs Before the Test

Randolph kept a numbered notebook. It held customized baseball bats, personalized shampoo by mail, and custom-blended dog food at number 95. Machine-shaped surfboards matched to size, weight, strength and style sat at number 114. The filter cleared the whole class in weeks.

Toothpaste failed because a tube lasts a month. Videotapes surfaced after Randolph watched Aladdin at 2 a.m. with a crying three-year-old.

Video by mail died on Christina Kish's arithmetic. Tapes cost 75 to 80 dollars, rented for 4 dollars, and cleared about a dollar per rental after postage and handling. A tape had to rent 80 times to break even. Stores turned a tape twenty to twenty-five times a month on 24-hour rentals. By mail, with three days each way, four rentals a month was the optimistic case.

A Two-Dollar Test Can Still Lie

The DVD solved the problem on dimensions rather than content. US players went on sale in test markets on March 1, 1997. Movie rental was an 8 billion dollar a year category, and a disc was light enough for a business envelope and a 32-cent stamp.

Months later Randolph toured the Santa Cruz post office to design the mailer. Local mail was hand-sorted and handed straight to drivers. Everything else went to San Jose for machine sorting. He asked whether a naked CD mailed from anywhere else survives the machines, and the answer was most likely not.

Randolph names his own test a false positive. If Hastings had lived in Los Gatos or Saratoga, the disc breaks and the company never starts. The test was cheap, fast, decisive and wrong. Mailer engineering then ate months and still blew the budget after launch.

Launch Ends the Prediction Phase

Netflix went live at 9:00 a.m. on April 14, 1998, with 925 films in the archive. The team expected fifteen or twenty orders and got 137. They had no traffic measurement at all, so the orders lost to downtime stayed uncounted.

Two servers held for fifteen minutes. Eight desktops from Fry's bought about forty-five more. No error page existed, so they built one in forty-five minutes.

Sales worked and rental did not. June 1998 revenue passed 94,000 dollars, up 50 percent over the month before. Of that, 93,000 dollars was sales and barely 1,000 dollars was rental. Randolph read the sales line as a sugar high, since DVD retail is a commodity business. Rental was hard to run, which was the reason to keep it.

The Pivot Answered a Number

By mid-1999 nothing had worked. They had tried rent-one-get-one-free, giveaways, bundles, and every home page design available to them. The only way to get anyone to rent was to make it free, and free carried a price.

At CES in January 1998 Nickerson at Toshiba agreed to put a flyer in every player box, offering three free rentals. Each redemption cost more than 15 dollars all-in, and only about 5 percent of free-trial users ever rented again. Twenty subsidized freebies at 15 dollars each per convert works out to 300 dollars of acquisition cost per paying customer.

The pivot came from an observation. Tens of thousands of discs sat idle on warehouse shelves, and Randolph asked why Netflix was storing them. Let the customer store them instead. The question that followed was whether to do away with late fees.

The old model assumed an organized renter who plans days ahead, and nobody is that person. Most people decide about ten seconds after seeing something on the new-release rack. A stack already sitting on the television makes selection instant and mood-driven. The weakness inverted into the mechanism.

Three ideas went out together, and Randolph believed in none of them.

• Home Rental Library offered four DVDs at a time for 15.99 dollars a month, kept as long as you like.
• Serialized Delivery shipped the next disc automatically on return, and Randolph named the list the Queue.
• Subscription replaced per-swap pricing with a recurring monthly fee.

Randolph wanted three sequential two-week tests. Hastings refused for lack of time and told him to test all three at once. Every tenth customer clicking redeem on a free-rental coupon landed on a page offering Netflix Marquee. At month's end, absent cancellation, the card took a 15.99 dollar charge.

Day one, 90 percent of the people who clicked the banner handed over card details. Randolph had expected about 20 percent, and the rate held day after day. Sign-up ran four or five times the a la carte rate, and Marquee drove site traffic up 300 percent in three months.

Randolph insisted the trial rolls into a paid month unless the customer cancels, and Hastings called that unethical. Randolph won, and four weeks later customers let the charges through. By February 2000 a la carte was gone and the service was subscription only at 19.99 dollars a month.

Focus Is Amputation

Joy Covey called from Amazon in the summer of 1998. Amazon had more than 600 employees and more than 150 million dollars of revenue. Her offer for Netflix was low eight figures, which Randolph reads as between 14 and 16 million dollars.

They declined without ever taking a vote. On the flight home they decided to kill DVD sales, the only profitable line, because doing both confused customers and Amazon was coming. Hastings said it put all the eggs in one basket. Randolph answered that one basket is the only way to be sure you break none.

The same rule killed the rest. Canadian expansion offered about a 10 percent instant revenue bump. The translation, currency and postage work consumed effort that returned more than 10 percent inside the core. Randolph calls it the Canada Principle, and it killed Canada, the portal story, and later the kiosk business.

The last amputation was his own job. In September 1998 Hastings arrived with a laptop open to a slide reading ACCOMPLISHMENTS. Randolph closed the laptop and made him argue without it. Hastings had lost faith in Randolph leading alone, and proposed himself as CEO with Randolph as president.

Randolph separated two dreams that had fused: the company succeeding, and himself at the helm. He put his own entrepreneurial skill at the 98th percentile and Hastings at the 99.9th. Forty employees now had car payments attached to a dream that was no longer only his. He agreed. One third of the shares Hastings wanted came out of Randolph's own equity.

Fifty Million Was the Asking Price

Netflix filed its S-1 the week the Nasdaq fell 25 percent, the week of April 14, 2000. Deutsche Bank pulled the offering, and the 75 to 80 million dollars under discussion evaporated. Subscription cost the user 19.99 dollars and cost Netflix 4 dollars less on average to provide. Success was the problem: faster growth meant faster burn, because a free trial was paid up front and repaid in dribbles.

The scale gap was total. Netflix was on track for 5 million dollars of revenue in 2000, against Blockbuster aiming at 6 billion. Blockbuster had 60,000 employees, 9,000 stores and nearly twenty million active members, against 350 employees at Netflix.

Hastings chartered a Learjet 35A at roughly 20,000 dollars round trip. He justified it: a company on track to lose at least fifty million dollars absorbs another twenty thousand. Ed Stead asked for a number, Barry McCarthy started on comparables, and Hastings cut him off with fifty million. Randolph watched John Antioco struggling not to laugh.

The layoff came in September 2001. McCarthy wrote 2,000,000 on the whiteboard, the subscribers needed for profitability at then-current overhead. It sat seventy-three weeks away, and the cash ran out first. He erased the 2 and wrote 1.

They cut 40 percent of the company in forty-five minutes. The email went at 10:45 on a Tuesday, and the all-hands ran at 11:00 in the courtyard. Vita Droutman, one of the founding seven, was cut.

The company hit one million subscribers by Christmas 2001, months early. Netflix later passed 150 million subscribers and reached about 150 billion dollars of value. Blockbuster turned down the fifty million dollar price and is down to one store, in Bend, Oregon.

Practical Applications

Write the one assumption your plan cannot survive without. Then price the cheapest physical object that tests it. Randolph's was a disc in an envelope for under two dollars, and it produced an answer in a day.

Run the structural filter first. Ask what happens to the effort per customer when volume goes up ten times. An idea whose effort scales one-for-one with customers fails on arithmetic, and no test repairs that.

Audit the test that told you yes. Randolph toured the post office months later and found that hand-sorted local mail had saved his disc. Name the condition that made your test pass, then check whether it survives at volume.

Ask for money to measure enthusiasm. Randolph answers any declaration of love for the idea with a request for a few thousand dollars, and watches the backpedaling.

When the clock is short, run the tests together. Subscription exists because Hastings refused three sequential two-week tests. Three half-decent ideas combined into a business none of them was alone.

Who This Is For

Founders before their first real test get the most from this book. It covers how to kill ideas on structure, what a test costs, and what to do when a test lies to you.

Skip it for the streaming story. Randolph left in 2003, and the book stops at the DVD-by-mail company and the 2002 IPO, which netted nearly 80 million dollars.

This is a memoir of the first act, written by the founder who left before streaming existed. The Author's Note states that conversations are reconstructed from twenty-year-old memory, so read the dialogue as reconstruction and the figures as printed. Hastings tells parts of it differently, and Randolph spends the book correcting the late-fee origin story while endorsing it as branding. The lesson is also easier to state than to run. A two-dollar disc settles a question in a day. A test that costs a quarter of engineering time gets argued about instead of run.

The Decision

Name the assumption your plan cannot survive without, and write it so someone can prove it wrong. Then build the smallest object that puts it in front of a real person this week.

That object can be a disc in an envelope, a landing page carrying a price, or a phone number that rings your own phone. Budget two dollars and one hour before you budget a quarter.

Mail the disc.