When should an idea that smells like research be a startup?

Link post

Some ideas “smell” like research. The concept is incredibly nebulous and I’m not going to attempt to offer a definition beyond what they’re not: straightforward business ideas or products that one could hire a team of engineers to build. Some researchy ideas fit neatly into academia: that interconnected system of labs, papers, and grants that dominates the research landscape. However, many do not, for one of a thousand reasons. These ideas need a different institutional structure (another nebulous term that is roughly some system of answers to “how is the work organized? How is it incentivized and funded? What does winning look like?”) to gestate within and become something that affects more than just the people working on it.

I don’t have good answers to the general question of “what is the right structure for this idea?” That requires much more, ahem, research. However, I do want to explore the common wisdom (at least among a certain group of people) that researchy ideas should be housed in for-profit Silicon-Valley-style startups. Sometimes being a startup is the right move, but all too often being a startup can kill an otherwise promising idea. When should an idea that smells like research be a startup?

An aside for few notes about language. Apologies for needing to be a bit pseudo-technical – trying to come up with decision frameworks requires drawing distinctions between similar-seeming situations, which in turn requires precision in language and thought.

I’m going to be trying to use the word ‘uncertainty’ in the technical sense of Knightian Uncertainty that carries a lot more meaning than just “we don’t know for sure what will happen.” I use the term unpredictability for that latter situation.
I’m referring to the unit of activity as ‘ideas’ to avoid terms that have different technical meanings to different people like ‘projects,’ ‘programs,’ etc.
I lean heavily on Venkatesh Rao’s take on ‘critical paths,’ which he defines as “a zone where there exists a sensitive dependence of ends on means.” That is, where a critical path exists, you can be pretty sure what to prioritize, what numbers to optimize, and how delays or additional resources will affect your final goal. For example, a SaaS startup almost always knows that, whatever else is going on, consistently increasing monthly recurring revenue will lead to success. Where critical paths do not exist, it’s hard to know which pieces of work or metrics will ultimately lead to success. A global critical path is a situation where you can optimize for whatever your final goal is (even if you don’t know that goal clearly!) while a local critical path is a situation where you’re optimizing for an intermediate goal that may or may not be on the critical path for the final goal.

We can say for certain some things about ideas that smell like research:

To some extent the idea requires attempting to do something, usually discovery or invention-wise, that nobody has done before. It’s impossible to know with certainty how to optimize for an end goal that nobody has reached. In other words, they do not have a global critical path.
The idea will not be able to generate cash flow directly from its main activities for some amount of time.
As a result of #1, the idea will need some kind of money factory.
That money factory (or multiple money factories) will need to put money into the idea for some amount of time before it produces a ‘working’ result — a good business, a technology that other people can use, an acquisition, etc.
There is unpredictability about the timescale to get to a ‘working’ result. More uncertainty about critical paths means more unpredictability on the timeline, or even the timescale.
In large part because of the timescale unpredictability, there is unpredictability about the amount of money the idea needs to get to that ‘working’ result. Time is money. Hence, more critical path unpredictability → more time unpredictability → more cost unpredictability.
The time and money are needed to address technology risk (the risk that the technology won’t hit a performance metric) and technology uncertainty (not knowing what the metric actually is).
Researchy ideas have some amount of uncertainty about potential outcomes: what the form of the output will be, whether there is a market for that output, whether the output will be of a form that can capture value from that market, whether the program will organizationally be able to be a business that can do that value capture.

These attributes immediately set constraints on institutional structures. If the idea is housed in a new profit-seeking organization, the combination of points #1, #2, and #4 mean that except in rare cases, the organization needs to be a high-growth startup. The growth imperative of a startup becomes necessary because in order to provide competitive time-discounted returns on the money that the company needs to do the work over a potentially long timescale, it needs to target massive value. Being a Grahamian startup then subjects the organization to a whole set of constraints and incentives.

Outliers dominate this type of conversation, so I’m going to call out the elephants in the room:^[1]

SpaceX (it’s even debatable whether this counts as an idea that smelled like research — at first they weren’t even trying to do something nobody had done before — there was a pretty clear but very hard critical path).
Genentech
A number of therapeutic companies
OpenAI
DeepMind

The reasons each of these organizations reached stand-out status are idiosyncratic and well-analyzed, so I won’t go into them here. However, arguments that an idea should be a startup based on analogies to these successes are unconvincing. You can tell a story in which almost^[2] any idea could succeed as a startup. The nature of research and Knightian Uncertainty means that it’s impossible to say with certainty “this project shouldn’t be carried forward by a new for-profit organization.” It’s possible to flip all heads in a row on a large-dimensional, non-stationary coin and pull off a successful researchy startup. The question is whether, if you care about the ultimate success of the idea, a startup is the structure that gives it the best chance.

The fact that the list of standouts is so short compared to the list of researchy startups that have failed to deliver on their promises seems indicative. I don’t have data, but I suspect the ‘failure rate’ of researchy startups is even higher than the average for startups.^[3] The list of outliers also points to the fact that the question of whether a research idea should be a startup might be very discipline dependent. SpaceX is the only non-therapeutic, non AI company on the list.

“Ah, but what if you create a for-profit organization without the demands of a high growth startup?” It could get away with targeting lower growth and/or eventual value if funding comes from non-dilutive grants or ‘friendly’ investment where the investors don’t expect to make as much return on their money as they could elsewhere. However, both of these options come with their own downsides. These downsides include needing to keep investors happy because ‘friendly’ investment generally means ‘not market competitive,’ so investors become much less replaceable. Too much reliance on a few investors or grant-givers creates the danger that the output becomes ‘things that make investors or grant-givers happy’ rather than things that maximize the potential of the idea, or even make the most money in the long run. These warped outputs can be everything from brilliant demos that go nowhere to extremely justifiable projects that shy away from the uncertainty inherent in doing something truly new.

While there are no definitive criteria that can say whether a researchy idea should be a startup, the more the unpredictability in each of the steps smells like uncertainty instead of risk, the less it’s a good idea for the project to be carried forward by a new for-profit organization. I’ve come up with a rough decision tree that hints at the amount of uncertainty in an idea. These questions are ordered in terms of importance, with “no” or “not very” suggesting (but not asserting) that a startup might ultimately hamstring the ideas ultimate success:Can the desired output be sold?

Is there a global critical path? How clear is it?
Is there an objective ‘measure’ of what ‘working’ means? How smooth is it?
Is there a bounded amount of market uncertainty?
Is the technology a modular product (ie. does it slot into existing systems)?
Would investors consider a low return on investment (ROI)^[4] acceptable?
Would investors be willing to wait a long and unpredictable amount of time to get that return?
Are investors ok with an illegible process without clear milestones?

Digging into each of these points:

Can the desired output be sold or unlock a product?

If you are setting out to create something that could never be sold, the idea probably should not be a startup. There are examples of accidental but profitable discoveries that happened in the course of pursuing something out of motivations besides profit — discovering a heart drug while studying tree frog poison in the Amazon for example — but you might as well just buy stocks chosen by a random number generator.

Is there a global critical path? How clear is it?

The less clear a critical path is, the more you need to run a “fat” process without clear justification. (Justification requires explaining how it fits into a bigger picture which requires a global critical path). Without a global critical path, “progress” is nothing but a narrative and the fat process makes it look like you’re doing a lot of dicking around. As a result, investors will get fed up at some point and either force the research into a premature product or shut the whole thing down.

Of course, most researchy ideas don’t have a clear critical path – many successes have a charismatic leader who convinces everybody else that there is a critical path long enough to find it. This approach is strapping a ticking time bomb to the idea and betting that you can keep it smelling like money long enough to find a critical path. It can work, but the fallout of failure can also set an idea back decades. (See: AI winters, climate tech, etc)

Is there a smooth objective measure of what ‘working’ means?

One way to look at uncertainty is that there’s no definition of what ‘works’ means: this could be in terms of the technology itself (technology uncertainty) or in terms of how it becomes a good business (market uncertainty).^[5]

Creating new technology will almost always involve some uncertainty. Technology’s modular/combinatorial nature^[6] means that there will be uncertainty at some level of the hierarchy. In many cases, an uncertain subsystem can be easily solved or replaced. But sometimes that uncertainty will propagate up to the top-level technology if it is an irreplaceable subsystem. My extremely hand-wavy intuition is that you cannot have massive uncertainty in subsystems without it propagating upwards.

Even if you have a clear measure of what ‘working’ means and a clear metric to improve, the way in which that metric changes matters. A smooth metric means that incremental changes can cumulatively move the needle, which means that you can show nice graphs trending towards success. However, some researchy ideas involve situations where the needle doesn’t move at all until it does a massive jump (ie. They depend on breakthroughs). The former situations lend themselves to startups while the latter do not.

Is there a bounded amount of market uncertainty?

Situations where you’re guaranteed a massive market if you can hit a clearly defined target lend themselves to researchy startups. Therapeutics are a prime example here: if you can create a cancer drug that passes FDA approval, you are basically guaranteed a multi-billion dollar market. In the past, mainframe computers had a bit of this flavor — if you could hit a clear performance metric, you had a guaranteed market.

A big reason that market uncertainty can kill researchy startups is less about investment and more about the fact that there is a tradeoff between an organizational culture that is good at addressing market uncertainty and one that is good at addressing technological uncertainty. At some point a researchy startup needs to do a dramatic gear shift into growth and product-market fit mode. This transition often either prematurely kills the research potential or the company dies because it’s being run by people with a research mindset.

Is the technology a modular product (ie. does it slot into existing systems)?

Startups are good at point changes. If there is a component in a system that can be improved or swapped out in order to make the system cheaper/more profitable/more efficient, startups are an excellent mechanism for making that happen. The same goes for projects in industrial research labs — if you can, say, save the company a ton of money by swapping out the sheathing on telephone cables, the path to success is straightforward. However, to improve, many systems need many simultaneous changes (or to be swapped out for a new process entirely).^[7]

Additionally, many systems must take performance hits to get out of local optima and people are rarely willing to pay to decrease system performance.^[8]

Would investors consider a low ROI acceptable?

A lot of research involves dicking around, or “fat” processes.^[9] If you need to target a high ROI from day one, you need to be as efficient as possible. ROI is all about efficiency, and efficiency biases systems towards false negatives. For a startup, spending $1B creating $10B of value is preferable to spending $10B creating $50B of value. As a result, there will just be less slack^[10] in the system, cutting off lines of work that can’t be justified up front (especially if they require resources to evaluate). Ironically, the pressure towards justifiable spending also occurs with friendly investment or donations. A fundamental tension in doing researchy work is that if you aren’t being judged on output, people want you to spend money responsibly, and if you are targeting a relative return on investment, you want to spend money responsibly. But often powerful research requires being a bit irresponsible.

Would investors be willing to wait a long and unpredictable amount of time to get that return?

A key characteristic of research is that, regardless of how long it takes, that amount of time is unpredictable.^[11] Combined with the fact that progress is often not smooth, it takes a constitution of steel for investors not to start getting antsy at some point. People are patient until they’re not. The pressure to show some progress towards a goal makes sense from a business sense, but can be at odds with developing a powerful technology. Premature specialization is a common failure mode in researchy startups.

Are investors ok with an illegible process without clear milestones?

In addition to unpredictable timelines, a lot of technical research involves constantly shifting targets. You can set milestones (and it is often useful to do so), but they need to be incredibly fluid. Unlike business progress, which can be boiled down to a few core metrics like customers or revenue regardless of how much pivoting is happening, research progress is much harder to boil down to a legible metric.

So when should a researchy idea be a startup?

At the end of the day, it’s impossible to create hard and fast rules about whether an idea should be a startup. However, the worse the idea does on the questions above, the more skeptical you should be that a startup is the right home for it. But the fact of the matter is that sometimes, a startup is the best way to push an idea forward, despite the risks. In some cases, it’s a choice between a startup and nothing. That’s a bad choice. To get up on my soapbox for a moment: new institutional structures with different constraints, like PARPA and many others, can break that dichotomy. Giving ideas more options for how to grow will enable more diverse ideas to blossom, hopefully creating a richer more textured world.

Thanks to Matt Clifford, Sam Arbesman, and Niamh Gavin for reading drafts of this piece.

^
This list is obviously incomplete, but it’s the right order of magnitude.
^
Perhaps a hard criterion might be if the desired output could never be sold. (Though even then someone could point to profitable but unintended discoveries as a reason for researchy work to be a startup).
^
With all things held constant in a way that is probably impossible.
^
ROI is calculated in different ways but roughly I’m using it as shorthand for the ratio of money out to money in.
^
Note that technology/market uncertainty is different from technology or market risk. Risk means that you know what winning looks like but there are a lot of reasons why you wouldn’t get there. Technology risk sounds like “there’s a chance that we can’t get the efficiency of this process above 35% but the back of the envelope says there’s nothing that should prevent that and we have a development plan to get there.” More generally, technology risk implies a metric and a vague sense of a probability distribution over that metric, while uncertainty just goes 🤷‍♂️.
^
See https://www.santafe.edu/research/results/working-papers/the-logic-of-invention.
^
See https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3470660.
^
This is a major contributor to The Valley of Death phenomena.
^
See GUTS again.
^
See https://slatestarcodex.com/2020/05/12/studies-on-slack/
^
See https://www.goodreads.com/book/show/29954040-cycles-of-invention-and-discovery