The scariest thing for AI startups: Mistaking noise for signals | Kevin Scott, CTO of Microsoft
December 19, 2025, San Francisco.
During a conversation at South Park Commons in the Silicon Valley startup community, Kevin Scott, the CTO of Microsoft, talked about career turning points, cooperation with OpenAI, and why the more valuable questions in the AI era are always overlooked. The entire conversation lasted 56 minutes, and the topics ranged from startup trial - and - error to open - source and closed - source.
His most crucial statement was:
What AI startups fear most is not technological backwardness, but mistaking "noise" for signals.
What is noise?
Information that seems like a positive signal but has no value and no relation to what you're doing. Media hype, investors' interest, and trendy tech terms are easy to obtain and quantify, but they may be leading you in the wrong direction.
This article covers four things:
First, how Kevin learned to identify the right signals.
Second, what kind of environment today's entrepreneurs are facing.
Third, why ChatGPT was successful.
Fourth, how to identify the overlooked real signals.
Section 1 | Abandoning interesting technologies and choosing worthwhile things
Kevin Scott originally wanted to be a university professor. During his doctoral studies, he researched dynamic binary translation, a highly technical field, but he later gave it up.
The reason was simple: hardly anyone cared about it except me.
He said: I spent a lot of time researching these optimization methods, which could improve the system's performance by a few percentage points. Write papers, get cited, write more papers... And then what?
In academia, this is the standard path. But in the real world, these percentage improvements mean nothing. Kevin realized for the first time that the evaluation criteria in academia and the value judgments in the real world are not the same at all.
Academia values the number of papers, citation counts, and peer recognition. This mechanism is clear and easy to quantify. But the real world cares about: How many people benefit from it? How much value is created?
These two sets of standards are inconsistent and even conflicting.
So he left academia and joined Google. His first project was to automate the advertising review process. It didn't sound cool at all. Kevin said: It's actually just creating a set of automatic filtering rules to determine whether an advertising copy can use exclamation marks and whether it contains adult content.
But this problem was valuable. $50 million worth of ads were stuck every day, and manual review couldn't handle it. With just such a small change, it ultimately saved Google nearly $1 billion a year.
This earned him the Founder's Award.
From then on, he established a career principle. For every task, first check if it has the potential to make a real impact, and only then consider how interesting the technology is.
He said that life begins when you abandon the most complex technologies and turn to the most valuable work.
This was his first time identifying the right "signal".
Section 2 | Easily obtainable feedback is often noise
Kevin's choice 20 years ago was simple: academia or the business world, choose one.
The environment that today's AI entrepreneurs face is much more complex.
Starting a business is cheaper now than ever before. But precisely because it's cheap, there are more people trying and more noise.
What is noise?
He gave a definition:
Information that seems like positive feedback but has nothing to do with the product's value.
Media hot searches, like counts, and venture capitalists' interest may all be noise. Many entrepreneurs rely on these signals, which are exactly what can easily lead you astray.
Because there are their own interest mechanisms behind these signals.
- The media needs traffic and chooses stories with high topicality.
- Investors value a diverse portfolio of projects and show interest in popular sectors.
- The tech community chases the latest models and the largest number of parameters.
But these are different from someone being willing to pay for your product.
What's more troublesome is that this noise is especially easy to obtain.
You post a message and get hundreds of likes; you write an article and it makes the hot list; you meet a few investors and they all say they're interested. You start to think you're on the right track. But it may just mean that you've hit on a popular topic and become shareable content, rather than solving a real problem.
Many entrepreneurs start to align with popular concepts: "We're AI + education", "The browser in the AI era", "A new note - taking tool in the GPT era"...
These statements are novel and easy to tell, but it doesn't mean that people really need them.
Kevin provided a judgment method: You need to distinguish between two things.
One thing is what you hope will happen; the other thing is what will happen whether you're involved or not. The latter is truly suitable for entrepreneurship.
So, what are the real signals?
Kevin used ChatGPT to illustrate.
Section 3 | What was OpenAI doing when everyone was focusing on models?
When ChatGPT was launched, it used an old model. Many people in the industry had seen it. Kevin said that including himself, no one thought it would become a hit.
So why did it succeed?
Because while everyone was chasing one signal, OpenAI was chasing another.
At the end of 2022, every lab was chasing easily quantifiable numbers: a larger number of parameters, a higher benchmark, and a more advanced architecture. But these were noise.
The signals that OpenAI focused on were:
Can ordinary people use it without any barriers?
Is the interaction natural enough?
Can it become a daily tool?
These are the real needs.
ChatGPT made minimal changes: an old model, plus RLHF, plus an input box. There was no technological breakthrough. But it allowed ordinary people to have direct conversations with AI for the first time without any technical barriers.
Kevin clearly stated:
"It's not the most powerful model we've seen, but it's the first time that AI has directly entered users' lives."
In his view, such opportunities have three characteristics:
The technological capabilities are already sufficient.
But no one has seriously designed the usage.
It's overlooked because it seems too ordinary, too basic, and lacks popularity.
These opportunities won't appear on the hot - word lists and can't tell grand stories, but they may open up a trillion - dollar market.
The key is: How to identify such opportunities?
Section 4 | Three criteria for identifying real signals
For this, Kevin provided three judgment criteria.
Criterion 1: Look at the gap between capabilities and usage
He said: It's not that AI isn't strong enough now; it's that many people don't know how to use it well.
What does this mean? It means that the opportunities lie not in the capabilities but in the usage.
He gave the example of long - term memory. Current large models can have conversations but can't remember history. Users have to repeat the background every time, and the AI is like an "intern" who's always drunk and forgetful.
Can it be solved technically? Absolutely. You just need to create a data pipeline, compress content, and keep historical records.
But no one is doing it.
Why? Because it won't make it into papers and won't get media attention.
Kevin said: Many people are reluctant to do this because it seems like patching rather than creating. But this is exactly the real signal. Because users really need it, the technological capabilities are sufficient, but no one is seriously working on it.
Criterion 2: See who is creating the noise
If the media is reporting, investors are chasing, and big companies are making plans, it's likely to be noise.
This is like Kevin's choice 20 years ago: He did what everyone thought wasn't cool, but it solved a real problem. The same is true today. Too many resources are chasing those popular sectors, and it's difficult for you as an entrepreneur to win.
The real opportunities are often in the overlooked areas: Big companies think it's too trivial, the media thinks it's not eye - catching, and investors think it's not grand enough.
Precisely because it's overlooked, the competition is smaller.
Criterion 3: Conduct small experiments for verification
Kevin said that the cost of making tools has never been lower. What's really lacking now is people willing to take action.
For example:
Make the AI remember the user's history. Build a simple context cache and see if users really need it.
Use existing tools to build an end - to - end process. Connect AI conversations, automation tools, and document systems to create a complete closed - loop and see if it can really replace manual work.
Don't write PPTs; just create interactive prototypes. Start from the product experience rather than concept packaging.
Now is the best time for AI startups because you don't need to predict the future. You can find a good direction just by doing a small experiment.
The key is to distinguish between noise and real signals.
Are you chasing easy - to - tell stories or solving real problems?
Conclusion | Signals are more important than directions
Kevin Scott said that he doesn't pursue happiness; he just wants to do meaningful things.
Because meaningful things come with clear signals.
20 years ago, Kevin only needed to choose between academia and the business world. Today's entrepreneurs face a more complex environment: media hype, investors' interest, trendy tech terms. Which are noise? Which are signals?
Kevin's method is: Don't guess by judgment; verify by action.
The cost of making tools has never been lower, but the noise has never been more.
Finding the right signals is more important than finding the right direction.
Reference materials:
https://www.youtube.com/watch?v=Vut9hUEKyfk&t=10s
https://news.microsoft.com/signal/articles/5-ai-insights-from-microsoft-cto-kevin-scott/
https://www.linkedin.com/posts/adityaagarwal3_what-did-kevin-scott-see-that-others-didnt-activity-7407467737867784193-0DGC
https://podcasts.apple.com/ie/podcast/minus-one/id1759014294
Source: Official media/Online news
This article is from the WeChat official account "AI Deep Researcher". Author: AI Deep Researcher. Editor: Shen Si. Republished by 36Kr with permission.