Don't rush to build a "software factory": without these 4 hard metrics, you are just piling up technical debt with AI.

Shen Translation Bureau is a translation team under 36Kr, focusing on fields including technology, business, workplace and daily life, with a core focus on introducing new technologies, new perspectives and new trends from overseas markets.

Editor's Note: You did not adjust your workflow, simply purchased Copilot and claimed to be AI-native? Do not deceive yourself. Actual tests from hundreds of teams show that the vast majority of companies have not even passed the "Phase 0". This article comes from a translated content.

"AI-native engineering" has become a buzzword that everyone talks about but no one can define clearly. Vendors use it to promote tools; founders use it to polish their board meeting PPTs; CTOs use it to literally mean nothing more than "we have purchased the Copilot license".

None of these can be regarded as real AI-native engineering.

The so-called AI-native engineering refers to the scenario where AI fundamentally changes the way the team operates. Workflows, code review mechanisms, quality gates and evaluation systems are all reshaped around AI as a core participant.

Most teams have not achieved this. They only added some extra tools on top of the operating model that has been used for three years.

Where exactly are you located

According to data from Akkodis, this year the confidence of CTOs in large-scale AI application has dropped to 48%, far lower than the 82% in 2024. A report from HCLTech states that an estimated 43% of enterprise-level AI deployment projects will end in failure. Gartner conducted a survey of 350 large enterprises that raised funds for AI through layoffs, and found that the vast majority of these enterprises obtained at best negligible benefits.

The tools themselves are indeed effective, but the operating models have not kept pace.

This gap is wider than most CTOs imagine, and the cost of ignoring it is becoming increasingly high.

Everyone is talking about software factories, but most teams are not ready yet

Discussions about "software factories" have exploded in full swing. Zach Lloyd published an internal memo, asserting that his engineers are "factory engineers, not product engineers"; Chamath talked freely about the factory his team built at 8090; Microsoft is also vigorously promoting Agent Factories.

But software factories belong to Phase 3 or Phase 4, while the vast majority of teams are actually still stuck in Phase 0.

We have hundreds of AI-native engineers who are now deeply embedded in more than 100 companies, many of which are private equity (PE) backed mid-sized enterprises.

We do not provide empty AI implementation consulting services.

We embed engineers directly into customer teams to build solutions together with them.

Almost every team we have cooperated with is using AI tools, but almost none of them changed the operating model around these tools before we joined.

The five phases of AI (and why you skipped four of them directly)

Therefore, we summarized the experience of actual deployment in more than 100 engineering teams, and built a framework on this basis. These are by no means theories conceived out of thin air, but verifiable regular patterns that have repeatedly appeared in a large number of project practices.

Phase 0: Diagnosis Period

At this stage, you don't even know where you are. Your engineers may be using AI, but you can't figure out who is using it and to what extent, let alone judge whether it creates actual value or technical debt.

There is no baseline, let alone an infrastructure with measurement indicators.

Solution: Deploy developer intelligent analysis tools, establish a DORA-style baseline covering R&D speed, quality and AI adoption rate, and conduct a full-scan of the code base. It takes 2 to 4 weeks with very low cost.

Most companies are actually at this stage, but claim externally that they have reached Phase 2.

The leap from Phase 0 to Phase 1 is where most teams get stuck. The reason is not a technical bottleneck, but that no one is responsible for this transformation. There is no AI productivity lead, no promotion and implementation plan, and no fixed measurement rhythm. Those Copilot licenses are just hanging there, lonelyly generating code completion suggestions in the void.

Phase 1: AI Empowerment (Individual Level)

Every engineer becomes an AI-native engineer. AI agents have real and complete code base context; the AI productivity lead provides targeted guidance to the team; senior engineers act as gatekeepers for code reviews. It takes 1 to 3 months.

Phase 2: AI Empowerment (Organizational Level)

The team's workflow has undergone fundamental changes. The process from Task work orders to PR (Pull Request) is fully agentized: planning, implementation, test running, and PR submission for manual review are all completed automatically in the sandbox environment.

Let's share a real case we served: a logistics platform with more than 50 engineers. With only 2 engineers, 122 PRs were submitted and successfully merged within 3 months, of which about 90% of the code was generated by AI. AI usage cost: only about $200 per developer per month. It takes 2 to 4 months.

Phase 3: AI Factory (Product Dimension)

AI shifts from "how you build products" to "what products you sell". Large language model-driven features are directly integrated into customers' own products, bringing new revenue growth and stronger user retention.

Most discussions about "software factories" start from this point. But in our view, this is already Phase 3, not Phase 1. It takes more than 3 to 6 months.

Phase 4: AI Factory (Governance Dimension)

Build a complete control plane. It includes model gateway, policy control for a single code base, full-link tracing, impact analysis, compliance review and audit logs.

The core measurement indicator changes to "actual value generated per unit of Token" (impact-per-token), instead of the simple number of lines of generated code. It takes 6 to 12 months.

Before you call anything a "factory"

Answer the following four questions first:

Can your team review AI outputs as fast as AI generates code?

Do you have enough senior engineers to accurately identify and correct mistakes made by AI?

Is your system architecture documentation machine-readable?

Have you clearly sorted out and recorded what types of work AI is actually a "drag burden" rather than an "efficiency multiplier" at present?

If you cannot answer all four questions affirmatively, it means you are not ready to enter the "factory" stage, and you are at the stage where "empowerment" is needed.

The real position you are most likely in

The vast majority of teams I have communicated with think they are already in Phase 2.

However, the vast majority of teams actually only bought Copilot licenses and still stay in Phase 0.

Teams that take the lead in clarifying this difference will open an irreparable generation gap in delivery speed; teams that fail to notice this gap can only try to make up for the gap in vain by constantly hiring more people.

Piling up manpower can no longer bridge this gap.

Today, how many of these four questions can your engineering team answer clearly?

Translator: boxi.