Not all products are suitable for AI transformation: A judgment framework for AI product managers
AI is not a universal key. Blindly integrating AI into products may instead make them more "clunky". From the practical perspective of AI application product managers, this article proposes a four-layer judgment framework covering scenario, capability, business and product, to help identify which tasks truly require AI and which are more suitable for traditional rules. The core view is that a good AI product makes users feel faster and more natural, rather than deliberately making them perceive the "large language model".
AI is entering our work and life from all directions. In office scenarios, AI-generated PPTs and AI-written documents are already very common; in daily consumption scenarios, AI ride-hailing, AI video summarization and AI-assisted search have gradually become familiar experiences for users.
Many products hope to convey a mindset when adding AI: as long as it is "AI + XX", it means smarter, more efficient and easier to use. However, in actual use, it is not hard to find that some products become slower, more convoluted, or even "clunkier" after adding AI.
For example, an operation that originally only required clicking a button to complete is transformed into requiring users to input a prompt; a judgment that could be stably completed relying on fixed rules is handed over to the large language model, which leads to problems such as unstable results, slower response and uncontrollable user experience.
The fundamental reason is that large language models are better at handling tasks with high information density, high contextual complexity and high uncertainty. They are suitable for helping people complete information integration, understanding, induction and generation. But for scenarios with low information volume, clear operation paths and clear rule boundaries, AI is not necessarily faster than a button, a rule, or even a line of command.
As an AI application product manager, the core thinking should be based on the understanding and judgment of model capabilities, taking the insight of business scenarios and user needs as the framework, to judge whether "after integrating AI, the task can really be completed faster, more stably and more naturally". It is not necessary to make users strongly perceive that "a large language model is used here", but to make users feel that: this task is completed faster, the understanding cost is lower, the result is more useful, and the process is more natural.
I. Basic Model Capability Assessment: Judge Whether Capabilities Can Be Transformed into Business Increment
When a new model is launched, I will first pay attention to its advantageous capabilities, such as text understanding, multimodal recognition, complex reasoning, code generation, long context processing, etc., and then combine with specific scenarios in the current business to judge whether these capabilities have the opportunity to be transformed into real product functions.
The focus of this stage is not to "see a new model and immediately find a scenario to reflect it", but to judge whether it can really solve business problems. A model that performs well in public evaluations does not necessarily mean it is suitable for the current business. The inputs in product scenarios are often non-standard, user expressions are not necessarily clear, business rules may be very complex, and the results need to be understandable and trustworthy for users.
If some business modules have completed the transformation of the AI application layer, it is necessary to continuously evaluate whether the new model can bring better results in the follow-up. Usually I will focus on five dimensions: effect, speed, cost, stability and transformation cost.
Taking proofreading products as an example, proofreading pursues high recall and high accuracy, with a wide inspection range and high inspection speed requirements. Therefore, while the model effect is important, cost and speed will also become very heavy constraints. If a model only performs well on a small number of samples, but cannot run stably and at low cost in real traffic, it is difficult to directly enter productization.
II. Model Effect Verification: Do Not Only Verify That It "Looks Usable"
In model evaluation, the most critical thing is not simply "running tests", but how to design test cases and evaluation criteria. Test samples and evaluation standards should be closely related to business indicators, otherwise the evaluation results will easily stay at the demo level.
Test cases need to cover at least three types of samples: high-frequency samples, that is, the scenarios that users most often encounter and use; abnormal samples, that is, situations with chaotic formats, conflicting information and much noise; failure samples, that is, situations where the model performed poorly in history and received poor user feedback.
The value of doing this is that the model evaluation will not only answer "whether it can give an answer", but further judge whether it is robust enough in the real product environment.
In terms of evaluation criteria, I will not only look at a single indicator, but comprehensively consider factors such as accuracy, completeness, stability, response speed, and cost. Among them, I will pay special attention to controllability and fallback capability. Because in the product, the model is not a one-time demo, users care about whether the result is reliable, the process is smooth, and there is a comprehensible fallback solution when it fails.
In batch testing, the judge model can help complete the initial scoring, which is especially suitable for horizontal comparison of a large number of samples. But the judge model cannot completely replace manual judgment. It may prefer answers with complete expressions and clear formats, but may not really understand the key judgments in business scenarios. Therefore, I usually use the judge model as a batch preliminary screening tool, and then carry out correction through manual spot check.
III. Product Design and Implementation: Translate Model Capabilities into User Value
After the model capability is verified, the next step is product design. For me, the value of AI in the product workflow is mainly reflected in several links: information processing, solution verification, prototype implementation and iterative optimization.
In the early stage of the product, there are often many scattered inputs: user pain points, business demands, inspiration from competitors, optimization points proposed by the boss, data anomalies, personal judgments, etc. AI can act as a structuring assistant to help me connect these information and organize them into a more complete product thinking framework, including direction, pros and cons analysis and next to-dos.
When the product direction is initially clarified, the most important thing is to verify whether this idea is valid. Whether a function is worth doing cannot only rely on the product manager's own judgment, but needs to go back to users, the market and business scenarios to verify: do users really have this demand? Are there similar solutions on the market? Does our solution have differentiated value?
In this link, AI can assist in completing competitor research and solution analysis through Web Search, knowledge base and data summary. But I will not directly take AI's conclusion as the final judgment. AI tends to attribute "seemingly reasonable information" to a certain direction, so I pay more attention to whether it can provide key information and sources, and the final trade-off is still done by humans.
In the solution design stage, I will use vibe coding more to quickly build interactive prototypes and simulate the real user usage process. Compared with static prototypes, interactive prototypes are easier to expose whether the path is smooth, the state is complete, and whether users need additional learning. In this link, the value of AI is not to show off skills, but to reduce the cost of building demos, so that product managers can discover process problems faster.
After the product is launched, AI can also undertake a large number of repetitive but necessary tasks, such as data analysis and preliminary attribution, user feedback clustering, version iteration review, etc. It can reduce dirty work and allow product managers to focus more on judgment, decision-making and design. However, this kind of analysis cannot be completely handed over to AI either, especially user feedback analysis. Strong expression does not mean the largest problem scale, nor the highest priority, and it still needs to be judged in combination with data.
IV. My AI Productization Judgment Framework
To sum up, I believe that AI application product managers do not simply integrate AI capabilities into products, nor do they immediately look for usage scenarios when they see new models. A truly effective AI transformation requires four layers of judgment.
The first layer is scenario judgment: does this scenario really need AI? If a scenario has clear rules, fixed paths and low information volume, traditional rules, buttons or automated processes may be more stable and efficient than AI. AI is more suitable for scenarios with complex context, large amount of information, and the need for understanding and generation.
The second layer is capability judgment: are the model capabilities stable, controllable and evaluable? The model can run through in the demo, which does not mean it can be used stably in the product. Product managers need to judge whether the model is reliable under real inputs, boundary samples and abnormal scenarios, and also consider cost, speed, controllability and fallback solutions.
The third layer is business judgment: does AI transformation bring real increment? Does it improve efficiency, enhance experience, reduce costs, or make tasks that originally cannot be completed at scale scalable? If there is no clear business benefit, AI will easily become a packaging layer rather than part of the product value.
The fourth layer is product judgment: is AI naturally embedded in the user process? A good AI product does not create additional demands for users, nor should it require users to learn a complex process in order to use AI. AI should be embedded in the task path that users are accustomed to, reduce understanding costs, and not add new operational burdens.
If there are no clear answers to these questions, this AI function may only "look very AI" but does not really create value.
Conclusion: The Value of AI Product Managers Also Includes Judging What "Should Not Be AI-ized"
For me, the real value of AI tools is to help product managers complete information processing, solution verification and prototype implementation faster. It cannot replace us to understand users, nor can it replace us to judge whether a function is really worth doing.
Product managers still need to answer the most critical question: does this AI capability really make users' work and life simpler? If the answer is yes, AI is part of the product value; if the answer is no, restraining from doing AI transformation is also a kind of product judgment.
A truly good AI application does not necessarily make users strongly perceive that "a large language model is used here", but makes users feel that: this task is completed faster, the understanding cost is lower, the result is more useful, and the process is more natural.
This article is from the WeChat Official Account "Everyone is a Product Manager" (ID: woshipm), the author is Honghong Jiang, and it is published with authorization from 36Kr.