Apple's new research on Personal Agent: Being able to help you handle your tasks does not mean it truly understands you

Personal Agent is now able to independently complete many complex operations, but it still faces a problem when making decisions: when should it make inferences directly, and when should it ask the user for additional information.

For example, when a user asks to book the restaurant they usually visit, the Agent can identify which restaurant it is from historical records, and can also recognize the gathering time and number of people that need to be confirmed. If it makes arbitrary guesses, it may deviate from the user's real intention; if it repeatedly asks even for information that is clearly stated in historical records, the automated process will become lengthy and reduce usage efficiency.

In this work, the Apple team proposed iPrefBench, which is specifically designed to evaluate the "inference vs. inquiry" capability of mobile-side Agents in such ambiguous requests. This benchmark runs in an executable iOS environment, covering 26 applications, 130 incompletely specified tasks, and 3051 activity records constructed around the same user. The Agent can only obtain existing information from the user's history, and can ask questions to the simulated user through the ask_user action.

Paper link: https://arxiv.org/abs/2610.34735

The core design of iPrefBench is to mark a unique source for each omitted user-related piece of information. One type of information can be recovered from historical activities, such as the restaurant, seat or travel mode the user usually chooses; the other type of information can only be provided by the user in the current context, such as the time and number of people for this gathering. The Agent must first identify which decisions in the task are still undetermined, then judge whether these decisions should be inferred from history or obtained by asking questions.

Figure | iPrefBench evaluates both preference inference and active inquiry capabilities at the same time

This distinction fills a gap in traditional computer usage Agent evaluations. Many evaluations use tasks with complete information, where the Agent does not need to infer or ask questions. Other personalized evaluations provide user history but do not allow the Agent to interact with users; interactive evaluations allow inquiries, but do not have sufficient user history to judge whether a certain question is redundant. iPrefBench puts two types of information sources into the same environment, separately measuring whether the task is completed, whether it meets the specific user's needs, and whether the Agent disturbs the user only when necessary.

The research team used the same set of mobile-side Agent Harness to evaluate 7 cutting-edge models. The results show that there is a clear gap between the execution capability and the personalization capability of Agents. The best-performing Claude Opus 5 achieved an action completion rate of 91.6%, but the full task success rate was only 47.7%. Agents are often able to complete operations such as booking, purchasing or filling in forms, but may not make choices that match the user's intention.

Figure | Key results on the iPrefBench benchmark

All tested Agents performed significantly weaker when dealing with information that can only be obtained by asking the user. Taking Claude Opus 5 as an example, for information that can be inferred from historical records, its task completion rate is 76.3%; for information that must be obtained by asking the user, the completion rate drops to 51.3%.

Overall, each Agent initiates only 0.08 to 0.35 inquiries per task on average, while each task contains an average of 0.89 pieces of information that must be confirmed with the user. For this type of information, the highest inquiry recall rate is only 44.3%. This indicates that Agents often make decisions directly when they need to confirm with the user.

However, a small number of inquiries is only part of the problem. The inquiry precision rate of each model ranges from 60.4% to 77.1%, indicating that a considerable part of the questions they raise are still unnecessary, for example, the relevant answer already exists in historical records, the inquiry content is irrelevant to the current task, or the existing information is repeatedly asked. Therefore, the key capability that Agents need to have is to accurately judge which information can be inferred from existing records and which information really needs to be confirmed with the user.

The research team also designed three types of perturbation tasks to test whether Agents will blindly copy historical preferences. The first type is that the current instruction conflicts with long-term habits; the second type is that recent behaviors show that preferences have changed; the third type is that the originally preferred option is not available in the current environment. The evaluation results show that the personalization scores of Agents in the first two types of tasks are lower than the overall level, while environmental perturbations prompt them to significantly increase the number of questions. When the preferred option is unavailable, the average number of questions initiated by the Agent reaches 2.3 times the original number, and the processing of information that can only be obtained by inquiry is also improved.

Figure | Performance of iPrefBench on the three subsets of perturbation tasks

Error analysis further reveals four typical failures. Facing information that can only be obtained by inquiry, Agents sometimes realize that the information is missing but still guess on their own, and sometimes even fail to identify the missing information as an item that needs to be decided, directly using the application's default value. Facing information that can be inferred from history, Agents may also completely fail to find the relevant preference, or draw wrong conclusions from history after finding it. The research team believes that current Agents face two interrelated problems: first, find out all unresolved user-related choices, and then decide for each choice whether to infer or inquire.

Figure | Distribution of personalization failures in two dimensions

The research team also tested a pre-execution clarification stage, allowing the Agent to centrally identify doubts and ask questions before starting operations. The experimental results show that after Claude Sonnet 5 adds this stage, the full task success rate increases from 39.2% to 53.8%, and the processing score for information that can only be obtained by inquiry increases from 25.7% to 85.0%; but the inquiry precision rate drops from 60.4% to 42.5%, and the average number of questions per task increases from 0.24 to 1.37. Similar changes also occurred in Qwen3.8-27B. Advance clarification can reduce missed inquiries, but at the same time increase unnecessary questions.

Figure | The impact of adding an explicit clarification stage before execution on Claude Sonnet 5 and Qwen3.8-27B

The number of user histories is not always the more the better, experiments show that providing a small number of task-related historical events can significantly improve the processing of inferable preferences; continuing to add complete history will not bring the same magnitude of benefits. More history helps Agents avoid inquiring about information that can already be inferred, but does not help them find out which information still can only be provided by the user, and may even reduce the inquiry recall rate for such information.

Figure | The impact of providing different amounts of user historical records to the Agent context

iPrefBench reveals a common challenge faced by current mobile-side Agents: they are already able to perform interface operations well, but are still not good at dealing with unknown information in user intentions. A truly personalized Agent needs to combine historical records and the current context to understand user needs, not only avoiding mechanically applying past habits, but also reducing unnecessary follow-up inquiries. How to make accurate judgments between autonomous execution and active confirmation is the key to improving the personalization capability of Agents.

This article is from the WeChat official account "Academic Headline" (ID: SciTouTiao), author: Academic Headline, authorized for release by 36Kr.