You don't need real test takers to take the actual exam to figure out how difficult the questions are: this AI test question evaluation framework has "quantified" exam difficulty.

How difficult can a test question be for a human? Can we find clues from the problem-solving process of large language models?

A research team from Virginia Tech, University of Maryland, MBZUAI and University of Pittsburgh proposes Epi2Diff: it splits the reasoning traces generated by large language models into "cognitive episodes", and predicts human item difficulty in combination with the content of the question.

On four real evaluation datasets including SAT Math, SAT Reading and Writing, Cambridge and USMLE, Epi2Diff ranks first in all metrics of the single main experiment. Among them, the difficulty classification accuracy of SAT Math reaches 73.0%, which is 3.2 percentage points higher than the strongest baseline.

The relevant paper has been accepted by Transactions on Machine Learning Research (TMLR), a machine learning journal.

Research Background: Beyond Question Semantics, There Exists the Problem-Solving Process

Judging the difficulty of new questions is the basic work of test paper assembly and question bank construction. Traditional educational measurement relies on the calibration of test takers' response data, which is reliable but has a high pre-test cost; only analyzing the semantics of the question stem makes it difficult to figure out where the problem-solving process requires extra effort.

Epi2Diff extends the observation window to the problem-solving process: how long does the model deduce, does it focus more on analysis or execution, and does it switch strategies repeatedly? These statistically measurable behaviors provide new process signals for difficulty prediction.

△

It needs to be emphasized that what the research aims to predict is the difficulty of questions for humans. The model trace is a proxy evidence that can be obtained at scale, which is ultimately supervised by existing human difficulty labels, rather than directly measuring human cognition.

Method Design: Convert Reasoning Traces into Structured Features

Eight Types of Cognitive Episodes: What Is the Model Doing?

A piece of reasoning often contains different activities such as understanding the question, formulating strategies, performing calculations, and checking answers. Drawing on Schoenfeld's problem-solving episode theory and the ThinkARM extension, the study classifies each sentence of the trace into eight categories: reading, analysis, planning, execution, exploration, verification, monitoring, and response.

These "episodes" are sentence function labels, not a fixed eight-step problem-solving process. The RoBERTa-based classifier achieves 79.9% accuracy and 0.761 macro-averaged F1 on the 7067 manually annotated mathematical reasoning test set of ThinkARM; the accuracy of supplementary annotation evaluation in the target domain ranges from 62.9% to 74.2%, indicating that there are still errors in cross-domain automatic classification.

83-dimensional Process Features: How Does the Model Solve Problems?

Epi2Diff further extracts 83-dimensional features from the trace: 3 length statistics, the number of tokens of 8 types of episodes, the token proportion of 8 types of episodes, and 64 types of adjacent episode transition counts, which respectively describe "how much effort is invested", "where the effort is invested" and "how the process unfolds".

For example, two traces have similar lengths and similar proportions of analysis and execution, but one focuses on analysis first and then executes continuously, while the other switches frequently between analysis and execution. The transition feature retains such order differences.

△

In the experiment, QwQ-32B and Qwen3-32B are used to generate traces under the conditions of direct response and prompts marked as "weak, medium, strong", and then the features are averaged. These prompts are used to induce different problem-solving paths, and do not represent verified student ability levels.

Finally, the process features are concatenated with the question semantic vector obtained by Sentence-BERT, and the difficulty is predicted through the supervised model. After the four datasets are selected by the validation set, XGBoost is finally used as the predictor.

Experimental Results: Four Types of Real Evaluation Tasks

Classification and Regression Performance

The evaluation includes 1075 SAT Math questions, 1338 SAT Reading and Writing questions, 793 Cambridge English Reading questions, and 667 USMLE medical questions. The two SAT tasks predict three categories: "easy, medium, difficult"; the other two predict continuous difficulty values. The baselines cover fine-tuning of small models such as BERT, LLM zero/few-shot prompting, full-parameter and LoRA fine-tuning.

△

The accuracy of SAT Math increases from 69.8% of the strongest baseline to 73.0%, and the macro-averaged F1 increases from 0.705 to 0.728. The improvement of SAT Reading and Writing is relatively small: the accuracy rises from 62.3% to 63.1%, and the macro-averaged F1 rises from 0.622 to 0.626.

In the regression task, the RMSE of Cambridge decreases from 8.200 to 7.612, and that of USMLE decreases from 0.298 to 0.291; the corresponding R² increases from 0.299 to 0.396 and from 0.076 to 0.121 respectively.

Repeated Experiments and Result Boundaries

In the five supplementary runs, the average accuracy and macro-averaged F1 of the two SAT tasks are still the highest for Epi2Diff. However, the leading margin of the Reading and Writing task is smaller than the fluctuation between runs, so the advantage should be interpreted with caution. The repeated experiments fix the traces and episode features, and only change the random seed of the downstream predictor.

Ablation Analysis: Longer Reasoning Does Not Necessarily Mean More Difficult Questions

The SAT Math ablation results show that the performance improvement does not only come from the trace length. When only question semantics is used, the macro-averaged F1 is 0.607; after adding the length feature, it is 0.671; when the complete representation is used, it reaches 0.728. Even if other features are retained, removing the episode transition information will reduce it to 0.676.

△

Further permutation experiments retain the trace length and episode frequency, and only change the arrangement of intermediate reasoning sentences and labels. The original order performs better, which supports that the local transition order carries predictive information.

In SAT Math, more difficult questions are also accompanied by more intermediate deductions, higher execution investment, and more obvious transition signals of "analysis → execution", "execution → execution", and "execution → analysis". However, these patterns have limited replication in other domains, which are process correlations, and cannot be regarded as direct observations or causal proofs of human cognition.

Application Value: Provide Interpretable Clues for Question Bank Construction

Epi2Diff attempts to convert "how the question is solved" into comparable features, providing auxiliary clues for pre-evaluation of new question difficulty and manual question review. In actual use, data training with labels, target population verification and expert review are still required. Its value is not to replace educational measurement, but to add a set of evidence from the reasoning process for question difficulty judgment.

Paper Title: Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

Authors: Chenguang Wang*, Ming Li*, Xinyue Zeng, Zhuochun Li, Hong Jiao, Tianyi Zhou, Dawei Zhou (* equal contribution)

Paper Link: https://arxiv.org/abs/2606.28186

Project Code: https://github.com/c-steve-wang/Epi2Diff

This article is from the WeChat official account "QbitAI", authored by the Epi2Diff team, and released with authorization from 36Kr.