Can large models write industrial-grade optimization algorithms? MIT introduces FrontierOR to set up an examination room for AI

Over the past two years, LLMs have made rapid progress in the fields of "natural language to mathematical model" and "natural language to solver code". Models can now understand problem descriptions, write MIP formulations, and call Gurobi or other solvers, seemingly demonstrating preliminary optimization modeling capabilities. However, this is still far from sufficient when tackling real-world industrial-scale problems.

The real challenge is not translating individual constraints into mathematical expressions, but designing algorithms that can run on large instances efficiently, accurately, and quickly. Even a perfectly formulated MIP model, when handed to a general-purpose solver, may fail to produce a provable high-quality solution within an hour. This is why OR engineers in practice still need to implement decomposition algorithms, column generation, Benders decomposition, local search, metaheuristics, and math programming-heuristic hybrid methods.

Recently, researchers from institutions including the Massachusetts Institute of Technology proposed FrontierOR: an LLM evaluation benchmark focused on large-scale optimization algorithm design capabilities.

Unlike traditional benchmarks that only examine "modeling ability" or "solver invocation capability", FrontierOR focuses on whether LLMs can, like real OR researchers and engineers, design scalable, high-quality, and efficient algorithms tailored to complex problem structures.

Paper link: arxiv.org/abs/2605.25246

Project homepage: frontieror.vercel.app

Code repository: github.com/Minw913/FrontierOR

Dataset link: SmartOR/FrontierOR

The core question FrontierOR addresses is: Can today's most powerful large models, starting from real-world problems, independently design competitive and efficient algorithms? Can they go beyond merely "calling solvers" and, like OR experts, select decomposition, heuristics, search, and hybrid strategies based on problem structures?

The significance of this work lies in shifting the evaluation focus of LLM-for-OR from "ability to write models" to "ability to design algorithms". This is a critical threshold that large models must cross to advance into real industrial decision-making systems.

Research Background

Many existing benchmarks focus on the modeling capabilities of LLMs in optimization problems, such as requiring models to generate mathematical programs from natural language descriptions, invoke solvers, or verify answers on small-scale instances. These tasks are important, but they often fail to answer a question more relevant to industrial deployment: can models actively create more efficient algorithmic paths on large-scale instances where solver performance has already saturated?

In the practice of operations research and optimization, general-purpose solvers are only the starting point, not the destination. Real-world problems often have special structures: network flow structures, time decomposition structures, vehicle routing structures, inventory-routing coupling, machine-operation coupling in scheduling, and capacity-coverage coupling in facility location, among others. Skilled algorithm engineers leverage these structures to decompose, approximate, relax, and recombine the original problem, then solve it through a mix of heuristic and exact methods.

Therefore, a truly OR-oriented large model benchmark must satisfy three conditions simultaneously: sufficiently realistic problem sources, sufficiently large instances, and sufficiently rigorous evaluation. FrontierOR is proposed under this context: it does not present large models with a set of "optimization practice exercises", but translates peer-reviewed complex problems from over 30 years of OR literature into algorithm design tasks that can be automatically evaluated.

Table 1: Multi-dimensional comparison between FrontierOR and representative OR/LLM-for-optimization benchmarks

Methodology

The construction process of FrontierOR can be summarized into four steps: selecting problems from literature, converting paper problems into standardized task components, undergoing dual automatic and expert quality checks, and then filtering out a more challenging Hard subset.

• Step 1: Problem selection from real literature. The data source covers more than 20 OR journals from 1992 to 2025, totaling 180 papers. Selected tasks must have clear problem definitions, and the original literature must have demonstrated the engineering value of specialized algorithms over general solvers.
• Step 2: Standardizing task components. Each paper is converted into a natural language problem description, mathematical model, Gurobi reference implementation, reference solution, and an independent feasibility checker.
• Step 3: Two-layer quality validation. First, automatic cross-validation checks whether the Gurobi reference solution is consistent with the feasibility checker; subsequently, 15 OR experts conduct multiple rounds of review to verify the consistency of the model, description, code, and checker.
• Step 4: Hard subset filtering. 50 more difficult tasks are selected from 180 tasks, focusing on scenarios with combinatorial explosion, larger scales, more tightly coupled constraints, and where Gurobi cannot prove optimality within a 1-hour time budget.

Figure 1: Full overview of the FrontierOR benchmark: problem categories, application domains, instance scales, and construction workflow

Evaluation Protocol

The evaluation process also emphasizes end-to-end capabilities. The model first generates a complete algorithm program based on the natural language task. The program undergoes pre-screening for executability, feasibility, and solution quality on small instances: if it times out, is infeasible, or has a gap of more than 10% compared to Gurobi's small instance solution, it will not proceed to the large instance evaluation.

After passing the pre-screening, the program runs on multiple large-scale instances for each task and is compared with expert-reviewed Gurobi reference solutions. FrontierOR uses four metrics: Execution rate, Feasibility, Solution quality, and Quality-Time Efficiency (QTE, a composite of quality and speed). QTE is the most stringent: a result is considered successful only if the relative gap between its objective value and the Gurobi reference solution does not exceed 1%, or if it outperforms the Gurobi solution.

Figure 2: The two-stage evaluation process of FrontierOR: pre-screening on small instances, evaluating quality and speed on large instances

Experimental Results

One-shot: Executability Approaches the Upper Limit

In the one-shot setting, models are required to generate complete algorithm programs from scratch, with limited self-debugging allowed based on execution errors, but no repeated rewriting of algorithms according to evaluation feedback. This setting examines the model's comprehensive ability to read problems, build models, design algorithms, and code in a single pass.

The results show that the executability of top-tier models has already reached a very high level. For example, GPT-5.3-Codex achieves an Execution rate of 0.98 on the Full dataset, while Gemini 3.1 Pro and Claude Opus 4.6 both reach 0.93. This indicates that for state-of-the-art models, "whether the code can run" is no longer the main bottleneck.

However, executability does not equal problem-solving ability. Feasibility, Solution quality, and QTE remain significantly lower than the Execution rate. In other words, large models can now write formally complete optimization programs, but it is still difficult to make these programs remain feasible, near-optimal, and faster than Gurobi at industrial scales.

From an overall hierarchical perspective, state-of-the-art models significantly outperform other mainstream models on both the Full dataset and the Hard subset. On the full FrontierOR dataset, the Feasibility of top-tier models is concentrated between 0.60 and 0.62, while other mainstream models range from approximately 0.18 to 0.42. The gap persists on the Hard subset: top-tier models score 0.49–0.64, while other mainstream models drop to 0.13–0.37.

The Hard subset further widens the gap in algorithmic capabilities among top-tier models. The QTE of the three leading models on the full dataset falls within a narrow range of 0.25–0.31, seemingly close; but on the Hard subset, the QTE of Claude Opus 4.6 still reaches 0.32, while GPT-5.3-Codex drops to 0.18, a nearly 2-fold difference. The Hard subset thus becomes a true "watershed of algorithm engineering capability".

Table 2: FrontierOR one-shot evaluation results: Execution rate, Feasibility, Solution quality, and QTE on the Full dataset and Hard subset

Divergence in Algorithm Selection Emerges

The research team further analyzed the solving methods adopted by model-generated programs, classifying them into five categories: pure solver invocation, decomposition, constructive heuristics, local search/metaheuristics, and math programming-heuristic hybrid methods. This analysis is critical, as it directly reveals whether models truly possess algorithm design awareness.

The results show that weaker models rely heavily on pure solver invocation. For example, approximately 99% of LLaMA-4-Maverick's programs are monolithic solver calls, essentially handing the problem over to a general-purpose solver. In contrast, Claude Opus 4.6 has the most balanced method distribution: about 37% pure solver calls, 27% local search/metaheuristics, and 27% math programming-heuristic hybrids.

More importantly, non-pure-solver methods generally perform better on the QTE metric. This means that "method diversity" itself is a competitive advantage: the more a model can select decomposition, heuristic, and hybrid algorithms based on problem structures, the more likely it is to achieve both quality and speed on large instances.

Figure 3: Distribution of solving methods and failure mode analysis for programs generated by different models

Failure Mode Shift: From "Poor Modeling" to "Insufficient Search Depth"

Failure mode analysis shows that as model capabilities improve, the location of errors is systematically shifting downstream. Weaker models mainly make mistakes in early stages such as mathematical model design, constraint specification, and I/O schema; stronger models have significantly fewer errors in these foundational stages, with new bottlenecks turning to the depth and quality of heuristic search.

This is very similar to the growth path of human algorithm engineers. Beginners first make modeling mistakes: unclear variable definitions, missing constraints, mismatched inputs and outputs; more experienced engineers are less likely to make these low-level errors, but face more difficult problems: whether the search strategy is sufficiently robust, whether neighborhood design is effective, and whether relaxation and repair can balance speed and quality.

Therefore, FrontierOR not only tells us "who scores higher", but also "where the capability bottlenecks lie". This is particularly important for the design of next-generation LLM-for-OR systems: future breakthroughs may not come from models that are better at writing formulas, but from systems that are better at searching, combining algorithmic skills, and self-improving through feedback.

Self-Evolution

Single-pass generation is only the first step. In practice, algorithm design is never a one-and-done process, but an iterative cycle of running, analyzing failures, modifying strategies, and re-running. FrontierOR therefore further evaluates three test-time self-evolution frameworks: OpenEvolve, EoH, and CORAL.

The experiment selects the hardest 40% of tasks in the Hard subset as the self-evolve test set, using GPT-5.3-Codex's single-pass generated program as the initial seed. Each framework is uniformly limited to 30 candidate programs, with the final best result taken as the end state. This ensures that differences primarily stem from the search mechanism rather than variations in initial programs.

The results are remarkable: under the three self-evolution frameworks, the best candidate programs significantly outperform single-pass generation across all metrics. QTE increases from 0.15 in one-shot to a maximum of 0.50, meaning that on the hardest tasks, about half of the large instances can now satisfy both "quality close to Gurobi" and "speed no slower than Gurobi" using LLM-generated algorithms.

Among them, CORAL achieves the most stable improvement with its multi-agent shared memory mechanism, reaching a QTE of 0.50; OpenEvolve follows closely with a QTE of 0.49; EoH also brings noticeable improvements, but with greater performance fluctuations, achieving a QTE of 0.33.

Table 3: Performance of the three test-time self-evolution frameworks on the hardest tasks: QTE increases from 0.15 to a maximum of 0.50

Further observation of the evolution trajectory reveals an enlightening phenomenon: the speed dimension often breaks through the Gurobi baseline within the first 5 attempts, while the solution quality dimension is much more difficult to improve. The reason is not hard to understand: making an algorithm run faster can be achieved by adopting lightweight constructive heuristics; but to approach global optimality while remaining faster requires more refined neighborhoods, repair strategies, relaxation strategies, and search control.

This demonstrates that LLM self-evolution is not simply "trying code multiple times". Truly effective self-evolution requires remembering historical failures, identifying performance bottlenecks, dynamically adjusting search directions, and making structured trade-offs between speed and quality.

Figure 4: The quality-speed 2D evolution trajectory of the three self-evolution frameworks: speed is easier to break through first, while quality improvement is more challenging