A pelican has become the biggest star in the Agent era.

"Every day, tens of thousands of videos of pelicans riding bicycles are posted to groups in the community, asking me if this pelican has lost its intelligence." Lambda, initiator of the open-source project CodexRadar, received a huge number of pelican videos every day after the release of GPT-6 Astra, so he simply launched a "Pelican Cup" competition in the community.

Participants take the task of "drawing a pelican riding a bicycle" to generate works with the model, and then send the results to the community.

"It went online late at night, and we received nearly a hundred submissions without any promotion." Lambda did not expect that everyone's enthusiasm for the "pelican test" was so high.

Every time a new model is launched, some people will use a pelican riding a bicycle to test the model's capabilities.

On the one hand, this task is simple and intuitive enough. Many complex capabilities of the model are difficult to judge quickly with a single sentence, but a pelican riding a bicycle makes it very easy to see the stability at a glance, such as whether the legs are aligned with the pedals, whether the wheels are moving accordingly, whether the movements have obvious flaws, and whether the generation is consistent before and after.

On the other hand, this task is complex enough. It seems to be just a short prompt, but behind it involves action understanding, temporal consistency, spatial relationship, code organization and continuous execution capabilities. For a model that has entered the Agent stage, this kind of task is very suitable to act as a "physical feeling thermometer".

After GPT-6 Astra went online, this pelican became even more popular. The reason is that this most advanced model has very unstable "intelligence", which will have a direct impact on work results.

The pelican just makes this kind of hard-to-quantify physical feeling very intuitive.

01 How Does a Pelican Test Whether the Model Has "Lost Intelligence"

The Pelican Cup competition is divided into two tracks. The Classic track uses a unified prompt: "Create an HTML file containing a 2D animation of a pelican riding a bicycle drawn with SVG". As long as the model is in normal state, the pelican generated by GPT-6 Astra usually maintains relatively consistent quality and picture structure.

Figure: The pelican image uploaded by users to social networks shows that if the intelligence of GPT-6 Astra remains stable under various inference intensities, the generated pelican images are basically consistent

If a large number of users suddenly find one day that under the same prompt, the pelican starts to frequently miss the pedal, the wheel relationship is disordered, and the structure is obviously deviated, discussions about "intelligence loss" will quickly emerge.

This method certainly cannot be called a strict scientific benchmark. A publicly available question repeated at high frequency also has the possibility of being gradually familiarized by the model. But it is very suitable for observing short-term changes in physical feeling. Users do not need to understand complex indicators, and can spot differences just by looking at a few pelicans.

Figure: Screenshot of the works of the "Pelican Cup", marking the "intelligence-lost" works, you can see that the pelican is "obviously wrong"

The Open track does not set a unified prompt, and participants can give full play to their creativity freely. It is more like testing how far Astra can go with an open task.

One of the works was finally made into a 30-minute "one-shot" cycling story.

The whole work shares a 1800-second timeline and the same rider. The ground, mountains, cities, planets, wheels, feathers and particles are all composed of SVG elements, and JavaScript is only responsible for calculating positions, joints and time. The entire project does not use Canvas, images, videos, external fonts or additional large model interfaces.

The camera can pan, zoom out and zoom in continuously, and all scenes exist in the same vector world. To control performance, the drawing of areas outside the screen is suspended; after the user drags the progress bar, all objects will directly return to the corresponding time point without re-random generation.

Even the most flaw-prone action of riding a bicycle is processed separately: the wheel rotation and pedal phase are driven by the travel distance, so after the character stops, there will be no situation where the bicycle has stopped but the feet are still pedaling in the air.

The first 20 minutes of the work pass through 30 cities and natural attractions, among which the Shenzhen chapter successively shows Tencent Penguin Island, Sangfor Building and Talent Park; the last 10 minutes enter the universe, traveling all the way from Mars to the Sun, Pluto, black hole, and then back to Earth.

The author also specifically noted in the work which content comes from reality and which is fictional. For example, cycling in space, small lamps that store sunlight, Pluto snowmen, and traveling through black holes to return home are all just story settings; the black hole event horizon is not packaged as a real traversable tunnel.

This work shows that the capabilities of GPT-6 Astra are truly amazing.

The model needs to first understand an open goal, then complete the page architecture, animation logic, time system, character movement, camera scheduling, scene design, performance optimization and documentation, and make all these parts remain consistent continuously in a 30-minute work.

02 Why Does Astra Make People Extra Sensitive

The initiator of the CodexRadar open-source project originally just wanted to solve his own problem: to monitor whether the model has become stupid today and the reset of quota, so that the token quota can be used to the maximum efficiency. Later, he made the monitoring tool into a community open-source project, which unexpectedly quickly attracted a group of heavy Codex users to participate in the crowdsourced testing.

The "Crowdsourced Testing Radar" of CodexRadar is set with 112 real open-source programming questions, which are run by users themselves, and the results are uploaded and then re-verified by the server in a clean environment. The official website shows that there are more than 500 participating volunteers, and the cumulative contribution has reached the level of tens of billions of Tokens.

A small tool originally designed to monitor "model intelligence" for himself can attract so many people to continuously contribute quotas, entirely because model capability fluctuation is becoming a common pain point for heavy users.

After Astra went online, this kind of fluctuation has a greater impact on work.

A large model algorithm engineer said that intelligence loss will directly affect automated tasks. "When GPT-6 loses intelligence and is restricted by rate limits, automated tasks will accumulate, and eventually concurrency conflicts will occur. If the ticket session loses intelligence, errors may spread to other places through the central system, and eventually break the entire repository."

After entering the Agent stage, one model call is often only a part of the entire task chain. If there is a obvious deviation in the judgment of a certain step, subsequent steps may continue to move forward along the wrong result. The longer the task, the more obvious this risk is.

Therefore, when the model truly enters the production process, users are no longer only concerned about the average capability. Whether there is fluctuation in capability is equally important.

"Astra is comprehensively strong, especially in computer use and multi-agent collaborative management. According to the data from the radar, Astra takes half as many steps to solve problems as the previous generation of models. The fewer steps, the stronger the model, and Astra has achieved a qualitative leap."

When releasing Astra, OpenAI placed Computer Use, professional work, software engineering and long-term Agent tasks in a very important position. Many developers in the large model field said that "Astra has a generational gap in strength".

It is precisely because of this strength that users hope to assign some more complex and longer-term automated tasks in the past to it, so "intelligence loss" will have a greater impact on the actual effect of work.

03 What Exactly Does "Intelligence Loss" Reduce

"Intelligence loss" is actually a very community-style statement. From the user's side, as long as the same model name and the same product entrance are obviously more difficult to use today than yesterday, it will be summarized as "intelligence loss".

But from an engineering perspective, the final result seen by users is determined by many layers together.

The underlying model is one layer, the inference intensity is one layer, the context management is one layer, and Harness, Skills, tool invocation, model routing, concurrency scheduling and service capacity will all affect the final effect.

So the "getting stupid" felt by users does not necessarily mean that the model weights themselves have changed.

For example, a Coding Agent that originally took the initiative to run three rounds of tests now only runs one round; it originally called multiple sub-Agents to check the results, but now reduces parallel exploration; it originally retained more complete context, but now loses some early information in long tasks. For the backend, these may correspond to completely different problems, but for users, the final physical feeling is that it does not feel as smart as before.

This is also the reason why CodexRadar emerged.

The benchmark score of the model at release can show how high the peak capability of the model is, but it is difficult to show the real-time state of the model, such as what state the Astra called at 10 a.m. today is actually in.

But for people who use Agents to do real work, this real-time state is very important.

04 The Problems Behind "Intelligence Loss"

At present, there is no public evidence to prove that OpenAI is massively and actively replacing Astra with a weaker underlying model.

But there are relatively clear signals of supply pressure after Astra went online. On September 10, OpenAI suspended new subscriptions and upgrades for the $200/month Pro 20x plan. Existing users are temporarily unaffected. This plan provides a usage quota about 20 times that of Plus, and it is also an important choice for heavy Codex users.

Astra itself is also a very expensive model.

Its standard API price is $10 per million input Tokens and $50 per million output Tokens; Fast mode can provide higher speed, and the price will double again.

The problem is that Astra's most popular capabilities are exactly very "computing power intensive".

Computer Use requires continuous understanding of the screen, execution of operations, observation of results, and then deciding the next step; multi-agent workflows may start multiple subtasks at the same time; a complete task of a Coding Agent will continuously read files, execute codes, run tests, and review results.

When a large number of heavy users start using the model in this way at the same time, the supply pressure is completely at a different magnitude from the past.

So behind this round of discussions about "intelligence loss", there is actually a mixture of model quality fluctuation, rate limit, resource scheduling, and the amplification of system stability by long tasks themselves.

Some people in the community believe that the recent problems are related to excessive demand for Astra, a large influx of Pro 20x users, and tight computing power supply. But OpenAI's suspension of new 20x subscriptions at least proves that this cutting-edge model has a very real contradiction between supply and demand.

In the past, competition among large models was more focused on who could train a stronger model. Now, after the model is trained, whether it can stably supply this intelligence to a large number of users at scale has also begun to become part of the competition.

05 Large Models Begin to Require "Intelligence SLA"

A pelican has become a star in the Agent era, which shows that users are actively monitoring "intelligence" itself. The task success rate of the model in different time periods, the stability of long tasks, and whether the same workflow can be reproduced continuously can be understood as a new kind of "Intelligence SLA".

SLA originally stands for Service Level Agreement. In cloud services, indicators such as availability rate, latency, and fault recovery time are usually agreed, such as 99.99% uptime.

The stronger the model, the more prominent this problem will be. In the future, the competition among model companies will further extend from peak capability to another level: supplying high-level intelligence stably and at scale for a long time.

Whether the model can be equally smart every day has also become a bottleneck for AI Agents to truly move into work scenarios.

This article is from "Tencent Tech", author: Xiao Jing, editor: Xu Qingyang, published with authorization from 36Kr.