Is Prompt Engineering Dead? Google DeepMind Proposes a New Paradigm of "Visual Prompt Engineering": More Effective Than Text Instructions

Prompt engineering (prompt engineering) has become a key means to improve the performance of large language models. The current consensus is that as long as the input space of the model is sufficiently rich, there is an opportunity to obtain better downstream performance by optimizing the input.

Then, is this principle only applicable to language models, or can it also be extended to other modalities, such as the computer vision field?

In a latest paper, Google DeepMind proposed "Visual Prompt Engineering" (VIPE), which successfully extends prompt engineering to the vision domain.

Paper link: https://arxiv.org/abs/2607.25537

The experimental results show that the visual prompt engineering method can improve video reasoning performance on multiple tasks, and for video models, this method is sometimes more effective than classic text prompt engineering or test-time scaling.

Just as text prompt engineering can systematically improve the performance of language models, visual prompt engineering can also be used as a simple, computationally efficient method to stimulate better visual reasoning performance of video models.

The research team stated that visual prompt engineering points to a new paradigm, that is, "visual prompts themselves can also be optimized just like text prompts". As cutting-edge video models receive richer multimodal context, visual prompt engineering may expand from single image editing to multiple reference images and even videos.

Visual Prompt Engineering: Solving the Same Problem in New Visual Scenarios

Visual prompt engineering is a prompt engineering method for the visual input of video models. Without changing the task logic, it converts the original image into a visual form that the model can more easily understand and reason about, so that the model can handle the same problem in new visual scenarios. For example, the abstract schematic diagram of a ball and a slope can be changed to a realistic scene of a billiard ball and a wooden board, with the key spatial relationship kept consistent.

Figure | Visual prompt engineering starts with a visual prompt, such as a sketch image of a ball and a slope, as well as a text prompt like "the ball rolls down along... and finally falls into one of the buckets".

The specific process is as follows:

1. Conceptualizer: Responsible for proposing how to modify the image. It generates a natural language editing instruction based on the task samples and constraints. The instruction will clearly specify which task-related elements must be retained, such as the number, position, orientation and relative relationship of objects

2. Editor: Modify the original image according to the editing instructions to generate one or more candidate images. This step can be completed by an image editing model or by manual editing.

3. Filter: An optional step. When the editor generates multiple candidate images, the filter will select a more appropriate version from them. The filter will give priority to candidate versions with better image quality and retaining the key task information of the original image.

More Effective Than Text Instructions

The research team tested and verified automated visual prompt engineering on the VPCT physical reasoning task. Overall, visual prompt engineering can improve the visual reasoning performance of video models, and optimizing visual context is usually more effective than optimizing text instructions.

Visual Realism Is What Actually Drives Performance Improvement

After using visual prompt engineering, the accuracy of Veo 3.1 increased from 41.3% to 59.3%, and that of Omni Flash increased from 56.3% to 67.5%. Video models systematically prefer photorealistic context rather than abstract input.

Figure | Visual prompt engineering improves physical reasoning ability on VPCT.

Can Be Automated and Serves as an Effective Test-time Scaling Strategy

The automated process of visual prompt engineering uses a vision-language model (VLM) to generate rewriting instructions, an image editing model to generate candidate images, and an automatic scorer to be responsible for filtering or feedback iteration. The research team compared two routes: free-form conceptualization and Atomic Concept Editing (ACE). The results show that both methods can automatically generate better rewritten images; free-form visual prompt engineering reduces the error rate by more than 75% on some tasks. On the Sort 3 Numbers task, if the best visual version is selected per sample, the error rate can be reduced to 0.

Figure | Automated visual prompt engineering improves performance on a variety of tasks.

In the physical reasoning task of a rolling ball, without using visual prompt engineering, Veo 3.1 increased the accuracy from 41.3% to 50.0% through multiple samplings; after adding visual prompt engineering, the accuracy of single sampling reached 59.3%. If multiple samplings are continued on the visual prompts rewritten by visual prompt engineering, the accuracy can reach 68.0%.

Figure | Visual prompt engineering can also be combined with test-time scaling to further improve performance.

Optimizing Visual Context Is Often More Effective Than Optimizing Text Instructions

Take Sort 3 Numbers as an example, the accuracy of the optimal text prompt and the optimal image prompt are 86% and 76% respectively.

Figure | For video models, visual prompt engineering may be more effective than text prompt engineering.

No Systematic Improvement in the Reasoning Performance of Native Image Generation Models

The research team tested models such as Nano BananaPro and Nano Banana 2. The results show that after replacing the input with photorealistic images, the reasoning performance of the image models does not improve stably. This may be because native image generation models cover more visual styles and are more familiar with both abstract sketches and real images.

Figure | Results of native image generation models on VPCT, reporting the best prompt results for each configuration.

Effective Image Rewriting Usually Has Three Characteristics: More Realistic Scenes, More Prominent Targets, and More Direct Semantics

This kind of rewriting can convert abstract sketches into visual scenes closer to the real world, thereby reducing the gap between abstract inputs and the visual representations familiar to video models. For example, on the VPCT task, after rewriting the sketch into a photorealistic scene, the accuracy of Veo 3.1 increased to 59.3%. Therefore, when the same task can be converted into a more realistic visual scene, the realistic scene version should be prioritized.

Figure | Every step towards realism improves generation consistency.

Of course, the current method also has certain limitations.

For example, the current visual prompt engineering method relies on the quality of the image editing model. If the editor changes task information or introduces artifacts, the downstream video model may not be able to solve the original task. In the future, if image editing models can more stably retain task information, additional filtering and quality control may no longer be needed.

At the same time, visual prompt engineering will bring additional costs. However, image editing is usually much cheaper than video generation, so this cost may also be converted into an advantage in practice. One cheap image editing call can reduce the probability of failure of one expensive video generation.

This article is from the WeChat Official Account "SciTouTiao" (ID: SciTouTiao), authored by SciTouTiao, and authorized for release by 36Kr.