DeepMind Visual Prompt Engineering (VIPE) Workflow for Video Models

You are a Visual Prompt Engineering (VIPE) Planner. Analyze the input visual content and reasoning requirements to produce image rewriting instructions for an image editing pipeline.

Input Conditions:
1. Original Image Type: {{image_type}}
2. Target Reasoning Task: {{reasoning_task}}
3. Transformation Strategy (Free-form Conceptualization or Atomic Concept Editing ACE): {{transformation_strategy}}

Generation Requirements:
- Modify only visual representation qualities (such as photorealism, textures, lighting) without changing physical geometry, relative object positions, or spatial boundaries.
- Explicitly state which elements to edit and which geometric constraints must remain untouched.

Output Format (JSON):
{
  "rewrite_instruction": "<Detailed image editing prompt>",
  "preserved_constraints": "<Description of precise spatial and physical constraints to preserve>"
}

Best for

Visual Prompt Engineering (VIPE) is designed for video and multimodal foundation models handling visual and physical reasoning tasks (such as predicting ball trajectories). It converts abstract line drawings or rough sketches into photorealistic, detail-rich visual prompts without altering spatial constraints or task logic [1][3].

Workflow Steps

1. VLM Rewriting Instruction Generation (Planner): A Vision-Language Model analyzes the input diagram or sketch and generates explicit image editing prompts based on Free-form or ACE strategies while enforcing strict spatial preservation [1][2].
2. Image Editing Execution (Editor): An image editing model applies the visual instructions to convert abstract inputs into realistic scenes while maintaining physical geometry [1][3].
3. Automatic Scorer Filtering (Filter): An automated filter evaluates candidate images and discards any samples that violate spatial constraints, passing only verified visual prompts to the downstream video model (such as Veo 3.1) [1][3].

Variables

• image_type: The format of the source visual input, such as line sketches or abstract diagrams.
• reasoning_task: The visual reasoning objective, such as physical trajectory prediction or spatial collision forecasting.
• transformation_strategy: The visual transformation approach, selecting either Free-form Conceptualization or Atomic Concept Editing (ACE).

How to Use

Pass the source diagram and target variables into the VLM Planner prompt. Route the output rewriting instruction to the image editor model, filter out geometrically distorted outputs using the automated scorer, and supply the photorealistic visual prompt to the video foundation model. On physical trajectory prediction tasks, VIPE boosts single-sample Veo 3.1 accuracy from 41.3% to 59.3%, reaching 68.0% when paired with test-time sampling [1].

Quality Checks

• Spatial Preservation Check: Verify that object coordinates, trajectory angles, and spatial ratios remain identical to the original input sketch.
• Artifact Prevention: Ensure the image editor does not introduce misleading visual noise or extraneous elements that degrade model reasoning.
• Pipeline Complexity Warning: Monitor VLM and image editor execution latency to balance workflow overhead against reasoning accuracy gains [3].

Sources

• arXiv / Google DeepMind (2026-07-28): Visual prompt engineering for video models [1]
• Hugging Face (2026-07-29): Visual prompt engineering for video models [2]
• 36Kr (2026-07-30): Google DeepMind Proposes Visual Prompt Engineering [3]