Tobias Stähle¹, Péter Ferenc Gyarmati¹, Thilo Spinner¹, Rita Sevastjanova¹,
Dominik Moritz², Mennatallah El-Assady¹
¹ETH Zürich · ²Carnegie Mellon University
A context protocol to provide AI agents native access to interactive Visual Analytics applications.
Fig. 1 — AI agents interacting with Visual Analytics (VA) interfaces using VACP (Visual Analytics Context Protocol). VACP maintains the application state and available interactions informed through the different types of knowledge representation of the VA entities. To perform analytical tasks in the interface, an AI agent can use the provided capabilities to perceive the VA interface and understand possible interactions by using VACP. Further, it can interact with the VA interface through the provided interactions. Interactions (e.g., I₁ or I₂) manipulate the interface appearance, triggering a state transition (e.g., S₂→S₃) in the application within the VACP layer.
AI agents are becoming a new user group for Visual Analytics (VA). Today they mostly get there the same way browser-automation agents do: a screenshot and a raw DOM dump. That works for buttons and forms. It falls apart on a <canvas> scatterplot, a dense parallel-coordinates plot, or an SVG chart with thousands of anonymous path elements — the exact interfaces VA systems are built from.
VACP is our answer to that mismatch. It's a small in-app contract that exposes what a VA interface is (a graph of entities) and what can be done with it (a catalog of validated, intent-level actions) — so an agent can read precise state and act on it without guessing from pixels or DOM shapes. We describe the protocol, ship it as a TypeScript library, and evaluate it against current agentic baselines. Full details are in the paper; this page is the short version, with pointers to the parts of this documentation site that go deep.
The rise of AI agents introduces a fundamental shift in Visual Analytics (VA), in which agents act as a new user group. Current agentic approaches — based on computer vision and raw DOM access — fail to perform VA tasks accurately and efficiently. This paper introduces the Visual Analytics Context Protocol (VACP), a framework designed to make VA applications "agent-ready" that extends generic protocols by explicitly exposing application state, available interactions, and mechanisms for direct execution. To support our context protocol, we contribute a formal specification of AI agent requirements and knowledge representations in VA interfaces. We instantiate VACP as a library compatible with major visualization grammars and web frameworks, enabling augmentation of existing systems and the development of new ones. Our evaluation across a compelling set of VA tasks demonstrates that VACP-enabled agents achieve higher success rates in interface interpretation and execution compared to current agentic approaches, while reducing token consumption and completion time. VACP closes the gap between human-centric VA interfaces and machine perceivability, ensuring agents can reliably act as collaborative users in VA systems.
VA interfaces are designed on the assumption that a human perceives an encoding visually and interacts with a mouse. Recent agentic approaches try to reuse that same channel: computer-use agents click and drag based on screenshots, and web agents parse the DOM the way they would for a form-based site.
Both hit a wall in VA:
<canvas>-rendered chart exposes nothing; a D3.js or Vega SVG scatterplot exposes thousands of undifferentiated <path> elements. There's no reliable signal for what a mark represents, let alone what dragging it would do.In our early evaluations, this "interface mismatch" is the dominant failure mode: agents don't just get answers wrong, they frequently can't even execute the interaction they correctly decided on.
We found it useful to decompose what a VA interface encodes into four layers, from what's rendered down to why it was built that way:
| Layer | What it captures | What an agent needs to read it |
|---|---|---|
| L1 Pixel-based | The rendered view — layout, visually detectable patterns (clusters, outliers, trends) | Computer vision over a screenshot |
| L2 DOM-based | The live component tree, event listeners, retrievable values and positions | Runtime DOM access and text parsing |
| L3 Declarative grammar | The explicit spec behind the chart — encodings, interactions, data mappings (Vega-Lite, Mosaic vgplot, …) | Access to the grammar spec at runtime |
| L4 Semantic / production logic | Design rationale and intent — why a mark is encoded this way, which task it serves | Free-text parsing and NL understanding, when it's exposed at all |
Most agent tooling today only reaches L1 and L2 — and L4 is rarely exposed to anyone, human or agent. VACP's capability graph and action catalog are how we expose L3 and L4 knowledge at runtime, through two small functions: get_capabilities() for the semantic structure, and get_state() for the current values.
Fig. 2 — Overview of knowledge representation layers in VA applications and the VACP functions that perceive and act at each level. Each layer abstracts the knowledge, including the data to be analyzed, data encoding, and interface functionalities defined by production logic and design decisions.
VACP formalizes what an agent needs into eight design principles (P1–P8), grouped into two concerns: Contextualization (informing the agent) and Interactive Dynamics (letting it act reliably). They land on three protocol components and a transport layer:
Capabilities
A structured description of what exists in the interface and what can be done.
State
Read Selections and Params as values keyed by stable refs.
Actions
Execute intent-level interactions with validated parameters.
Transports
Reach the in-page contract from an external agent or in-app tooling.
get_capabilities(), including parameter types, required/optional flags, and valid ranges (P5). It updates dynamically as the application state changes, so the catalog never offers an action that's no longer legal (P6).execute_interaction(), that resolves the reference, validates the interaction is still legal for the current state, checks parameters, and applies the update synchronously (P8). Success returns a structured result; failure returns a structured, correctable error. Data access is tailored rather than bulk: state exposes structure and availability, with get_schema() and inspect_data() for on-demand queries (commonly backed by DuckDB), so raw datasets never have to be serialized into agent context (P7).
Fig. 3 — Example VACP application-state representation: the capability graph captures semantic structure and interaction-relevant relations; the state snapshot stores currently active values under the same stable references.
Read the full breakdown of state, actions, and the execution gateway in core runtime, and the concrete tool surface in the tool contract reference.
We ship VACP as a TypeScript library that decouples the semantic contract from the DOM layout, rendering backend, and UI framework. It ships adapters for common declarative grammars and a plain API for custom imperative charts:
import { installVacpOnVegaLiteView } from '@vacp/vega-lite';
import vegaEmbed from 'vega-embed';
const { view } = await vegaEmbed('#chart-container', vegaLiteSpec);
// Connects the rendered chart to the agent-facing capability graph and state
installVacpOnVegaLiteView({
root: document.getElementById('chart-container'),
view,
spec: vegaLiteSpec,
});Retrofitting a declarative Vega-Lite or Mosaic vgplot chart takes on the order of tens of lines of adapter code with zero hand-written semantic-action code — the adapter derives it from the spec. A fully custom React + D3.js chart takes more integration work, since there's no existing declarative model to parse, but the contract stays the same. See Providers for the adapter pattern and Transports for how an external agent reaches the same in-page contract over MCP.
Fig. 4 — Overview of the evaluation pipeline: (a) the overall task execution and evaluation workflow, (b) the agent setup and the six evaluated context scenarios S1–S6.
We built a benchmark of five VA use cases (bubble charts with temporal animation, coordinated multi-view dashboards, geospatial node-link diagrams, and a high-dimensional parallel-coordinates plot, across both declarative-grammar and custom React/D3.js implementations), validated with a 7-participant expert study (85.71% human task-completion rate, confirming the tasks are solvable and representative).
Fig. 5 — Overview of the VA interfaces for the five defined use cases spanning declarative grammars (Vega-Lite, Mosaic vgplot) and custom imperative code (React + D3.js).
We then ran four current frontier models — GPT-5.2, Claude Sonnet 4.5, Gemini 3 Pro Preview, and Gemini 3.1 Pro — across six scenarios, from current state-of-the-art approaches (S1, mirroring today's general-purpose web agents) up to full VACP access with visual and DOM fallback (S6):
| Scenario | Description |
|---|---|
| S1 UI + DOM | Mirroring current general-purpose web agents, agents see only screenshots (L1) and raw DOM (L2). Complex visualizations (Vega-Lite/Mosaic) are rendered as canvas elements, so their data is hidden from DOM parsers — only the PCP use case uses SVG. Agents act via emulated mouse events and JavaScript execution. |
| S2 UI + DOM + State | Building on L1 and L2, we inject the instantaneous VACP application state into the DOM as serialized JSON in a hidden node. Agents can discover and access it via DOM queries. |
S3 MCP (with inspect_data) | To compare VACP against simpler structured tool access, agents interact exclusively via use-case-specific static tools from an MCP server. This provides state, interactions, and inspect_data(), but no visual and DOM access. |
| S4 VACP (State + Interaction Gateway) | This ablation removes visual and DOM access. Agents interact exclusively via tools from the VACP MCP server (L3, L4), relying entirely on the semantic API — testing the VACP schema's expressiveness without visual grounding. |
S5 VACP + UI + DOM (no inspect_data) | Agents receive the full toolset (L1–L4) except inspect_data(), letting us study the contribution of interaction grounding and data access through inspect_data(). |
| S6 VACP + UI + DOM | Agents receive the full toolset (L1–L4): VACP tools, DOM access, and screenshots — letting us study synergies or conflicts between visual perception and semantic data retrieval. |
The visual gap is real and large. S1 scored 28–73% completion across models — agents routinely couldn't extract precise values from canvas or SVG charts. Injecting state into the DOM (S2) barely moved the needle (max. +4.4%). Scenarios built on VACP's semantic interface (S4, S6) reached near-perfect completion across models and tasks.
Fig. 6 — Summary of the agent evaluation across use cases and tasks. Each tile shows successful runs out of three; percentages give overall success rate per model across scenarios S1–S6.
It's also cheaper and faster. S4 used far fewer tokens and less wall-clock time than S1, S2, S5, or S6 — Claude Sonnet 4.5 needed under 2,000 tokens on average in S4, versus over 30,000 for the same tasks on GPT-5.2 in a visually-grounded scenario. Agents that also got inspect_data (S6 vs. S5) completed more tasks and used fewer tokens than agents forced to reverse-engineer values from the DOM.
Fig. 7 — Median token consumption and execution time per scenario, averaged across use cases and models. S4 (VACP) clusters at low token counts with high completion rates.
Looking at interaction traces (pragmatic vs. syntactic vs. semantic actions, following Buxton's taxonomy), agents in S1 operated almost entirely at the pragmatic/syntactic level (semantic actions: ~4%). With VACP alone (S4) that flips to ~99.8% semantic. In the hybrid S6, the strongest agents used semantic actions for reasoning and dropped to the pragmatic layer specifically to visually double-check a result before answering — a genuinely human-like verification pattern.
Fig. 8 — Claude Sonnet 4.5 solving Task UC1L in S6: it inspects the screenshot and VACP state, sets the year, verifies the UI, queries the data, hovers to confirm "Japan, 81.57", and answers correctly.
VACP's current benchmark deliberately targets atomic, elementary analytical tasks (locate, identify, compare) to isolate perception and execution failures — we're not claiming it already supports open-ended exploratory workflows, just that it provides the operational foundation for them. Two other honest caveats: restricting an agent purely to L3/L4 semantic context loses visual nuance (clustering, Gestalt effects, dense-pattern perception) that screenshots still add value for; and adopting VACP is real developer effort — cheap for declarative-grammar charts, more involved for fully custom imperative visualizations.
Where this points next: generalizing the core principles (dynamic app state, available interactions, precise execution) beyond the web to other data-rich environments like GIS or 3D modeling, and studying how autonomous agents adapt their analysis strategy once interaction access stops being the bottleneck.
A selection of the work VACP builds on and relates to — the full reference list is in the paper:
Everything referenced in this blog post — task definitions, ground-truth solutions, full per-model results, and the annotated agent traces — is documented in more depth outside this page:
vacp_capabilities / vacp_state / vacp_execute surface shared across transports.For the full methodology, task definitions, and per-model results, see the paper and the source on GitHub.