Browser annotation

Annotate elements in the embedded browser to create structured descriptions for coding agents.

Overview

Browser Annotation lets you select any element on a web page, capture its location and attributes, add a note, and export everything as a structured Markdown payload ready to paste into an AI coding agent.

This bridges the gap between seeing a UI issue and describing it precisely:

URL: https://example.com/settings
Tag: BUTTON
Primary Selector: #save-settings
Element Role: primary action button
UI Region: Profile
Ancestor Context: form section
Bounds: x=812 y=64 w=92 h=36
Note: Make this the primary CTA, increase horizontal padding.

Using Annotation Mode

Entering Annotation Mode

1. Open the Browser panel
2. Click the pointer icon (MousePointer2) in the browser toolbar
3. The button turns highlighted to indicate active mode
4. The cursor changes to a crosshair

Selecting an Element

1. Hover over any element to see a highlight overlay and element label
2. Click on an element to select it
3. A popup appears with:
   • Element tag name
   • Selector preview
   • Element dimensions
   • Note textarea

Adding a Note

1. Type your annotation in the textarea
2. Click Copy Annotation to copy the full structured output to clipboard, or Send to agent to open a Clanker preview
3. In the preview, edit the message and select a named agent marked Ready or Open · status unverified. The latter means the agent process is still open but optional turn tracking is off; check its pane before sending. Agents known to be running, requesting input, closed, or in their fallback shell cannot receive an automatic send. Copy message remains available.
4. Click Cancel to deselect without copying or sending

Exiting Annotation Mode

• Press Escape anywhere on the page
• Click the pointer icon again in the toolbar
• Navigate to a different page (annotation mode resets on page load)

Captured Data

Each annotation captures:

Selector Strategy

The annotation system uses a priority-based selector strategy:

1. data-testid / data-test / data-qa — Most stable for testing
2. id — Usually stable
3. role + aria-label — Semantic selectors
4. Tag + class — Used only when it identifies one element and the class does not look generated
5. Structural nth-of-type path — Last resort when classes repeat

Fallback selectors are included only when they identify the selected element. Positional paths can change when a page layout changes, so check the captured text and context before using one later.

Context Extraction

The annotation system analyzes the element's surroundings to provide richer context:

• UI Region: Nearest heading, aria-label, or title above the element
• Ancestor Context: Inferred position (left sidebar, form section, navigation, etc.)
• Element Role In Context: What the element means locally (primary action button, repository list entry, etc.)
• Nearby Text: Sibling text that helps identify the element's purpose

The handoff and copy actions include the same captured context. Form field values are excluded from captured attributes.

Each Copy or Send click stores its own annotation snapshot. Clanker processes those clicks in order, even when several happen between browser polls.

This is especially useful for complex pages like GitHub, where you might want to distinguish between "Save" buttons in different sections.

Example Output

## Page Annotation
- URL: https://github.com/settings/profile
- Title:Public profile · GitHub
- Captured At: 2026-04-13T15:04:00Z

### Selected Element
- Tag: BUTTON
- Primary Selector: `button[data-testid="public-profile-save"]`
- Fallback Selectors: `.btn-primary`, `.js-edit-profile`
- Element Role: primary action button
- UI Region: Public profile
- Element Role In Context: action button
- Nearby Text: `Cancel`
- Ancestor Context: form section
- Bounds: x=812 y=340 w=120 h=36
- Attributes: type="button", data-testid="public-profile-save"
- Accessible Name: Save changes

### Annotation
Make this button more prominent — it's the primary CTA for the profile form.

Workflow with AI Agents

1. Browse to the page with the issue
2. Enable annotation mode
3. Click the element you're describing
4. Add your note
5. Copy the annotation and paste it into an agent, or choose Send to agent
6. Review the message and destination workspace before sending

Send to agent also works for remote SSH agent terminals that Agent Attention has registered and that are ready to receive input; it writes into the existing terminal and opens no new SSH connection. If the agent is closed, busy, a plain shell, or outside the workspace, use Copy message and paste it yourself.

The structured output helps the agent understand exactly which element you mean, its position, and your intended change.

Architecture

┌─────────────────────────────────────────────────────────────┐
│ Main Process                                                │
│                                                              │
│  AnnotationController                                        │
│  ├── enable(workspaceId) → inject runtime + arm             │
│  ├── disable() → cleanup runtime                            │
│  ├── capture() → return-value from page                     │
│  ├── reinitialize() → re-inject after navigation            │
│  └── formatAnnotationMarkdown() → clipboard export           │
│                                                              │
│  AnnotationIPC                                              │
│  ├── ANNOTATION_ENABLE                                      │
│  ├── ANNOTATION_DISABLE                                     │
│  ├── ANNOTATION_CAPTURE                                     │
│  ├── ANNOTATION_EXPORT                                      │
│  ├── ANNOTATION_PREPARE_SEND                                │
│  └── ANNOTATION_ESCAPE                                      │
│                                                              │
│  Escape Handler (before-input-event)                         │
│  └── Two-layer: main process + injected runtime              │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│ Browser Content (WebContentsView)                           │
│                                                              │
│  Injected Runtime                                           │
│  ├── Crosshair cursor on enable                             │
│  ├── Hover highlight (position: fixed, z-index)             │
│  ├── Element capture (selector, bounds, context)             │
│  ├── In-page popup (note textarea, Copy/Send/Cancel)         │
│  └── Annotation storage (__clankerAnnotationData__)         │
└─────────────────────────────────────────────────────────────┘

Key Implementation Details

• Injected runtime: JavaScript injected via webContents.executeJavaScript()
• No browser preload: Communication uses return-value capture
• Trusted handoff preview: The page only signals a request. Clanker captures the annotation, displays an editable preview, and rechecks the destination agent in main before writing to its PTY.
• In-page popup: Renders in page DOM with proper z-index (2147483647)
• Escape handling: Two-layer detection (main process + injected runtime)
• Navigation handling: Runtime re-injected on subsequent navigations
• Double-injection guard: Runtime checks window.__clankerAnnotation__

Known Limitations

IPC Channels

Future Enhancements

Potential additions for future versions:

• Screenshots: Element or viewport capture
• Multi-select: Annotate multiple elements at once
• Persistent storage: Save annotations per page/session
• Source mapping: Map selected elements to source code