Tommy D. Rossi
802e6768e2
- getAriaSnapshot() discovers aria refs and caches ElementHandles - showAriaRefLabels() overlays yellow badges on interactive elements - hideAriaRefLabels() removes labels from page - test captures screenshots from HN, Google, GitHub
353 lines
12 KiB
Markdown
353 lines
12 KiB
Markdown
# Plan: Visual Aria Ref Labels (Vimium-style)
|
|
|
|
## Goal
|
|
|
|
Add functions to overlay visual ref labels on interactive elements, allowing agents to:
|
|
1. Take a screenshot with visible ref labels (e.g., `[e1]`, `[e2]`, `[e3]`)
|
|
2. See which elements are interactive and their ref IDs
|
|
3. Use `page.locator('aria-ref=e5')` to interact with labeled elements
|
|
|
|
## Reference Implementation
|
|
|
|
Based on `/Users/morse/Documents/GitHub/unframer-private/autofill-extension/src/content/`:
|
|
|
|
### gui.js - Container setup
|
|
```js
|
|
const guiRoot = document.createElement('div')
|
|
guiRoot.style = `
|
|
position: absolute;
|
|
left: 0;
|
|
top: 0;
|
|
width: 100%;
|
|
height: 100%;
|
|
z-index: 2147483647; // Max z-index to be on top of everything
|
|
pointer-events: none; // Don't block clicks on underlying elements
|
|
`
|
|
document.documentElement.appendChild(guiRoot)
|
|
```
|
|
|
|
### HintRenderer.tsx - Label positioning
|
|
```tsx
|
|
// Position label at element's top-left corner, accounting for scroll
|
|
left: `${window.scrollX + rect.left - 16}px`
|
|
top: `${window.scrollY + rect.top - 16}px` // 16px above element
|
|
```
|
|
|
|
### Vimium-style CSS
|
|
```css
|
|
font-family: Helvetica, Arial, sans-serif;
|
|
font-weight: bold;
|
|
font-size: 12px;
|
|
padding: 0px 2px;
|
|
background: -webkit-gradient(linear, left top, left bottom, color-stop(0%,#FFF785), color-stop(100%,#FFC542));
|
|
border: 1px solid #E3BE23;
|
|
border-radius: 4px;
|
|
color: black;
|
|
text-shadow: rgba(255, 255, 255, 0.6) 0px 1px 0px;
|
|
```
|
|
|
|
## Existing Code Context
|
|
|
|
### `getAriaSnapshot()` in `playwriter/src/aria-snapshot.ts`
|
|
|
|
Already implemented - discovers refs and provides lookup:
|
|
|
|
```ts
|
|
export async function getAriaSnapshot({ page }: { page: Page }): Promise<AriaSnapshotResult> {
|
|
// 1. Calls page._snapshotForAI() to populate Playwright's internal ref tracking
|
|
// 2. Probes aria-ref=e1, e2, e3... to discover all refs
|
|
// 3. Caches ElementHandles in refHandles array
|
|
// 4. Returns { snapshot, refToElement, refHandles, getRefsForLocators, ... }
|
|
}
|
|
```
|
|
|
|
The `refHandles` array contains `{ ref: string, handle: ElementHandle }` for each interactive element. We'll use these handles to get bounding boxes and create labels.
|
|
|
|
### Which elements get refs?
|
|
|
|
Playwright's `_snapshotForAI()` uses `mode: 'ai'` which sets `refs: 'interactable'`:
|
|
|
|
```ts
|
|
// From playwright/packages/injected/src/ariaSnapshot.ts
|
|
function computeAriaRef(ariaNode, options) {
|
|
if (options.refs === 'interactable' && (!ariaNode.box.visible || !ariaNode.receivesPointerEvents))
|
|
return; // Skip non-interactive elements
|
|
// ... assign ref like 'e1', 'e2', etc.
|
|
}
|
|
```
|
|
|
|
**Included:** buttons, links, inputs, textareas, selects, checkboxes, radios, sliders, any clickable element
|
|
**Excluded:** hidden elements, static text, decorative images, elements behind overlays
|
|
|
|
## API Design
|
|
|
|
```ts
|
|
// Show Vimium-style labels on all interactive elements
|
|
export async function showAriaRefLabels({ page }: { page: Page }): Promise<{
|
|
snapshot: string // The accessibility snapshot text
|
|
labelCount: number // Number of labels shown
|
|
}>
|
|
|
|
// Remove all labels from the page
|
|
export async function hideAriaRefLabels({ page }: { page: Page }): Promise<void>
|
|
```
|
|
|
|
## Implementation
|
|
|
|
Add to `playwriter/src/aria-snapshot.ts`:
|
|
|
|
```ts
|
|
const LABELS_CONTAINER_ID = '__playwriter_labels__'
|
|
|
|
// Use css`` tagged template for IDE syntax highlighting (identity function)
|
|
const css = String.raw
|
|
|
|
const LABEL_STYLES = css`
|
|
.__pw_label__ {
|
|
position: absolute;
|
|
font: bold 11px Helvetica, Arial, sans-serif;
|
|
padding: 1px 4px;
|
|
background: linear-gradient(to bottom, #FFF785 0%, #FFC542 100%);
|
|
border: 1px solid #E3BE23;
|
|
border-radius: 3px;
|
|
color: black;
|
|
text-shadow: 0 1px 0 rgba(255, 255, 255, 0.6);
|
|
white-space: nowrap;
|
|
}
|
|
`
|
|
|
|
const CONTAINER_STYLES = css`
|
|
position: absolute;
|
|
left: 0;
|
|
top: 0;
|
|
z-index: 2147483647;
|
|
pointer-events: none;
|
|
`
|
|
|
|
export async function showAriaRefLabels({ page }: { page: Page }) {
|
|
// Get snapshot and cached element handles
|
|
const { snapshot, refHandles } = await getAriaSnapshot({ page })
|
|
|
|
// Single evaluate call: create container, styles, and all labels
|
|
// Pass ElementHandles which get unwrapped to DOM elements in browser context
|
|
const labelCount = await page.evaluate(
|
|
({ refs, containerId, containerStyles, labelStyles }) => {
|
|
// Remove existing labels if present (idempotent)
|
|
document.getElementById(containerId)?.remove()
|
|
|
|
// Create container - absolute positioned, max z-index, no pointer events
|
|
const container = document.createElement('div')
|
|
container.id = containerId
|
|
container.style.cssText = containerStyles
|
|
|
|
// Inject Vimium-style CSS
|
|
const style = document.createElement('style')
|
|
style.textContent = labelStyles
|
|
container.appendChild(style)
|
|
|
|
// Create label for each interactive element
|
|
let count = 0
|
|
for (const { ref, element } of refs) {
|
|
const rect = element.getBoundingClientRect()
|
|
|
|
// Skip elements with no size (hidden)
|
|
if (rect.width === 0 || rect.height === 0) continue
|
|
|
|
const label = document.createElement('div')
|
|
label.className = '__pw_label__'
|
|
label.textContent = ref
|
|
|
|
// Position above element, accounting for scroll
|
|
// Use scrollX/scrollY so labels scroll with the page
|
|
label.style.left = `${scrollX + rect.left}px`
|
|
label.style.top = `${scrollY + Math.max(0, rect.top - 16)}px`
|
|
|
|
container.appendChild(label)
|
|
count++
|
|
}
|
|
|
|
document.documentElement.appendChild(container)
|
|
return count
|
|
},
|
|
{
|
|
refs: refHandles.map(({ ref, handle }) => ({ ref, element: handle })),
|
|
containerId: LABELS_CONTAINER_ID,
|
|
containerStyles: CONTAINER_STYLES,
|
|
labelStyles: LABEL_STYLES,
|
|
}
|
|
)
|
|
|
|
return { snapshot, labelCount }
|
|
}
|
|
|
|
export async function hideAriaRefLabels({ page }: { page: Page }) {
|
|
await page.evaluate(
|
|
(id) => document.getElementById(id)?.remove(),
|
|
LABELS_CONTAINER_ID
|
|
)
|
|
}
|
|
```
|
|
|
|
## Scroll Behavior
|
|
|
|
Labels use `position: absolute` with `scrollX/scrollY` offsets:
|
|
- Labels stay positioned on their elements when user scrolls
|
|
- Labels move with the document (not fixed to viewport)
|
|
- This matches Vimium's behavior
|
|
|
|
## Export from index.ts
|
|
|
|
```ts
|
|
export { getAriaSnapshot, showAriaRefLabels, hideAriaRefLabels } from './aria-snapshot.js'
|
|
```
|
|
|
|
## Test Implementation
|
|
|
|
Add to `playwriter/src/mcp.test.ts`:
|
|
|
|
```ts
|
|
it('should show aria ref labels and capture in screenshot', async () => {
|
|
const browserContext = getBrowserContext()
|
|
const serviceWorker = await getExtensionServiceWorker(browserContext)
|
|
|
|
// Create page with interactive elements
|
|
const page = await browserContext.newPage()
|
|
await page.setContent(`
|
|
<html>
|
|
<body style="padding: 50px; font-family: sans-serif;">
|
|
<h1>Test Page</h1>
|
|
<button id="submit-btn">Submit Form</button>
|
|
<a href="/about" style="margin-left: 20px;">About Us</a>
|
|
<input type="text" placeholder="Enter your name" style="margin-left: 20px;" />
|
|
<select style="margin-left: 20px;">
|
|
<option>Option 1</option>
|
|
<option>Option 2</option>
|
|
</select>
|
|
</body>
|
|
</html>
|
|
`)
|
|
await page.bringToFront()
|
|
|
|
// Enable extension for this tab
|
|
await serviceWorker.evaluate(async () => {
|
|
await globalThis.toggleExtensionForActiveTab()
|
|
})
|
|
await new Promise(r => setTimeout(r, 400))
|
|
|
|
// Connect via CDP
|
|
const browser = await chromium.connectOverCDP(getCdpUrl({ port: TEST_PORT }))
|
|
const cdpPage = browser.contexts()[0].pages().find(p => p.url().startsWith('about:'))
|
|
expect(cdpPage).toBeDefined()
|
|
|
|
// Import and show labels
|
|
const { showAriaRefLabels, hideAriaRefLabels } = await import('./aria-snapshot.js')
|
|
const { snapshot, labelCount } = await showAriaRefLabels({ page: cdpPage! })
|
|
|
|
console.log('Snapshot:', snapshot.slice(0, 200))
|
|
console.log('Label count:', labelCount)
|
|
|
|
expect(labelCount).toBeGreaterThan(0)
|
|
expect(snapshot).toContain('button')
|
|
expect(snapshot).toContain('link')
|
|
|
|
// Verify labels are in DOM
|
|
const labelElements = await cdpPage!.evaluate(() =>
|
|
document.querySelectorAll('.__pw_label__').length
|
|
)
|
|
expect(labelElements).toBe(labelCount)
|
|
|
|
// Verify label text matches refs
|
|
const labelTexts = await cdpPage!.evaluate(() =>
|
|
Array.from(document.querySelectorAll('.__pw_label__')).map(el => el.textContent)
|
|
)
|
|
expect(labelTexts.every(t => /^e\d+$/.test(t!))).toBe(true)
|
|
console.log('Label texts:', labelTexts)
|
|
|
|
// Take screenshot with labels visible
|
|
const screenshot = await cdpPage!.screenshot({ type: 'png' })
|
|
expect(screenshot.length).toBeGreaterThan(5000) // Not empty
|
|
|
|
// Save screenshot for visual verification
|
|
const fs = await import('node:fs')
|
|
const path = await import('node:path')
|
|
const snapshotsDir = path.join(path.dirname(new URL(import.meta.url).pathname), 'snapshots')
|
|
if (!fs.existsSync(snapshotsDir)) fs.mkdirSync(snapshotsDir, { recursive: true })
|
|
fs.writeFileSync(path.join(snapshotsDir, 'aria-labels-screenshot.png'), screenshot)
|
|
console.log('Screenshot saved to snapshots/aria-labels-screenshot.png')
|
|
|
|
// Test that labels can be used to find elements
|
|
const buttonLabel = labelTexts.find(t => t === 'e2') // Typically button is e2
|
|
if (buttonLabel) {
|
|
const buttonViaRef = cdpPage!.locator(`aria-ref=${buttonLabel}`)
|
|
const buttonText = await buttonViaRef.textContent()
|
|
console.log(`Button via ${buttonLabel}:`, buttonText)
|
|
}
|
|
|
|
// Cleanup
|
|
await hideAriaRefLabels({ page: cdpPage! })
|
|
|
|
// Verify labels removed
|
|
const labelsAfterHide = await cdpPage!.evaluate(() =>
|
|
document.getElementById('__playwriter_labels__')
|
|
)
|
|
expect(labelsAfterHide).toBeNull()
|
|
|
|
await browser.close()
|
|
await page.close()
|
|
}, 60000)
|
|
```
|
|
|
|
## Visual Result
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ │
|
|
│ Test Page │
|
|
│ │
|
|
│ [e2] [e3] [e4] [e5] │
|
|
│ ┌──────────────┐ ┌──────────┐ ┌───────────────┐ ┌────────┐ │
|
|
│ │ Submit Form │ │ About Us │ │ Enter name... │ │Option 1│ │
|
|
│ └──────────────┘ └──────────┘ └───────────────┘ └────────┘ │
|
|
│ button link textbox combobox │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
Yellow Vimium-style badges positioned 16px above each interactive element.
|
|
Agent sees screenshot → identifies `[e3]` is the "About Us" link → uses `page.locator('aria-ref=e3').click()`
|
|
|
|
## Files to Modify
|
|
|
|
| File | Change |
|
|
|------|--------|
|
|
| `playwriter/src/aria-snapshot.ts` | Add `showAriaRefLabels()`, `hideAriaRefLabels()`, `LABELS_CONTAINER_ID` constant |
|
|
| `playwriter/src/index.ts` | Add exports: `showAriaRefLabels`, `hideAriaRefLabels` |
|
|
| `playwriter/src/mcp.test.ts` | Add screenshot test |
|
|
| `playwriter/src/snapshots/` | Directory for test screenshots (gitignored) |
|
|
|
|
## Usage Example (Agent Workflow)
|
|
|
|
```ts
|
|
// 1. Show labels on page
|
|
const { snapshot, labelCount } = await showAriaRefLabels({ page })
|
|
console.log(`Found ${labelCount} interactive elements`)
|
|
|
|
// 2. Take screenshot - labels are visible
|
|
const screenshot = await page.screenshot()
|
|
// Agent analyzes screenshot, sees [e5] on "Submit" button
|
|
|
|
// 3. Interact using ref
|
|
await page.locator('aria-ref=e5').click()
|
|
|
|
// 4. Hide labels when done
|
|
await hideAriaRefLabels({ page })
|
|
```
|
|
|
|
## Edge Cases to Handle
|
|
|
|
1. **Elements near top edge** - Label would go off-screen, position below element instead
|
|
2. **Overlapping labels** - Could add offset, but keep simple for now
|
|
3. **Very long pages** - Labels use absolute positioning, works with scroll
|
|
4. **Dynamic content** - Call `showAriaRefLabels` again to refresh after DOM changes
|
|
5. **Iframes** - Current implementation only handles main frame (could extend later)
|