playwriter
Code Issues

screenshotWithAccessibilityLabels

Browse Source
This commit is contained in:
Tommy D. Rossi
2026-01-05 10:11:36 +01:00
parent da24f0e011
commit 09a363d2dc
6 changed files with 874 additions and 789 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+13 -12
View File
@@ -95,24 +95,25 @@ server.close()
### Visual Aria Ref Labels
Playwriter includes Vimium-style visual labels that overlay interactive elements, making it easy for AI agents to identify and click elements from screenshots. These functions are available in the MCP context.
Playwriter includes Vimium-style visual labels that overlay interactive elements, making it easy for AI agents to identify and click elements from screenshots. The `screenshotWithAccessibilityLabels` function is available in the MCP context.
```javascript
// Show labels on all visible interactive elements
const { snapshot, labelCount } = await showAriaRefLabels({ page })
console.log(`Showing ${labelCount} labels`)
// Take a screenshot with labels overlaid on interactive elements
// Labels are shown, screenshot is captured, then labels are removed
await screenshotWithAccessibilityLabels({ page })
// Image and accessibility snapshot are automatically included in MCP response
// Take a screenshot with labels visible
await page.screenshot({ path: '/tmp/labeled-page.png' })
// Use the snapshot to find elements, then interact using aria-ref selector
// Use the snapshot from the response to find elements, then interact using aria-ref selector
await page.locator('aria-ref=e5').click()
// Labels auto-hide after 30 seconds, or remove manually:
await hideAriaRefLabels({ page })
// Can take multiple screenshots in one execution
await screenshotWithAccessibilityLabels({ page })
await page.click('button')
await screenshotWithAccessibilityLabels({ page })
// Both images are included in the response
```
**Important:** Labels auto-hide after 30 seconds to prevent stale labels. If the page HTML changes (navigation, dynamic content), call `showAriaRefLabels()` again to get fresh labels matching the current DOM.
The function automatically shows labels, takes a screenshot, hides labels, and includes both the image and accessibility snapshot in the MCP response. Screenshots are saved to `./tmp/` with unique filenames.
**Features:**
- **Role filtering** - Only shows labels for interactive elements (buttons, links, inputs, etc.)
@@ -277,7 +278,7 @@ const source = await getReactSource({ locator: page.locator('aria-ref=e5') });
**User Collaboration Features:**
- **Right-click → "Copy Playwriter Element Reference"** - Pin any element and reference it as `globalThis.playwriterPinnedElem1` in your automation code
- **Vimium-style visual labels** - `showAriaRefLabels()` overlays clickable labels on all interactive elements
- **Vimium-style visual labels** - `screenshotWithAccessibilityLabels()` captures screenshots with clickable labels on all interactive elements
- **Tab group organization** - Connected tabs are grouped together with a green "playwriter" label
- **Bypass automation detection** - Disconnect the extension temporarily to pass bot detection (e.g., Google login), then reconnect
playwriter/src/assets/aria-labels-google-snapshot.txt
+1 -1
View File
@@ -31,7 +31,7 @@
- generic [ref=e55]:
- text: "Google offered in:"
- link [ref=e56] [cursor=pointer]:
- /url: https://www.google.com/setprefs?sig=0_46HrevRB93iEOCE9jEsq18qNx-Q%3D&hl=it&source=homepage&sa=X&ved=0ahUKEwjGpZr4lPKRAxXcFTQIHQEtATMQ2ZgBCBU
- /url: https://www.google.com/setprefs?sig=0_U_LREl4THfXa2sQ-b49g0Jd45jk%3D&hl=it&source=homepage&sa=X&ved=0ahUKEwiYgJn4hfSRAxU00xoGHUTZBMEQ2ZgBCBU
- text: Italiano
- contentinfo [ref=e58]:
- generic [ref=e59]: Italy
playwriter/src/assets/aria-labels-hacker-news-snapshot.txt
+745 -751
View File
File diff suppressed because it is too large Load Diff
playwriter/src/mcp.test.ts
+70
View File
@@ -2307,6 +2307,76 @@ describe('MCP Server Tests', () => {
console.log(`Screenshots saved to: ${assetsDir}`)
}, 120000)
it('should take screenshot with accessibility labels via MCP execute tool', async () => {
const browserContext = getBrowserContext()
const serviceWorker = await getExtensionServiceWorker(browserContext)
const page = await browserContext.newPage()
await page.setContent(`
<html>
<body>
<button id="submit-btn">Submit Form</button>
<a href="/about">About Us</a>
<input type="text" placeholder="Enter your name" />
</body>
</html>
`)
await page.bringToFront()
await serviceWorker.evaluate(async () => {
await globalThis.toggleExtensionForActiveTab()
})
await new Promise(r => setTimeout(r, 400))
// Take screenshot with accessibility labels via MCP
const result = await client.callTool({
name: 'execute',
arguments: {
code: js`
let testPage;
for (const p of context.pages()) {
const html = await p.content();
if (html.includes('submit-btn')) { testPage = p; break; }
}
if (!testPage) throw new Error('Test page not found');
await screenshotWithAccessibilityLabels({ page: testPage });
`,
timeout: 15000,
},
})
expect(result.isError).toBeFalsy()
// Verify response has both text and image content
const content = result.content as any[]
expect(content.length).toBe(2)
// Check text content
const textContent = content.find(c => c.type === 'text')
expect(textContent).toBeDefined()
expect(textContent.text).toContain('Screenshot saved to:')
expect(textContent.text).toContain('.jpg')
expect(textContent.text).toContain('Labels shown:')
expect(textContent.text).toContain('Accessibility snapshot:')
expect(textContent.text).toContain('Submit Form')
// Check image content
const imageContent = content.find(c => c.type === 'image')
expect(imageContent).toBeDefined()
expect(imageContent.mimeType).toBe('image/jpeg')
expect(imageContent.data).toBeDefined()
expect(imageContent.data.length).toBeGreaterThan(100) // base64 data should be substantial
// Verify the image is valid JPEG by checking base64
const buffer = Buffer.from(imageContent.data, 'base64')
const dimensions = imageSize(buffer)
expect(dimensions.type).toBe('jpg')
expect(dimensions.width).toBeGreaterThan(0)
expect(dimensions.height).toBeGreaterThan(0)
await page.close()
}, 60000)
})
playwriter/src/mcp.ts
+35 -14
View File
@@ -21,7 +21,7 @@ import { Editor } from './editor.js'
import { getStylesForLocator, formatStylesAsText, type StylesResult } from './styles.js'
import { getReactSource, type ReactSourceLocation } from './react-source.js'
import { ScopedFS } from './scoped-fs.js'
import { showAriaRefLabels, hideAriaRefLabels } from './aria-snapshot.js'
import { screenshotWithAccessibilityLabels, type ScreenshotResult } from './aria-snapshot.js'
const __filename = fileURLToPath(import.meta.url)
const __dirname = path.dirname(__filename)
@@ -86,8 +86,7 @@ interface VMContext {
getStylesForLocator: (options: { locator: any }) => Promise<StylesResult>
formatStylesAsText: (styles: StylesResult) => string
getReactSource: (options: { locator: any }) => Promise<ReactSourceLocation | null>
showAriaRefLabels: (options: { page: Page; interactiveOnly?: boolean }) => Promise<{ snapshot: string; labelCount: number }>
hideAriaRefLabels: (options: { page: Page }) => Promise<void>
screenshotWithAccessibilityLabels: (options: { page: Page; interactiveOnly?: boolean }) => Promise<void>
require: NodeRequire
import: (specifier: string) => Promise<any>
}
@@ -864,6 +863,13 @@ server.tool(
return getReactSource({ locator: options.locator, cdp })
}
// Collector for screenshots taken during this execution
const screenshotCollector: ScreenshotResult[] = []
const screenshotWithAccessibilityLabelsFn = async (options: { page: Page; interactiveOnly?: boolean }) => {
return screenshotWithAccessibilityLabels({ ...options, collector: screenshotCollector })
}
let vmContextObj: VMContextWithGlobals = {
page,
context,
@@ -880,8 +886,7 @@ server.tool(
getStylesForLocator: getStylesForLocatorFn,
formatStylesAsText,
getReactSource: getReactSourceFn,
showAriaRefLabels,
hideAriaRefLabels,
screenshotWithAccessibilityLabels: screenshotWithAccessibilityLabelsFn,
resetPlaywright: async () => {
const { page: newPage, context: newContext } = await resetConnection()
@@ -901,8 +906,7 @@ server.tool(
getStylesForLocator: getStylesForLocatorFn,
formatStylesAsText,
getReactSource: getReactSourceFn,
showAriaRefLabels,
hideAriaRefLabels,
screenshotWithAccessibilityLabels: screenshotWithAccessibilityLabelsFn,
resetPlaywright: vmContextObj.resetPlaywright,
require: sandboxedRequire,
// TODO --experimental-vm-modules is needed to make import work in vm
@@ -943,6 +947,13 @@ server.tool(
responseText += 'Code executed successfully (no output)'
}
// Add screenshot info to response text
for (const screenshot of screenshotCollector) {
responseText += `\nScreenshot saved to: ${screenshot.path}\n`
responseText += `Labels shown: ${screenshot.labelCount}\n\n`
responseText += `Accessibility snapshot:\n${screenshot.snapshot}\n`
}
const MAX_LENGTH = 6000
let finalText = responseText.trim()
if (finalText.length > MAX_LENGTH) {
@@ -951,14 +962,24 @@ server.tool(
`\n\n[Truncated to ${MAX_LENGTH} characters. Better manage your logs or paginate them to read the full logs]`
}
return {
content: [
{
type: 'text',
text: finalText,
},
],
// Build content array with text and any collected screenshots
const content: Array<{ type: 'text'; text: string } | { type: 'image'; data: string; mimeType: string }> = [
{
type: 'text',
text: finalText,
},
]
// Add all collected screenshots as images
for (const screenshot of screenshotCollector) {
content.push({
type: 'image',
data: screenshot.base64,
mimeType: screenshot.mimeType,
})
}
return { content }
} catch (error: any) {
const errorStack = error.stack || error.message
const isTimeoutError = error instanceof CodeExecutionTimeoutError || error.name === 'TimeoutError'
playwriter/src/prompt.md
+10 -11
View File
@@ -206,24 +206,23 @@ const matches = await editor.grep({ regex: /console\.log/ });
await editor.edit({ url: matches[0].url, oldString: 'DEBUG = false', newString: 'DEBUG = true' });
```
**showAriaRefLabels** - overlay Vimium-style visual labels on interactive elements. Useful for taking screenshots where you can see element references. Labels auto-hide after 30 seconds. Call again if page HTML changes or scrolls to get fresh labels. Use a timeout of 10 seconds at least.
**screenshotWithAccessibilityLabels** - take a screenshot with Vimium-style visual labels overlaid on interactive elements. Shows labels, captures screenshot, then removes labels. The image and accessibility snapshot are automatically included in the response. Can be called multiple times to capture multiple screenshots. Use a timeout of 10 seconds at least.
```js
const { snapshot, labelCount } = await showAriaRefLabels({ page });
console.log(`Showing ${labelCount} labels`);
await page.screenshot({ path: '/tmp/labeled-page.png' });
// Use aria-ref from snapshot to interact
await screenshotWithAccessibilityLabels({ page });
// Image and accessibility snapshot are automatically included in response
// Use aria-ref from snapshot to interact with elements
await page.locator('aria-ref=e5').click();
// Can take multiple screenshots in one execution
await screenshotWithAccessibilityLabels({ page });
await page.click('button');
await screenshotWithAccessibilityLabels({ page });
// Both images are included in the response
```
Labels are color-coded: yellow=links, orange=buttons, coral=inputs, pink=checkboxes, peach=sliders, salmon=menus, amber=tabs.
**hideAriaRefLabels** - manually remove labels before the 30-second auto-hide:
```js
await hideAriaRefLabels({ page });
```
## pinned elements
Users can right-click → "Copy Playwriter Element Reference" to store elements in `globalThis.playwriterPinnedElem1` (increments for each pin). The reference is copied to clipboard: