Commit Graph

406 Commits

Author SHA1 Message Date
Tommy D. Rossi 09a363d2dc screenshotWithAccessibilityLabels 2026-01-05 10:11:36 +01:00
Tommy D. Rossi da24f0e011 feat: add media elements (img, video, audio) to aria labels
- Added img, video, audio to INTERACTIVE_ROLES for visual tasks
- Light blue color scheme for media element labels
- Agents can now reference images by aria-ref for visual tasks
2026-01-05 10:06:51 +01:00
Tommy D. Rossi 6feb459c1e docs: improve remote agents section with MCP JSON examples 2026-01-04 21:37:28 +01:00
Tommy D. Rossi 8e62b657cc fix: query playwriter tab group by title instead of caching ID
- Remove global playwriterGroupId cache that could become stale
- Query chrome.tabGroups by title on each sync instead
- Use batch chrome.tabs.ungroup() with array of IDs instead of loops
- Fixes issue where group wasn't created after debugger detach/reattach
2026-01-04 16:09:32 +01:00
Tommy D. Rossi b4689f390f fix tests adding more await globalThis.disconnectEverything() 2026-01-04 16:06:32 +01:00
Tommy D. Rossi b06425cbe8 add comparison vs Claude Code browser extension 2026-01-03 19:56:36 +01:00
Tommy D. Rossi 1ad94f3930 release: playwriter@0.0.39, extension@0.0.66
- fix icon not updating on WS disconnect
- increase aria-labels auto-hide timeout to 30s
2026-01-03 17:38:28 +01:00
Tommy D. Rossi f9a2312a58 diconnect tabs on close of ws 2026-01-03 17:36:18 +01:00
Tommy D. Rossi f7e4e90374 bump playwriter to 0.0.38 2026-01-03 16:45:58 +01:00
Tommy D. Rossi 018647aebf bump playwriter to 0.0.37 2026-01-03 16:43:19 +01:00
Tommy D. Rossi f2ee7755a0 bump version to 0.0.36, update changelog with aria ref labels feature 2026-01-03 16:05:51 +01:00
Tommy D. Rossi 71276ee888 fix auto-hide timer: cancel previous timer when showing new labels
- Store timer ID in window global so it can be cancelled
- Cancel pending timer when showAriaRefLabels is called again
- Cancel pending timer when hideAriaRefLabels is called manually
- Existing labels are removed before showing new ones
2026-01-03 16:04:20 +01:00
Tommy D. Rossi 1bb88853ad document showAriaRefLabels/hideAriaRefLabels in prompt.md 2026-01-03 15:57:12 +01:00
Tommy D. Rossi d889cea6b1 add showAriaRefLabels/hideAriaRefLabels to MCP context
- Functions available in execute tool context
- Auto-hide labels after 5 seconds to prevent stale overlays
- Updated README with MCP usage example
- Example saves screenshot to /tmp
2026-01-03 15:56:31 +01:00
Tommy D. Rossi d4072d41b6 document showAriaRefLabels feature in README 2026-01-03 15:52:02 +01:00
Tommy D. Rossi c59dbc0013 use warm color scheme for all label types
Color legend:
- Yellow: links
- Orange: buttons
- Coral/Red: text inputs
- Pink: checkboxes, radios, switches
- Peach: sliders
- Salmon: menu items
- Amber: tabs, options
2026-01-03 15:49:43 +01:00
Tommy D. Rossi 6f508795b4 update label colors: yellow for links, orange for buttons
Color scheme:
- Yellow: links (Vimium-style)
- Orange: buttons
- Blue: text inputs
- Purple: checkboxes, radios, switches
- Teal: sliders
- Pink: menu items
- Green: tabs, options
2026-01-03 15:48:15 +01:00
Tommy D. Rossi 3cd59600e9 filter labels for elements covered by opaque overlays
Uses elementsFromPoint() to check if element is blocked by opaque
overlays (modals, cookie banners, etc.). Skips elements where an
opaque element with pointer-events appears above the target in
the stacking order.

Correctly shows only visible, interactable elements.
2026-01-03 15:45:48 +01:00
Tommy D. Rossi d9783b15be add role-based color coding for aria ref labels
Color legend:
- Blue: links
- Green: buttons
- Orange: text inputs (textbox, combobox, searchbox)
- Purple: checkboxes, radios, switches
- Teal: sliders
- Pink: menu items
- Indigo: tabs, options
- Yellow: default/unknown

Also gitignore png screenshots in assets/
2026-01-03 15:31:00 +01:00
Tommy D. Rossi a31812be8d skip overlapping labels using greedy placement algorithm
Labels are placed in DOM order, skipping any that would overlap
with already-placed labels. Uses AABB intersection test.
2026-01-03 15:28:07 +01:00
Tommy D. Rossi 355053cbf9 filter aria ref labels to only interactive roles
Reduces label clutter by ~66-88% by only showing labels for:
button, link, textbox, combobox, searchbox, checkbox, radio,
slider, spinbutton, switch, menuitem, option, tab, treeitem

Added interactiveOnly option (default: true) to showAriaRefLabels()
2026-01-03 15:25:33 +01:00
Tommy D. Rossi 802e6768e2 add showAriaRefLabels/hideAriaRefLabels for Vimium-style visual labels
- getAriaSnapshot() discovers aria refs and caches ElementHandles
- showAriaRefLabels() overlays yellow badges on interactive elements
- hideAriaRefLabels() removes labels from page
- test captures screenshots from HN, Google, GitHub
2026-01-03 15:17:47 +01:00
Tommy D. Rossi 22a255e1dd refactor extension with ConnectionManager. log memory usage 2026-01-02 14:41:37 +01:00
Tommy D. Rossi a53342f36a add reload packge.json script 2026-01-02 14:41:05 +01:00
Tommy D. Rossi 1392bb8dc6 Update prompt.md 2026-01-02 14:40:57 +01:00
Tommy D. Rossi 5cf7d2756e more logs when url is empty 2026-01-02 14:40:03 +01:00
Tommy D. Rossi 7630ab7538 Update mcp.test.ts 2026-01-02 14:39:50 +01:00
Tommy D. Rossi e3cb7fd1fd add test to find out how to remove sleep after extension connect 2026-01-02 14:24:15 +01:00
Tommy D. Rossi cc128236eb agents.md 2026-01-01 12:00:10 +01:00
Tommy D. Rossi 2b0a5db49e adding docs 2026-01-01 12:00:02 +01:00
Tommy D. Rossi 90179dea17 handle connecting ws 2026-01-01 03:12:30 +01:00
Tommy D. Rossi ef6ab59351 Fix auto-enable to work without delay by skipping duplicate Target.attachedToTarget
Move auto-create logic from onOpen to Target.setAutoAttach handler so it runs
synchronously in the CDP command flow. Add skipAttachedEvent option to extension's
attachTab to prevent sending Target.attachedToTarget when the relay will send it
via the Target.setAutoAttach response handler.
2026-01-01 02:53:10 +01:00
Tommy D. Rossi 002f72af2c more logs 2026-01-01 02:23:25 +01:00
Tommy D. Rossi f6ae01866d use ts relay in dev 2026-01-01 02:23:16 +01:00
Tommy D. Rossi 70dc79ef8c pass env in spawn subprocess 2026-01-01 01:54:40 +01:00
Tommy D. Rossi 178387ec97 Persistent WS connection and auto-create initial tab
- Connect WS at extension startup via maintainConnection() loop
- Auto-retry every 5 seconds silently (no connecting badge)
- Never disconnect WS based on tab count
- Rename ConnectionState 'disconnected' to 'idle', remove global 'connecting'
- Fix race condition with tabGroupQueue for syncTabGroup/disconnectEverything/onTabUpdated
- Add auto-create initial tab when Playwright connects (PLAYWRITER_AUTO_ENABLE env var)
- Add test for auto-create feature
2026-01-01 01:14:16 +01:00
Tommy D. Rossi c15cf5947a Add unified mcpLog for relay server and stderr logging 2025-12-31 22:41:32 +01:00
Tommy D. Rossi 240ce5be74 Bump version to 0.0.34 2025-12-31 16:15:51 +01:00
Tommy D. Rossi be1b9514d3 Add ping/pong keep-alive to prevent service worker termination 2025-12-31 16:15:17 +01:00
Tommy D. Rossi 02fda90578 add more logs on detach 2025-12-31 16:07:57 +01:00
Tommy D. Rossi 5dca4094c8 dont restart server if version is higher than mcp 2025-12-31 16:05:34 +01:00
Tommy D. Rossi 3d0d280f6a Fix context menu duplicate ID crash and add global error logging 2025-12-31 15:45:32 +01:00
Tommy D. Rossi 0f9591722d Add Playwright CDPSession compatibility and export classes
- Add ICDPSession interface for Playwright CDPSession compatibility
- Add method aliases: addListener, removeListener, once, detach
- Make on/off return 'this' for chaining
- Export CDPSession, Editor, Debugger classes from index
- Classes accept ICDPSession but use typed CDPSession internally
2025-12-31 11:51:03 +01:00
Tommy D. Rossi a5f57ea5d6 add example locators 2025-12-31 11:43:49 +01:00
Tommy D. Rossi 9ad8428be1 Update prompt.md 2025-12-31 10:55:03 +01:00
Tommy D. Rossi 4a5bf3c3a3 Update prompt.md 2025-12-31 10:38:55 +01:00
Tommy D. Rossi 1f68e59e64 Update prompt.md 2025-12-31 10:36:50 +01:00
Tommy D. Rossi 97d971a993 nicer resources urls 2025-12-31 10:34:56 +01:00
Tommy D. Rossi 6753e1953f Update vite.config.ts 2025-12-30 17:45:45 +01:00
Tommy D. Rossi 2890d82470 Update package.json 2025-12-30 17:45:37 +01:00