feat: harden relay sessions and recording UX across extension + playwriter

This commit batches the pending workspace updates that were coupled in practice: extension reliability fixes, ghost cursor persistence across MPA navigations, timeout tuning, and aligned docs/versioning updates so behavior and guidance stay in sync.

Key updates:\n- extension: add MV3 keepalive via chrome.alarms, add alarms permission, and bump extension changelog/version to 0.0.73 to reduce silent relay disconnects while idle\n- playwriter runtime: raise default action timeout to 60s while keeping navigation timeout separate; increase relay /version fetch timeout to 2000ms\n- recording + cursor: add persistent cursor init scripts with Page.addScriptToEvaluateOnNewDocument, remove init script on teardown, and ensure cursor enablement before applying mouse actions\n- demo video defaults: use 0.5s side buffer (1s total) and default idle speed 6x for createDemoVideo/computeIdleSections\n- tests/docs: keep click-error tests deterministic with explicit 5000ms per-click timeout, remove outdated click-timeout section from skill mistakes, and update package/changelog entries
This commit is contained in:
Tommy D. Rossi
2026-02-28 22:12:07 +01:00
parent 9222a96a98
commit f793dd8ac4
13 changed files with 191 additions and 70 deletions
+6
View File
@@ -1,5 +1,11 @@
# Changelog
## 0.0.73
### Bug Fixes
- **Service worker keepalive via chrome.alarms**: Added `chrome.alarms` keepalive to prevent Chrome MV3 from terminating the service worker when idle. Without this, the `maintainLoop` stops, the WebSocket closes, and the extension silently disconnects from the relay server — causing `session new` to fail with "Extension did not connect within timeout."
## 0.0.72
### Bug Fixes
+2 -1
View File
@@ -1,9 +1,10 @@
{
"manifest_version": 3,
"name": "Playwriter",
"version": "0.0.72",
"version": "0.0.73",
"description": "Automate your Browser using Cursor, Claude, VS Code. More capable and context efficient than Playwright MCP.",
"permissions": [
"alarms",
"debugger",
"tabGroups",
"contextMenus",
+12
View File
@@ -1483,6 +1483,18 @@ async function onActionClicked(tab: chrome.tabs.Tab): Promise<void> {
resetDebugger()
connectionManager.maintainLoop()
// MV3 service workers can be terminated by Chrome despite setInterval.
// chrome.alarms is the only reliable way to keep a service worker alive.
// Without this, the maintainLoop stops, the WebSocket closes, and the
// extension silently disconnects from the relay server.
chrome.alarms.create('keepalive', { periodInMinutes: 0.25 })
chrome.alarms.onAlarm.addListener((alarm) => {
if (alarm.name === 'keepalive') {
// No-op — the alarm firing is enough to wake the service worker.
// The maintainLoop will handle reconnection if needed.
}
})
chrome.contextMenus
.remove('playwriter-pin-element')
.catch(() => {})
+30
View File
@@ -1,7 +1,37 @@
# Changelog
## 0.0.84
### Docs
- **Remove click-timeout section from `skill.md` mistakes list**: Deleted the dedicated post-click timeout section per docs cleanup request and renumbered the following mistake item.
## 0.0.83
### Docs
- **Correct click-timeout guidance in `skill.md`**: Replaced the incorrect "SPA does not fire load" explanation with the actual behavior: click succeeds, then Playwright times out during post-click navigation waiting when action timeout budget is exceeded.
## 0.0.82
### Improvements
- **Increase default action timeout to 60s**: Raised `context.setDefaultTimeout` from `5000ms` to `60000ms` to avoid false click failures on slower real-world flows where the interaction succeeds but Playwright post-action navigation waiting exceeds short budgets.
## 0.0.81
### Improvements
- **Increase default action timeout to 5s**: Raised `context.setDefaultTimeout` from `2000ms` to `5000ms` to reduce false `locator.click()` timeouts on slower navigation flows (for example GitHub/Turbo links) where the click succeeds but post-click navigation settling exceeds 2 seconds. Default navigation timeout remains `10000ms`.
## 0.0.80
### Bug Fixes
- **Relaxed relay server version check timeout**: Increased `getRelayServerVersion` timeout from 500ms to 2000ms to prevent false "server not running" detections that kill a healthy relay server and disconnect the extension. This was causing intermittent `session new` failures when the relay was briefly busy (e.g. processing recording chunks).
## 0.0.80 (previous)
### Improvements
- **Descriptive click timeout errors**: When `locator.click()` times out due to actionability failures, the error now includes the reason (e.g. "Element is not visible", "Element is not stable", "<button> intercepts pointer events") instead of just "Timeout exceeded."
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "playwriter",
"description": "",
"version": "0.0.80",
"version": "0.0.84",
"type": "module",
"main": "dist/index.js",
"types": "dist/index.d.ts",
+8 -6
View File
@@ -592,9 +592,10 @@ export class PlaywrightExecutor {
const contexts = browser.contexts()
const context = contexts.length > 0 ? contexts[0] : await browser.newContext()
// Action timeout (click, fill, hover, etc.) is short for fast agent failure.
// Navigation timeout (goto, reload) is longer since page loads are slower.
context.setDefaultTimeout(2000)
// Action timeout (click, fill, hover, etc.) is longer to tolerate slower
// SPA/Turbo navigations and post-click settling on real sites.
// Navigation timeout (goto, reload) remains separate.
context.setDefaultTimeout(60000)
context.setDefaultNavigationTimeout(10000)
context.on('page', (page) => {
@@ -674,9 +675,10 @@ export class PlaywrightExecutor {
const contexts = browser.contexts()
const context = contexts.length > 0 ? contexts[0] : await browser.newContext()
// Action timeout (click, fill, hover, etc.) is short for fast agent failure.
// Navigation timeout (goto, reload) is longer since page loads are slower.
context.setDefaultTimeout(2000)
// Action timeout (click, fill, hover, etc.) is longer to tolerate slower
// SPA/Turbo navigations and post-click settling on real sites.
// Navigation timeout (goto, reload) remains separate.
context.setDefaultTimeout(60000)
context.setDefaultNavigationTimeout(10000)
context.on('page', (page) => {
+8 -8
View File
@@ -14,8 +14,8 @@ import path from 'node:path'
// Constants
// ---------------------------------------------------------------------------
/** Seconds of normal-speed buffer kept before and after each execution */
export const INTERACTION_BUFFER_SECONDS = 1
/** Seconds of normal-speed buffer kept before and after each execution (0.5s each side = 1s total) */
export const INTERACTION_BUFFER_SECONDS = 0.5
// ---------------------------------------------------------------------------
// Hardware encoder detection
@@ -635,13 +635,13 @@ export interface ExecutionTimestamp {
export function computeIdleSections({
executionTimestamps,
totalDurationMs,
speed = 5,
speed = 6,
bufferSeconds = INTERACTION_BUFFER_SECONDS,
}: {
executionTimestamps: ExecutionTimestamp[]
/** Total recording duration in milliseconds (from stopRecording result) */
totalDurationMs: number
/** Speed multiplier for idle sections (default 5) */
/** Speed multiplier for idle sections (default 6) */
speed?: number
/** Override the default buffer around each execution (seconds) */
bufferSeconds?: number
@@ -711,7 +711,7 @@ export interface CreateDemoVideoOptions {
durationMs: number
/** Execution timestamps (from stopRecording result) */
executionTimestamps: ExecutionTimestamp[]
/** Speed multiplier for idle sections (default 5) */
/** Speed multiplier for idle sections (default 6) */
speed?: number
/** Output file path (defaults to recordingPath with `-demo` suffix) */
outputFile?: string
@@ -722,8 +722,8 @@ export interface CreateDemoVideoOptions {
* Create a demo video from a recording by speeding up idle sections
* (gaps between execute() calls) while keeping interactions at normal speed.
*
* A 1-second buffer (INTERACTION_BUFFER_SECONDS) is preserved around each
* interaction so viewers see context before and after each action.
* A 0.5-second buffer (INTERACTION_BUFFER_SECONDS) is preserved on each side of
* an interaction (1 second total) so viewers see context before and after each action.
*
* Requires `ffmpeg` and `ffprobe` installed on the system.
*
@@ -736,7 +736,7 @@ export async function createDemoVideo(
recordingPath,
durationMs,
executionTimestamps,
speed = 5,
speed = 6,
signal,
} = options
+3 -2
View File
@@ -247,7 +247,8 @@ function applyRuntimeVisualOptions(): void {
}
if (runtime.options.style === 'minimal') {
const triangleSvg = `<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 24 24"><path fill="${runtime.options.color}" d="m23.284 19.124l-6.866-6.895a.4.4 0 0 1-.118-.296a.43.43 0 0 1 .163-.282l4.439-3.077a1.48 1.48 0 0 0 .621-1.48a1.48 1.48 0 0 0-1.036-1.198L1.623.302a1.14 1.14 0 0 0-1.11.282A1.13 1.13 0 0 0 .29 1.649L5.928 20.44a1.48 1.48 0 0 0 1.183 1.035a1.48 1.48 0 0 0 1.48-.621l3.078-4.44a.37.37 0 0 1 .31-.118a.43.43 0 0 1 .296.104l6.91 6.91a1.48 1.48 0 0 0 2.087 0l2.086-2.086a1.48 1.48 0 0 0-.074-2.101"/></svg>`
// White fill with dark border stroke, like a standard macOS cursor
const triangleSvg = `<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="-1 -1 26 26"><path fill="white" stroke="${runtime.options.color}" stroke-width="1.5" stroke-linejoin="round" d="m23.284 19.124l-6.866-6.895a.4.4 0 0 1-.118-.296a.43.43 0 0 1 .163-.282l4.439-3.077a1.48 1.48 0 0 0 .621-1.48a1.48 1.48 0 0 0-1.036-1.198L1.623.302a1.14 1.14 0 0 0-1.11.282A1.13 1.13 0 0 0 .29 1.649L5.928 20.44a1.48 1.48 0 0 0 1.183 1.035a1.48 1.48 0 0 0 1.48-.621l3.078-4.44a.37.37 0 0 1 .31-.118a.43.43 0 0 1 .296.104l6.91 6.91a1.48 1.48 0 0 0 2.087 0l2.086-2.086a1.48 1.48 0 0 0-.074-2.101"/></svg>`
const triangleDataUrl = `url("data:image/svg+xml,${encodeURIComponent(triangleSvg)}")`
runtime.cursorElement.style.borderRadius = '0'
runtime.cursorElement.style.border = 'none'
@@ -258,7 +259,7 @@ function applyRuntimeVisualOptions(): void {
runtime.cursorElement.style.backgroundPosition = 'left top'
runtime.cursorElement.style.backdropFilter = 'none'
runtime.cursorElement.style.boxShadow = 'none'
runtime.cursorElement.style.filter = 'drop-shadow(0 1px 1px rgba(0, 0, 0, 0.3))'
runtime.cursorElement.style.filter = 'drop-shadow(0 1px 2px rgba(0, 0, 0, 0.4))'
runtime.cursorElement.style.opacity = getBaseOpacity()
return
}
+77 -19
View File
@@ -1,12 +1,19 @@
/**
* Node-side ghost cursor helpers.
* Injects the browser bundle and forwards mouse action events to the page overlay.
*
* Two injection strategies:
* 1. One-shot: page.evaluate() — used by enableGhostCursor() for single pages
* 2. Persistent: Page.addScriptToEvaluateOnNewDocument — used by
* addGhostCursorInitScript() so the cursor survives MPA navigations.
* Chrome re-runs the script on every new document automatically.
*/
import fs from 'node:fs'
import path from 'node:path'
import { fileURLToPath } from 'node:url'
import type { Page, MouseActionEvent } from '@xmorse/playwright-core'
import { getCDPSessionForPage, type ICDPSession } from './cdp-session.js'
export interface GhostCursorClientOptions {
style?: 'minimal' | 'dot' | 'screenstudio'
@@ -23,6 +30,7 @@ interface GhostCursorBrowserApi {
enable: (options?: GhostCursorClientOptions) => void
disable: () => void
applyMouseAction: (event: MouseActionEvent) => void
isEnabled: () => boolean
}
let ghostCursorCode: string | null = null
@@ -76,34 +84,84 @@ export async function disableGhostCursor(options: { page: Page }): Promise<void>
})
}
/**
* Register the ghost cursor bundle as a persistent init script via CDP
* Page.addScriptToEvaluateOnNewDocument. Chrome re-runs this on every new
* document (navigation), so the cursor survives MPA page loads.
*
* The script auto-enables when the DOM is ready (DOMContentLoaded or
* requestAnimationFrame fallback) so the cursor is visible immediately
* on each new page.
*
* Returns the CDP identifier needed to remove it later.
*/
export async function addGhostCursorInitScript(options: {
page: Page
cursorOptions?: GhostCursorClientOptions
}): Promise<{ cdp: ICDPSession; identifier: string }> {
const { page, cursorOptions } = options
const cdp = await getCDPSessionForPage({ page })
const code = getGhostCursorCode()
// Wrap the bundle: inject, then auto-enable once DOM is ready.
// DOMContentLoaded may have already fired if the script runs late,
// so check document.readyState first.
const optionsJson = JSON.stringify(cursorOptions ?? {})
const wrappedCode = `
${code}
;(function() {
function autoEnable() {
var api = globalThis.__playwriterGhostCursor;
if (api) {
var opts = ${optionsJson};
api.enable(Object.keys(opts).length ? opts : undefined);
}
}
if (document.readyState === 'loading') {
document.addEventListener('DOMContentLoaded', autoEnable, { once: true });
} else {
requestAnimationFrame(autoEnable);
}
})();
`
const result = await cdp.send('Page.addScriptToEvaluateOnNewDocument', {
source: wrappedCode,
})
return { cdp, identifier: result.identifier }
}
/**
* Remove a previously registered ghost cursor init script.
*/
export async function removeGhostCursorInitScript(options: {
cdp: ICDPSession
identifier: string
}): Promise<void> {
const { cdp, identifier } = options
await cdp.send('Page.removeScriptToEvaluateOnNewDocument', { identifier })
}
export async function applyGhostCursorMouseAction(options: {
page: Page
event: MouseActionEvent
}): Promise<void> {
const { page, event } = options
const applied = await page.evaluate(
({ serializedEvent }) => {
const api = (globalThis as { __playwriterGhostCursor?: GhostCursorBrowserApi }).__playwriterGhostCursor
if (!api) {
return false
}
api.applyMouseAction(serializedEvent)
return true
},
{ serializedEvent: event },
)
if (applied) {
return
}
await ensureGhostCursorInjected({ page })
await page.evaluate(
({ serializedEvent }) => {
const api = (globalThis as { __playwriterGhostCursor?: GhostCursorBrowserApi }).__playwriterGhostCursor
api?.applyMouseAction(serializedEvent)
if (!api) {
return
}
// Ensure cursor is enabled (may be a freshly injected init script
// where DOMContentLoaded hasn't fired yet)
if (!api.isEnabled()) {
api.enable()
}
api.applyMouseAction(serializedEvent)
},
{ serializedEvent: event },
)
+36 -1
View File
@@ -1,10 +1,21 @@
/**
* Encapsulates ghost cursor lifecycle for recording sessions.
* Keeps onMouseAction chaining/restoration isolated from executor logic.
*
* Uses Page.addScriptToEvaluateOnNewDocument so the cursor persists across
* MPA navigations — Chrome re-runs the init script on every new document.
*/
import type { BrowserContext, Page } from '@xmorse/playwright-core'
import { applyGhostCursorMouseAction, disableGhostCursor, enableGhostCursor, type GhostCursorClientOptions } from './ghost-cursor.js'
import type { ICDPSession } from './cdp-session.js'
import {
addGhostCursorInitScript,
applyGhostCursorMouseAction,
disableGhostCursor,
enableGhostCursor,
removeGhostCursorInitScript,
type GhostCursorClientOptions,
} from './ghost-cursor.js'
interface RecordingGhostCursorLogger {
error: (...args: unknown[]) => void
@@ -15,9 +26,15 @@ interface RecordingTargetOptions {
sessionId?: string
}
interface InitScriptHandle {
cdp: ICDPSession
identifier: string
}
export class RecordingGhostCursorController {
private readonly previousMouseActionByPage = new WeakMap<Page, Page['onMouseAction']>()
private readonly cursorApplyQueueByPage = new WeakMap<Page, Promise<void>>()
private readonly initScriptByPage = new WeakMap<Page, InitScriptHandle>()
private readonly logger: RecordingGhostCursorLogger
constructor(options: { logger: RecordingGhostCursorLogger }) {
@@ -52,7 +69,14 @@ export class RecordingGhostCursorController {
const { page } = options
try {
// Register persistent init script so cursor survives MPA navigations
const handle = await addGhostCursorInitScript({ page })
this.initScriptByPage.set(page, handle)
// Also enable on the current page immediately (init script only runs
// on future navigations, not the current document)
await enableGhostCursor({ page })
if (!this.previousMouseActionByPage.has(page)) {
this.previousMouseActionByPage.set(page, page.onMouseAction)
}
@@ -88,6 +112,17 @@ export class RecordingGhostCursorController {
this.previousMouseActionByPage.delete(page)
this.cursorApplyQueueByPage.delete(page)
// Remove the persistent init script so future navigations don't inject cursor
const handle = this.initScriptByPage.get(page)
if (handle) {
try {
await removeGhostCursorInitScript(handle)
} catch (error) {
this.logger.error('[playwriter] Failed to remove ghost cursor init script', error)
}
this.initScriptByPage.delete(page)
}
try {
await disableGhostCursor({ page })
} catch (error) {
+1 -1
View File
@@ -27,7 +27,7 @@ export type ExtensionStatus = {
export async function getRelayServerVersion(port: number = RELAY_PORT): Promise<string | null> {
try {
const response = await fetch(`http://127.0.0.1:${port}/version`, {
signal: AbortSignal.timeout(500),
signal: AbortSignal.timeout(2000),
})
if (!response.ok) {
return null
+6 -6
View File
@@ -1039,14 +1039,14 @@ describe('Relay Core Tests', () => {
name: 'execute',
arguments: {
code: js`
await state.errorTestPage.click('#hidden-btn');
await state.errorTestPage.click('#hidden-btn', { timeout: 5000 });
`,
},
})
const text = (result as any).content[0].text
// Strip stack traces and call logs to only match the descriptive error line
const errorLine = text.split('\n').find((l: string) => l.includes('Timeout') || l.includes('not visible') || l.includes('not stable'))
expect(errorLine).toMatchInlineSnapshot(`"Error executing code: page.click: Timeout 2000ms exceeded. Element is not visible — it may be hidden by CSS, inside a collapsed <details>, inactive tab, or closed accordion. Try: interact with the page to reveal it first, or use { force: true } to skip visibility checks"`)
expect(errorLine).toMatchInlineSnapshot(`"Error executing code: page.click: Timeout 5000ms exceeded. Element is not visible — it may be hidden by CSS, inside a collapsed <details>, inactive tab, or closed accordion. Try: interact with the page to reveal it first, or use { force: true } to skip visibility checks"`)
expect((result as any).isError).toBe(true)
// Cleanup
await client.callTool({ name: 'execute', arguments: { code: js`await state.errorTestPage.close(); delete state.errorTestPage;` } })
@@ -1071,13 +1071,13 @@ describe('Relay Core Tests', () => {
name: 'execute',
arguments: {
code: js`
await state.errorTestPage.click('#covered-btn');
await state.errorTestPage.click('#covered-btn', { timeout: 5000 });
`,
},
})
const text = (result as any).content[0].text
const errorLine = text.split('\n').find((l: string) => l.includes('Timeout') || l.includes('intercepts'))
expect(errorLine).toMatchInlineSnapshot(`"Error executing code: page.click: Timeout 2000ms exceeded. <div id="overlay">Overlay</div> intercepts pointer events"`)
expect(errorLine).toMatchInlineSnapshot(`"Error executing code: page.click: Timeout 5000ms exceeded. <div id=\"overlay\">Overlay</div> intercepts pointer events"`)
expect((result as any).isError).toBe(true)
await client.callTool({ name: 'execute', arguments: { code: js`await state.errorTestPage.close(); delete state.errorTestPage;` } })
}, 30000)
@@ -1096,13 +1096,13 @@ describe('Relay Core Tests', () => {
name: 'execute',
arguments: {
code: js`
await state.errorTestPage.click('#invisible');
await state.errorTestPage.click('#invisible', { timeout: 5000 });
`,
},
})
const text = (result as any).content[0].text
const errorLine = text.split('\n').find((l: string) => l.includes('Timeout') || l.includes('not visible'))
expect(errorLine).toMatchInlineSnapshot(`"Error executing code: page.click: Timeout 2000ms exceeded. Element is not visible — it may be hidden by CSS, inside a collapsed <details>, inactive tab, or closed accordion. Try: interact with the page to reveal it first, or use { force: true } to skip visibility checks"`)
expect(errorLine).toMatchInlineSnapshot(`"Error executing code: page.click: Timeout 5000ms exceeded. Element is not visible — it may be hidden by CSS, inside a collapsed <details>, inactive tab, or closed accordion. Try: interact with the page to reveal it first, or use { force: true } to skip visibility checks"`)
expect((result as any).isError).toBe(true)
await client.callTool({ name: 'execute', arguments: { code: js`await state.errorTestPage.close(); delete state.errorTestPage;` } })
}, 30000)
+1 -25
View File
@@ -440,31 +440,7 @@ await state.page.evaluate(() => document.querySelector('button').click())
await state.page.getByRole('radio', { name: 'Node.js' }).click()
```
**13. SPA/Turbo navigation makes `click()` time out**
Sites using Turbo (GitHub), Next.js soft nav, or other SPA routers intercept link clicks and update the URL without firing the browser's standard `load` event. Playwright's `click()` sees the navigation start and waits for it to complete — but the load event never fires, so it times out even though the navigation succeeded.
Symptoms: `locator.click: Timeout Xms exceeded` with logs showing `navigated to "..."` underneath — the click worked, Playwright just timed out waiting for cleanup.
Fix: add `.catch(() => {})` on the click, or raise `timeout`, or use `{ noWaitAfter: true }` then wait manually:
```js
// Pattern 1: swallow the navigation-wait timeout (click still fired)
await state.page.locator('a.notification-link').first().click({ timeout: 10000 }).catch(() => {})
await state.page.waitForTimeout(1500) // give SPA time to render
// Pattern 2: skip the post-click navigation wait entirely
await state.page.locator('a.notification-link').first().click({ noWaitAfter: true })
await state.page.waitForLoadState('domcontentloaded').catch(() => {})
await state.page.waitForTimeout(1000)
// Pattern 3: catch at waitForLoadState instead
await state.page.locator('a').first().click()
await state.page.waitForLoadState('domcontentloaded').catch(() => {})
```
Always verify the URL afterwards with `console.log(state.page.url())` — the navigation typically did happen.
**14. Over-investigating instead of just interacting**
**13. Over-investigating instead of just interacting**
When something doesn't respond to a click, do NOT start inspecting CDP event listeners, React fibers, canvas pixel data, or writing `page.evaluate()` to read class names and bounding boxes. This wastes massive context. Instead:
1. Take a `snapshot()` — it shows every interactive element and what to click