feat: add ghost cursor APIs and harden reconnect flow
Introduce a first-class ghost cursor system for recordings and screenshots with namespace APIs (, ) while preserving backward-compatible top-level recording helpers. Improve reliability after relay restarts by consolidating extension wait logic around , reducing false disconnected states during and execute flows. Refactor recording glue into dedicated modules (, in ) to keep executor orchestration thin and make cursor lifecycle state explicit and testable.
This commit is contained in:
@@ -1,5 +1,46 @@
|
||||
# Changelog
|
||||
|
||||
## 0.0.75
|
||||
|
||||
### Improvements
|
||||
|
||||
- **Switch minimal cursor to triangular pointer icon**: Updated the `minimal` ghost cursor style to use a stylized triangular SVG pointer (with subtle drop shadow) instead of the circular indicator, while keeping `dot` and `screenstudio` styles available.
|
||||
|
||||
## 0.0.74
|
||||
|
||||
### Improvements
|
||||
|
||||
- **Switch default ghost cursor to a stylized minimal look**: Updated cursor rendering defaults to a cleaner minimal style while preserving `dot` and `screenstudio` options for explicit overrides.
|
||||
|
||||
## 0.0.73
|
||||
|
||||
### Improvements
|
||||
|
||||
- **Simplify recording integration in executor**: Moved ghost-cursor-aware recording wrappers out of `executor.ts` into `screen-recording.ts` via `createRecordingApi(...)`, reducing executor complexity while preserving existing `recording.*` and backward-compatible top-level recording helpers.
|
||||
|
||||
## 0.0.72
|
||||
|
||||
### Improvements
|
||||
|
||||
- **Reduce false "extension disconnected" on relay restarts**: `playwriter session new` now waits longer for extension reconnect and adds a short polling grace window before failing, preventing transient post-restart races from surfacing as hard disconnect errors.
|
||||
|
||||
## 0.0.71
|
||||
|
||||
### Features
|
||||
|
||||
- **Add `recording` and `ghostCursor` namespaces in execute context**: New `recording.start/stop/isRecording/cancel` and `ghostCursor.show/hide` APIs are now exposed for cleaner scripting while keeping `startRecording`, `stopRecording`, `isRecording`, and `cancelRecording` as backward-compatible aliases.
|
||||
- **Manual cursor overlay controls**: Cursor overlay can now be shown/hidden explicitly outside recording flows for screenshot and demo generation.
|
||||
|
||||
## 0.0.70
|
||||
|
||||
### Features
|
||||
|
||||
- **Ghost cursor overlay during recording**: Playwriter now auto-enables a smooth in-page ghost cursor when `startRecording()` is called, driven by `page.onMouseAction` callbacks from the Playwright fork so both `page.mouse.*` and `locator.click()` actions are visualized.
|
||||
|
||||
### Tests
|
||||
|
||||
- **Add ghost-cursor integration coverage**: Extended `on-mouse-action.test.ts` to verify callback-driven cursor animation and teardown in real extension-connected runs.
|
||||
|
||||
## 0.0.69
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "playwriter",
|
||||
"description": "",
|
||||
"version": "0.0.69",
|
||||
"version": "0.0.75",
|
||||
"type": "module",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
|
||||
@@ -47,6 +47,11 @@ const BUNDLES: BundleConfig[] = [
|
||||
type: 'source',
|
||||
entry: 'a11y-client.ts',
|
||||
},
|
||||
{
|
||||
name: 'ghost-cursor-client',
|
||||
type: 'source',
|
||||
entry: 'ghost-cursor-client.ts',
|
||||
},
|
||||
|
||||
// Wrapper bundles (npm packages → globalThis)
|
||||
{
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
/**
|
||||
* Encodes the extracted Screen Studio cursor SVG as a data URL TS module.
|
||||
*/
|
||||
|
||||
import fs from 'node:fs'
|
||||
import path from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
const currentDir = path.dirname(fileURLToPath(import.meta.url))
|
||||
const sourcePath = path.join(currentDir, '..', 'src', 'assets', 'cursors', 'screen-studio', 'pointer-macos-tahoe.svg')
|
||||
const outputPath = path.join(currentDir, '..', 'src', 'assets', 'cursors', 'screen-studio', 'pointer-macos-tahoe-data-url.ts')
|
||||
|
||||
function main() {
|
||||
const svg = fs.readFileSync(sourcePath, 'utf-8').trim()
|
||||
const base64 = Buffer.from(svg, 'utf-8').toString('base64')
|
||||
const dataUrl = `data:image/svg+xml;base64,${base64}`
|
||||
|
||||
const output = `/**\n * Generated from pointer-macos-tahoe.svg via scripts/encode-screenstudio-cursor.ts.\n */\n\nexport const SCREENSTUDIO_POINTER_MACOS_TAHOE_DATA_URL = '${dataUrl}'\n`
|
||||
fs.writeFileSync(outputPath, output)
|
||||
console.log(`Wrote ${outputPath}`)
|
||||
}
|
||||
|
||||
main()
|
||||
@@ -0,0 +1,5 @@
|
||||
/**
|
||||
* Generated from pointer-macos-tahoe.svg via scripts/encode-screenstudio-cursor.ts.
|
||||
*/
|
||||
|
||||
export const SCREENSTUDIO_POINTER_MACOS_TAHOE_DATA_URL = 'data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNjE4IiBoZWlnaHQ9Ijk1OCIgdmlld0JveD0iMCAwIDYxOCA5NTgiIGZpbGw9Im5vbmUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CjxnIGZpbHRlcj0idXJsKCNmaWx0ZXIwX2RfMzg0XzI3KSI+CjxwYXRoIGZpbGwtcnVsZT0iZXZlbm9kZCIgY2xpcC1ydWxlPSJldmVub2RkIiBkPSJNMTI3LjA2MiAzNy4wMzMxTDU0MC42OTYgNDUxLjU1NUM1OTIuNjUzIDUwMy42NiA1NTUuNzk0IDU5Mi41NzQgNDgyLjIyNiA1OTIuNTc0TDQyMS44MzEgNTkyLjU2OUw0ODEuODIxIDczNS4wNTRDNDkyLjMzMSA3NjAuMDIxIDQ5Mi40NzkgNzg3LjY1MiA0ODIuMjY1IDgxMi43NjdDNDcyLjAwMiA4MzcuOTMyIDQ1Mi41NjEgODU3LjU3IDQyNy40OTYgODY4LjA4QzQxNC44NjQgODczLjM1OSA0MDEuNjQgODc2LjAyNCAzODguMTIxIDg3Ni4wMjRDMzQ3LjExNyA4NzYuMDI0IDMxMC4zNTggODUxLjYgMjk0LjQ3IDgxMy44MDRMMjMxLjQyIDY2My45MThMMTkwLjM2OCA3MDAuMzM3QzEzNy4wMjkgNzQ3LjUwOCA1MyA3MDkuNjYzIDUzIDYzOC40MTNWNjcuNjc0NEM1MyAyOC45OTAzIDk5LjcyNjggOS42NDgyOCAxMjcuMDYyIDM3LjAzMzFaIiBmaWxsPSJ3aGl0ZSIvPgo8cGF0aCBmaWxsLXJ1bGU9ImV2ZW5vZGQiIGNsaXAtcnVsZT0iZXZlbm9kZCIgZD0iTTEwMi4zMTYgOTkuNjUyQzEwMi4zMTYgOTMuMTg4MiAxMTAuMTYyIDg5LjkzMTYgMTE0LjcwMSA5NC41MjA0TDUwNC44OTcgNDg1LjU1NUM1MjYuMTY0IDUwNi44NzEgNTExLjA2NSA1NDMuMjM2IDQ4MC45NjcgNTQzLjIzNkwzNDcuNTQ2IDU0My4xNjFMNDM2LjM0MiA3NTQuMTQzQzQ0Ny41NDIgNzgwLjc4OCA0MzUuMDA5IDgxMS40MjkgNDA4LjQxNCA4MjIuNTgxQzM4MS43MiA4MzMuNzgxIDM1MS4xMjggODIxLjI5OCAzMzkuOTc3IDc5NC43MDJMMjUwLjI5MyA1ODEuMzUyTDE1OC41MTcgNjYyLjY0NEMxMzcuOTkxIDY4MC44MDEgMTA2LjMxOSA2NjguMTQ1IDEwMi42NjQgNjQyLjMyM0wxMDIuMzE2IDYzNy4zMzFWOTkuNjUyWiIgZmlsbD0iYmxhY2siLz4KPC9nPgo8ZGVmcz4KPGZpbHRlciBpZD0iZmlsdGVyMF9kXzM4NF8yNyIgeD0iMC4zNCIgeT0iMC43OTkyMTkiIHdpZHRoPSI2MTcuMzIiIGhlaWdodD0iOTU3LjE0NCIgZmlsdGVyVW5pdHM9InVzZXJTcGFjZU9uVXNlIiBjb2xvci1pbnRlcnBvbGF0aW9uLWZpbHRlcnM9InNSR0IiPgo8ZmVGbG9vZCBmbG9vZC1vcGFjaXR5PSIwIiByZXN1bHQ9IkJhY2tncm91bmRJbWFnZUZpeCIvPgo8ZmVDb2xvck1hdHJpeCBpbj0iU291cmNlQWxwaGEiIHR5cGU9Im1hdHJpeCIgdmFsdWVzPSIwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAxMjcgMCIgcmVzdWx0PSJoYXJkQWxwaGEiLz4KPGZlT2Zmc2V0IGR5PSIyOS4yNiIvPgo8ZmVHYXVzc2lhbkJsdXIgc3RkRGV2aWF0aW9uPSIyNi4zMyIvPgo8ZmVDb21wb3NpdGUgaW4yPSJoYXJkQWxwaGEiIG9wZXJhdG9yPSJvdXQiLz4KPGZlQ29sb3JNYXRyaXggdHlwZT0ibWF0cml4IiB2YWx1ZXM9IjAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAuNjUgMCIvPgo8ZmVCbGVuZCBtb2RlPSJub3JtYWwiIGluMj0iQmFja2dyb3VuZEltYWdlRml4IiByZXN1bHQ9ImVmZmVjdDFfZHJvcFNoYWRvd18zODRfMjciLz4KPGZlQmxlbmQgbW9kZT0ibm9ybWFsIiBpbj0iU291cmNlR3JhcGhpYyIgaW4yPSJlZmZlY3QxX2Ryb3BTaGFkb3dfMzg0XzI3IiByZXN1bHQ9InNoYXBlIi8+CjwvZmlsdGVyPgo8L2RlZnM+Cjwvc3ZnPg=='
|
||||
@@ -0,0 +1,18 @@
|
||||
<svg width="618" height="958" viewBox="0 0 618 958" fill="none" xmlns="http://www.w3.org/2000/svg">
|
||||
<g filter="url(#filter0_d_384_27)">
|
||||
<path fill-rule="evenodd" clip-rule="evenodd" d="M127.062 37.0331L540.696 451.555C592.653 503.66 555.794 592.574 482.226 592.574L421.831 592.569L481.821 735.054C492.331 760.021 492.479 787.652 482.265 812.767C472.002 837.932 452.561 857.57 427.496 868.08C414.864 873.359 401.64 876.024 388.121 876.024C347.117 876.024 310.358 851.6 294.47 813.804L231.42 663.918L190.368 700.337C137.029 747.508 53 709.663 53 638.413V67.6744C53 28.9903 99.7268 9.64828 127.062 37.0331Z" fill="white"/>
|
||||
<path fill-rule="evenodd" clip-rule="evenodd" d="M102.316 99.652C102.316 93.1882 110.162 89.9316 114.701 94.5204L504.897 485.555C526.164 506.871 511.065 543.236 480.967 543.236L347.546 543.161L436.342 754.143C447.542 780.788 435.009 811.429 408.414 822.581C381.72 833.781 351.128 821.298 339.977 794.702L250.293 581.352L158.517 662.644C137.991 680.801 106.319 668.145 102.664 642.323L102.316 637.331V99.652Z" fill="black"/>
|
||||
</g>
|
||||
<defs>
|
||||
<filter id="filter0_d_384_27" x="0.34" y="0.799219" width="617.32" height="957.144" filterUnits="userSpaceOnUse" color-interpolation-filters="sRGB">
|
||||
<feFlood flood-opacity="0" result="BackgroundImageFix"/>
|
||||
<feColorMatrix in="SourceAlpha" type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 127 0" result="hardAlpha"/>
|
||||
<feOffset dy="29.26"/>
|
||||
<feGaussianBlur stdDeviation="26.33"/>
|
||||
<feComposite in2="hardAlpha" operator="out"/>
|
||||
<feColorMatrix type="matrix" values="0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0.65 0"/>
|
||||
<feBlend mode="normal" in2="BackgroundImageFix" result="effect1_dropShadow_384_27"/>
|
||||
<feBlend mode="normal" in="SourceGraphic" in2="effect1_dropShadow_384_27" result="shape"/>
|
||||
</filter>
|
||||
</defs>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 1.7 KiB |
+28
-16
@@ -5,6 +5,7 @@ import path from 'node:path'
|
||||
import util from 'node:util'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { cac } from '@xmorse/cac'
|
||||
import pc from 'picocolors'
|
||||
|
||||
// Prevent Buffers from dumping hex bytes in util.inspect output.
|
||||
Buffer.prototype[util.inspect.custom] = function () {
|
||||
@@ -15,9 +16,10 @@ import { VERSION, LOG_FILE_PATH, LOG_CDP_FILE_PATH, parseRelayHost } from './uti
|
||||
import {
|
||||
ensureRelayServer,
|
||||
RELAY_PORT,
|
||||
waitForExtension,
|
||||
waitForConnectedExtensions,
|
||||
getExtensionOutdatedWarning,
|
||||
getExtensionStatus,
|
||||
type ExtensionStatus,
|
||||
} from './relay-client.js'
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url))
|
||||
@@ -26,15 +28,6 @@ const cliRelayEnv = { PLAYWRITER_AUTO_ENABLE: '1' }
|
||||
|
||||
const cli = cac('playwriter')
|
||||
|
||||
type ExtensionStatus = {
|
||||
extensionId: string
|
||||
stableKey?: string
|
||||
browser: string | null
|
||||
profile: { email: string; id: string } | null
|
||||
activeTargets: number
|
||||
playwriterVersion?: string | null
|
||||
}
|
||||
|
||||
cli
|
||||
.command('', 'Start the MCP server or controls the browser with -e')
|
||||
.option('--host <host>', 'Remote relay server host to connect to (or use PLAYWRITER_HOST env var)')
|
||||
@@ -136,8 +129,12 @@ async function executeCode(options: {
|
||||
if (!host && !process.env.PLAYWRITER_HOST) {
|
||||
const restarted = await ensureRelayServer({ logger: console, env: cliRelayEnv })
|
||||
if (restarted) {
|
||||
const connected = await waitForExtension({ logger: console, timeoutMs: 10000 })
|
||||
if (!connected) {
|
||||
const connectedExtensions = await waitForConnectedExtensions({
|
||||
logger: console,
|
||||
timeoutMs: 10000,
|
||||
pollIntervalMs: 250,
|
||||
})
|
||||
if (connectedExtensions.length === 0) {
|
||||
console.error('Warning: Extension not connected. Commands may fail.')
|
||||
}
|
||||
}
|
||||
@@ -212,15 +209,30 @@ cli
|
||||
.option('--host <host>', 'Remote relay server host')
|
||||
.option('--browser <stableKey>', 'Stable browser key when multiple browsers are connected')
|
||||
.action(async (options: { host?: string; browser?: string }) => {
|
||||
if (!options.host && !process.env.PLAYWRITER_HOST) {
|
||||
const isLocal = !options.host && !process.env.PLAYWRITER_HOST
|
||||
|
||||
let extensions: ExtensionStatus[] = []
|
||||
|
||||
if (isLocal) {
|
||||
await ensureRelayServer({ logger: console, env: cliRelayEnv })
|
||||
await waitForExtension({
|
||||
timeoutMs: 3000,
|
||||
extensions = await waitForConnectedExtensions({
|
||||
timeoutMs: 12000,
|
||||
pollIntervalMs: 250,
|
||||
logger: console,
|
||||
})
|
||||
|
||||
if (extensions.length === 0) {
|
||||
console.log(pc.dim('Waiting briefly for extension to reconnect...'))
|
||||
extensions = await waitForConnectedExtensions({
|
||||
timeoutMs: 10000,
|
||||
pollIntervalMs: 250,
|
||||
logger: console,
|
||||
})
|
||||
}
|
||||
} else {
|
||||
extensions = await fetchExtensionsStatus(options.host)
|
||||
}
|
||||
|
||||
const extensions = await fetchExtensionsStatus(options.host)
|
||||
if (extensions.length === 0) {
|
||||
console.error('No connected browsers detected. Click the Playwriter extension icon.')
|
||||
process.exit(1)
|
||||
|
||||
+60
-30
@@ -33,8 +33,10 @@ import { createGhostBrowserChrome, type GhostBrowserCommandResult } from './ghos
|
||||
export type { SnapshotFormat }
|
||||
import { getCleanHTML, type GetCleanHTMLOptions } from './clean-html.js'
|
||||
import { getPageMarkdown, type GetPageMarkdownOptions } from './page-markdown.js'
|
||||
import { startRecording, stopRecording, isRecording, cancelRecording } from './screen-recording.js'
|
||||
import { createRecordingApi } from './screen-recording.js'
|
||||
import { createDemoVideo } from './ffmpeg.js'
|
||||
import { type GhostCursorClientOptions } from './ghost-cursor.js'
|
||||
import { RecordingGhostCursorController } from './recording-ghost-cursor.js'
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url)
|
||||
const __dirname = path.dirname(__filename)
|
||||
@@ -948,18 +950,51 @@ export class PlaywrightExecutor {
|
||||
// Recording uses chrome.tabCapture which requires activeTab permission.
|
||||
// This permission is granted when the user clicks the Playwriter extension icon on a tab.
|
||||
const relayPort = this.cdpConfig.port || 19988
|
||||
// Recording will work on any tab where the user has clicked the icon.
|
||||
const withRecordingDefaults = <T extends { page?: Page; sessionId?: string }, R>(
|
||||
fn: (opts: T & { relayPort: number; sessionId?: string }) => Promise<R>,
|
||||
) => {
|
||||
return async (options: T = {} as T) => {
|
||||
const targetPage = options.page || page
|
||||
// Use Playwright's exposed tab session ID directly.
|
||||
const sessionId = options.sessionId || targetPage.sessionId() || undefined
|
||||
return fn({ page: targetPage, sessionId, relayPort, ...options })
|
||||
}
|
||||
}
|
||||
const self = this
|
||||
const recordingGhostCursor = new RecordingGhostCursorController({
|
||||
logger: {
|
||||
error: (...args: unknown[]) => {
|
||||
self.logger.error(...args)
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
const showGhostCursor = async (options?: ({ page?: Page } & GhostCursorClientOptions)) => {
|
||||
const targetPage = options?.page || page
|
||||
const cursorOptions: GhostCursorClientOptions | undefined = (() => {
|
||||
if (!options) {
|
||||
return undefined
|
||||
}
|
||||
|
||||
const { page: _ignoredPage, ...rest } = options
|
||||
return rest
|
||||
})()
|
||||
|
||||
await recordingGhostCursor.show({ page: targetPage, cursorOptions })
|
||||
}
|
||||
|
||||
const hideGhostCursor = async (options?: { page?: Page }) => {
|
||||
const targetPage = options?.page || page
|
||||
await recordingGhostCursor.hide({ page: targetPage })
|
||||
}
|
||||
|
||||
const recordingApi = createRecordingApi({
|
||||
context,
|
||||
defaultPage: page,
|
||||
relayPort,
|
||||
ghostCursorController: recordingGhostCursor,
|
||||
onStart: () => {
|
||||
self.recordingStartedAt = Date.now()
|
||||
self.executionTimestamps = []
|
||||
},
|
||||
onFinish: () => {
|
||||
self.recordingStartedAt = null
|
||||
self.executionTimestamps = []
|
||||
},
|
||||
getExecutionTimestamps: () => {
|
||||
return self.executionTimestamps
|
||||
},
|
||||
})
|
||||
|
||||
// Ghost Browser API - creates chrome object that mirrors Ghost Browser's APIs
|
||||
// See extension/src/ghost-browser-api.d.ts for full API documentation
|
||||
@@ -994,26 +1029,21 @@ export class PlaywrightExecutor {
|
||||
formatStylesAsText,
|
||||
getReactSource: getReactSourceFn,
|
||||
screenshotWithAccessibilityLabels: screenshotWithAccessibilityLabelsFn,
|
||||
startRecording: async (opts?: Parameters<typeof startRecording>[0]) => {
|
||||
const result = await withRecordingDefaults(startRecording)(opts)
|
||||
self.recordingStartedAt = Date.now()
|
||||
self.executionTimestamps = []
|
||||
return result
|
||||
ghostCursor: {
|
||||
show: showGhostCursor,
|
||||
hide: hideGhostCursor,
|
||||
},
|
||||
stopRecording: async (opts?: Parameters<typeof stopRecording>[0]) => {
|
||||
const result = await withRecordingDefaults(stopRecording)(opts)
|
||||
const executionTimestamps = [...self.executionTimestamps]
|
||||
self.recordingStartedAt = null
|
||||
self.executionTimestamps = []
|
||||
return { ...result, executionTimestamps }
|
||||
},
|
||||
isRecording: withRecordingDefaults(isRecording),
|
||||
cancelRecording: async (opts?: Parameters<typeof cancelRecording>[0]) => {
|
||||
const result = await withRecordingDefaults(cancelRecording)(opts)
|
||||
self.recordingStartedAt = null
|
||||
self.executionTimestamps = []
|
||||
return result
|
||||
recording: {
|
||||
start: recordingApi.start,
|
||||
stop: recordingApi.stop,
|
||||
isRecording: recordingApi.isRecording,
|
||||
cancel: recordingApi.cancel,
|
||||
},
|
||||
// Backward-compatible aliases
|
||||
startRecording: recordingApi.start,
|
||||
stopRecording: recordingApi.stop,
|
||||
isRecording: recordingApi.isRecording,
|
||||
cancelRecording: recordingApi.cancel,
|
||||
createDemoVideo,
|
||||
resetPlaywright: async () => {
|
||||
const { page: newPage, context: newContext } = await self.reset()
|
||||
|
||||
@@ -0,0 +1,367 @@
|
||||
/**
|
||||
* Browser-side ghost cursor renderer.
|
||||
* Injected into the page to visualize automated mouse actions with smooth easing.
|
||||
*/
|
||||
|
||||
import { SCREENSTUDIO_POINTER_MACOS_TAHOE_DATA_URL } from './assets/cursors/screen-studio/pointer-macos-tahoe-data-url.js'
|
||||
|
||||
type GhostCursorActionType = 'move' | 'down' | 'up' | 'wheel'
|
||||
type GhostCursorButton = 'left' | 'right' | 'middle' | 'none'
|
||||
type GhostCursorStyle = 'minimal' | 'dot' | 'screenstudio'
|
||||
|
||||
interface GhostCursorAction {
|
||||
type: GhostCursorActionType
|
||||
x: number
|
||||
y: number
|
||||
button: GhostCursorButton
|
||||
}
|
||||
|
||||
export interface GhostCursorClientOptions {
|
||||
style?: GhostCursorStyle
|
||||
color?: string
|
||||
size?: number
|
||||
zIndex?: number
|
||||
easing?: string
|
||||
minDurationMs?: number
|
||||
maxDurationMs?: number
|
||||
speedPxPerMs?: number
|
||||
}
|
||||
|
||||
interface GhostCursorRuntimeOptions {
|
||||
style: GhostCursorStyle
|
||||
color: string
|
||||
size: number
|
||||
zIndex: number
|
||||
easing: string
|
||||
minDurationMs: number
|
||||
maxDurationMs: number
|
||||
speedPxPerMs: number
|
||||
}
|
||||
|
||||
interface GhostCursorRuntimeState {
|
||||
cursorElement: ReturnType<typeof createCursorElement> | null
|
||||
options: GhostCursorRuntimeOptions
|
||||
x: number
|
||||
y: number
|
||||
scale: number
|
||||
hasPosition: boolean
|
||||
enabled: boolean
|
||||
}
|
||||
|
||||
interface GhostCursorApi {
|
||||
enable: (options?: GhostCursorClientOptions) => void
|
||||
disable: () => void
|
||||
applyMouseAction: (action: GhostCursorAction) => void
|
||||
isEnabled: () => boolean
|
||||
}
|
||||
|
||||
declare global {
|
||||
var __playwriterGhostCursor: GhostCursorApi | undefined
|
||||
}
|
||||
|
||||
const CURSOR_ID = '__playwriter_ghost_cursor__'
|
||||
const SCREENSTUDIO_POINTER_ASPECT_RATIO = 618 / 958
|
||||
const SCREENSTUDIO_HOTSPOT_X_RATIO = 0.14
|
||||
const SCREENSTUDIO_HOTSPOT_Y_RATIO = 0.06
|
||||
const MINIMAL_TRIANGLE_HOTSPOT_X_RATIO = 0.07
|
||||
const MINIMAL_TRIANGLE_HOTSPOT_Y_RATIO = 0.06
|
||||
|
||||
const DEFAULT_OPTIONS: GhostCursorRuntimeOptions = {
|
||||
style: 'minimal',
|
||||
color: '#111827',
|
||||
size: 22,
|
||||
zIndex: 2147483647,
|
||||
easing: 'cubic-bezier(0.22, 1, 0.36, 1)',
|
||||
minDurationMs: 40,
|
||||
maxDurationMs: 450,
|
||||
speedPxPerMs: 2.2,
|
||||
}
|
||||
|
||||
const runtime: GhostCursorRuntimeState = {
|
||||
cursorElement: null,
|
||||
options: DEFAULT_OPTIONS,
|
||||
x: 0,
|
||||
y: 0,
|
||||
scale: 1,
|
||||
hasPosition: false,
|
||||
enabled: false,
|
||||
}
|
||||
|
||||
function clamp(options: { value: number; min: number; max: number }): number {
|
||||
const { value, min, max } = options
|
||||
return Math.min(max, Math.max(min, value))
|
||||
}
|
||||
|
||||
function mergeOptions(options?: GhostCursorClientOptions): GhostCursorRuntimeOptions {
|
||||
if (!options) {
|
||||
return DEFAULT_OPTIONS
|
||||
}
|
||||
|
||||
return {
|
||||
style: options.style ?? DEFAULT_OPTIONS.style,
|
||||
color: options.color ?? DEFAULT_OPTIONS.color,
|
||||
size: options.size ?? DEFAULT_OPTIONS.size,
|
||||
zIndex: options.zIndex ?? DEFAULT_OPTIONS.zIndex,
|
||||
easing: options.easing ?? DEFAULT_OPTIONS.easing,
|
||||
minDurationMs: options.minDurationMs ?? DEFAULT_OPTIONS.minDurationMs,
|
||||
maxDurationMs: options.maxDurationMs ?? DEFAULT_OPTIONS.maxDurationMs,
|
||||
speedPxPerMs: options.speedPxPerMs ?? DEFAULT_OPTIONS.speedPxPerMs,
|
||||
}
|
||||
}
|
||||
|
||||
function getCursorDimensions(): { width: number; height: number } {
|
||||
if (runtime.options.style === 'screenstudio') {
|
||||
const height = runtime.options.size
|
||||
const width = Math.max(10, Math.round(height * SCREENSTUDIO_POINTER_ASPECT_RATIO))
|
||||
return { width, height }
|
||||
}
|
||||
|
||||
if (runtime.options.style === 'minimal') {
|
||||
const size = Math.max(12, runtime.options.size)
|
||||
return { width: size, height: size }
|
||||
}
|
||||
|
||||
return { width: runtime.options.size, height: runtime.options.size }
|
||||
}
|
||||
|
||||
function getHotspotOffsetPx(): { x: number; y: number } {
|
||||
const dimensions = getCursorDimensions()
|
||||
|
||||
if (runtime.options.style === 'screenstudio') {
|
||||
return {
|
||||
x: Math.round(dimensions.width * SCREENSTUDIO_HOTSPOT_X_RATIO),
|
||||
y: Math.round(dimensions.height * SCREENSTUDIO_HOTSPOT_Y_RATIO),
|
||||
}
|
||||
}
|
||||
|
||||
if (runtime.options.style === 'minimal') {
|
||||
return {
|
||||
x: Math.round(dimensions.width * MINIMAL_TRIANGLE_HOTSPOT_X_RATIO),
|
||||
y: Math.round(dimensions.height * MINIMAL_TRIANGLE_HOTSPOT_Y_RATIO),
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
x: Math.round(dimensions.width / 2),
|
||||
y: Math.round(dimensions.height / 2),
|
||||
}
|
||||
}
|
||||
|
||||
function getBaseOpacity(): string {
|
||||
if (runtime.options.style === 'screenstudio') {
|
||||
return '0.95'
|
||||
}
|
||||
|
||||
if (runtime.options.style === 'minimal') {
|
||||
return '1'
|
||||
}
|
||||
|
||||
return '0.72'
|
||||
}
|
||||
|
||||
function applyTransform(): void {
|
||||
if (!runtime.cursorElement) {
|
||||
return
|
||||
}
|
||||
|
||||
const hotspot = getHotspotOffsetPx()
|
||||
runtime.cursorElement.style.transform = `translate3d(${runtime.x - hotspot.x}px, ${runtime.y - hotspot.y}px, 0) scale(${runtime.scale})`
|
||||
}
|
||||
|
||||
function computeDurationMs(options: { targetX: number; targetY: number }): number {
|
||||
if (!runtime.hasPosition) {
|
||||
return 0
|
||||
}
|
||||
|
||||
const dx = options.targetX - runtime.x
|
||||
const dy = options.targetY - runtime.y
|
||||
const distance = Math.hypot(dx, dy)
|
||||
const rawDurationMs = distance / runtime.options.speedPxPerMs
|
||||
|
||||
return clamp({
|
||||
value: rawDurationMs,
|
||||
min: runtime.options.minDurationMs,
|
||||
max: runtime.options.maxDurationMs,
|
||||
})
|
||||
}
|
||||
|
||||
function createCursorElement() {
|
||||
const element = document.createElement('div')
|
||||
element.id = CURSOR_ID
|
||||
element.setAttribute('aria-hidden', 'true')
|
||||
element.style.position = 'fixed'
|
||||
element.style.left = '0'
|
||||
element.style.top = '0'
|
||||
element.style.pointerEvents = 'none'
|
||||
element.style.zIndex = `${runtime.options.zIndex}`
|
||||
element.style.opacity = getBaseOpacity()
|
||||
element.style.transitionProperty = 'transform, opacity'
|
||||
element.style.transitionTimingFunction = runtime.options.easing
|
||||
element.style.transitionDuration = '0ms'
|
||||
element.style.willChange = 'transform'
|
||||
|
||||
runtime.cursorElement = element
|
||||
applyRuntimeVisualOptions()
|
||||
|
||||
return element
|
||||
}
|
||||
|
||||
function ensureCursorElement() {
|
||||
const existing = document.getElementById(CURSOR_ID)
|
||||
if (existing) {
|
||||
runtime.cursorElement = existing
|
||||
return existing
|
||||
}
|
||||
|
||||
const element = createCursorElement()
|
||||
runtime.cursorElement = element
|
||||
const root = document.documentElement || document.body
|
||||
root.appendChild(element)
|
||||
return element
|
||||
}
|
||||
|
||||
function applyRuntimeVisualOptions(): void {
|
||||
if (!runtime.cursorElement) {
|
||||
return
|
||||
}
|
||||
|
||||
const dimensions = getCursorDimensions()
|
||||
runtime.cursorElement.style.width = `${dimensions.width}px`
|
||||
runtime.cursorElement.style.height = `${dimensions.height}px`
|
||||
runtime.cursorElement.style.zIndex = `${runtime.options.zIndex}`
|
||||
runtime.cursorElement.style.transitionTimingFunction = runtime.options.easing
|
||||
|
||||
if (runtime.options.style === 'screenstudio') {
|
||||
runtime.cursorElement.style.borderRadius = '0'
|
||||
runtime.cursorElement.style.border = 'none'
|
||||
runtime.cursorElement.style.backgroundColor = 'transparent'
|
||||
runtime.cursorElement.style.backgroundImage = `url("${SCREENSTUDIO_POINTER_MACOS_TAHOE_DATA_URL}")`
|
||||
runtime.cursorElement.style.backgroundRepeat = 'no-repeat'
|
||||
runtime.cursorElement.style.backgroundPosition = 'left top'
|
||||
runtime.cursorElement.style.backgroundSize = 'contain'
|
||||
runtime.cursorElement.style.backdropFilter = 'none'
|
||||
runtime.cursorElement.style.filter = 'none'
|
||||
runtime.cursorElement.style.boxShadow = 'none'
|
||||
runtime.cursorElement.style.opacity = getBaseOpacity()
|
||||
return
|
||||
}
|
||||
|
||||
if (runtime.options.style === 'minimal') {
|
||||
const triangleSvg = `<svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" viewBox="0 0 24 24"><path fill="${runtime.options.color}" d="m23.284 19.124l-6.866-6.895a.4.4 0 0 1-.118-.296a.43.43 0 0 1 .163-.282l4.439-3.077a1.48 1.48 0 0 0 .621-1.48a1.48 1.48 0 0 0-1.036-1.198L1.623.302a1.14 1.14 0 0 0-1.11.282A1.13 1.13 0 0 0 .29 1.649L5.928 20.44a1.48 1.48 0 0 0 1.183 1.035a1.48 1.48 0 0 0 1.48-.621l3.078-4.44a.37.37 0 0 1 .31-.118a.43.43 0 0 1 .296.104l6.91 6.91a1.48 1.48 0 0 0 2.087 0l2.086-2.086a1.48 1.48 0 0 0-.074-2.101"/></svg>`
|
||||
const triangleDataUrl = `url("data:image/svg+xml,${encodeURIComponent(triangleSvg)}")`
|
||||
runtime.cursorElement.style.borderRadius = '0'
|
||||
runtime.cursorElement.style.border = 'none'
|
||||
runtime.cursorElement.style.backgroundColor = 'transparent'
|
||||
runtime.cursorElement.style.backgroundImage = triangleDataUrl
|
||||
runtime.cursorElement.style.backgroundRepeat = 'no-repeat'
|
||||
runtime.cursorElement.style.backgroundSize = 'contain'
|
||||
runtime.cursorElement.style.backgroundPosition = 'left top'
|
||||
runtime.cursorElement.style.backdropFilter = 'none'
|
||||
runtime.cursorElement.style.boxShadow = 'none'
|
||||
runtime.cursorElement.style.filter = 'drop-shadow(0 1px 1px rgba(0, 0, 0, 0.3))'
|
||||
runtime.cursorElement.style.opacity = getBaseOpacity()
|
||||
return
|
||||
}
|
||||
|
||||
runtime.cursorElement.style.borderRadius = '999px'
|
||||
runtime.cursorElement.style.border = 'none'
|
||||
runtime.cursorElement.style.backgroundColor = runtime.options.color
|
||||
runtime.cursorElement.style.backgroundImage = 'none'
|
||||
runtime.cursorElement.style.backdropFilter = 'none'
|
||||
runtime.cursorElement.style.filter = 'none'
|
||||
runtime.cursorElement.style.boxShadow = '0 2px 10px rgba(0, 0, 0, 0.18), inset 0 0 0 2px rgba(255, 255, 255, 0.55)'
|
||||
runtime.cursorElement.style.opacity = getBaseOpacity()
|
||||
}
|
||||
|
||||
function moveCursor(options: { x: number; y: number }): void {
|
||||
if (!runtime.enabled) {
|
||||
return
|
||||
}
|
||||
|
||||
const element = ensureCursorElement()
|
||||
const durationMs = computeDurationMs({ targetX: options.x, targetY: options.y })
|
||||
element.style.transitionDuration = `${Math.round(durationMs)}ms`
|
||||
|
||||
runtime.x = options.x
|
||||
runtime.y = options.y
|
||||
runtime.hasPosition = true
|
||||
applyTransform()
|
||||
}
|
||||
|
||||
function setPressed(options: { pressed: boolean }): void {
|
||||
if (!runtime.enabled) {
|
||||
return
|
||||
}
|
||||
|
||||
const element = ensureCursorElement()
|
||||
runtime.scale = options.pressed
|
||||
? runtime.options.style === 'screenstudio'
|
||||
? 0.94
|
||||
: runtime.options.style === 'minimal'
|
||||
? 0.93
|
||||
: 0.82
|
||||
: 1
|
||||
element.style.opacity = options.pressed ? '1' : getBaseOpacity()
|
||||
applyTransform()
|
||||
}
|
||||
|
||||
function enable(options?: GhostCursorClientOptions): void {
|
||||
runtime.options = mergeOptions(options)
|
||||
runtime.enabled = true
|
||||
ensureCursorElement()
|
||||
applyRuntimeVisualOptions()
|
||||
|
||||
if (!runtime.hasPosition) {
|
||||
runtime.x = Math.round(window.innerWidth / 2)
|
||||
runtime.y = Math.round(window.innerHeight / 2)
|
||||
runtime.scale = 1
|
||||
runtime.hasPosition = true
|
||||
applyTransform()
|
||||
}
|
||||
}
|
||||
|
||||
function disable(): void {
|
||||
runtime.enabled = false
|
||||
runtime.scale = 1
|
||||
runtime.hasPosition = false
|
||||
|
||||
if (runtime.cursorElement) {
|
||||
runtime.cursorElement.remove()
|
||||
runtime.cursorElement = null
|
||||
}
|
||||
}
|
||||
|
||||
function applyMouseAction(action: GhostCursorAction): void {
|
||||
if (!runtime.enabled) {
|
||||
return
|
||||
}
|
||||
|
||||
if (action.type === 'move' || action.type === 'wheel') {
|
||||
moveCursor({ x: action.x, y: action.y })
|
||||
return
|
||||
}
|
||||
|
||||
if (action.type === 'down') {
|
||||
moveCursor({ x: action.x, y: action.y })
|
||||
setPressed({ pressed: true })
|
||||
return
|
||||
}
|
||||
|
||||
if (action.type === 'up') {
|
||||
moveCursor({ x: action.x, y: action.y })
|
||||
setPressed({ pressed: false })
|
||||
}
|
||||
}
|
||||
|
||||
const api: GhostCursorApi = {
|
||||
enable,
|
||||
disable,
|
||||
applyMouseAction,
|
||||
isEnabled: () => {
|
||||
return runtime.enabled
|
||||
},
|
||||
}
|
||||
|
||||
globalThis.__playwriterGhostCursor = api
|
||||
|
||||
export {}
|
||||
@@ -0,0 +1,110 @@
|
||||
/**
|
||||
* Node-side ghost cursor helpers.
|
||||
* Injects the browser bundle and forwards mouse action events to the page overlay.
|
||||
*/
|
||||
|
||||
import fs from 'node:fs'
|
||||
import path from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import type { Page, MouseActionEvent } from '@xmorse/playwright-core'
|
||||
|
||||
export interface GhostCursorClientOptions {
|
||||
style?: 'minimal' | 'dot' | 'screenstudio'
|
||||
color?: string
|
||||
size?: number
|
||||
zIndex?: number
|
||||
easing?: string
|
||||
minDurationMs?: number
|
||||
maxDurationMs?: number
|
||||
speedPxPerMs?: number
|
||||
}
|
||||
|
||||
interface GhostCursorBrowserApi {
|
||||
enable: (options?: GhostCursorClientOptions) => void
|
||||
disable: () => void
|
||||
applyMouseAction: (event: MouseActionEvent) => void
|
||||
}
|
||||
|
||||
let ghostCursorCode: string | null = null
|
||||
|
||||
function getGhostCursorCode(): string {
|
||||
if (ghostCursorCode) {
|
||||
return ghostCursorCode
|
||||
}
|
||||
|
||||
const currentDir = path.dirname(fileURLToPath(import.meta.url))
|
||||
const bundlePath = path.join(currentDir, '..', 'dist', 'ghost-cursor-client.js')
|
||||
ghostCursorCode = fs.readFileSync(bundlePath, 'utf-8')
|
||||
return ghostCursorCode
|
||||
}
|
||||
|
||||
async function ensureGhostCursorInjected(options: { page: Page }): Promise<void> {
|
||||
const { page } = options
|
||||
const hasGhostCursor = await page.evaluate(() => {
|
||||
return Boolean((globalThis as { __playwriterGhostCursor?: unknown }).__playwriterGhostCursor)
|
||||
})
|
||||
|
||||
if (hasGhostCursor) {
|
||||
return
|
||||
}
|
||||
|
||||
const code = getGhostCursorCode()
|
||||
await page.evaluate(code)
|
||||
}
|
||||
|
||||
export async function enableGhostCursor(options: {
|
||||
page: Page
|
||||
cursorOptions?: GhostCursorClientOptions
|
||||
}): Promise<void> {
|
||||
const { page, cursorOptions } = options
|
||||
await ensureGhostCursorInjected({ page })
|
||||
|
||||
await page.evaluate(
|
||||
({ optionsFromNode }) => {
|
||||
const api = (globalThis as { __playwriterGhostCursor?: GhostCursorBrowserApi }).__playwriterGhostCursor
|
||||
api?.enable(optionsFromNode)
|
||||
},
|
||||
{ optionsFromNode: cursorOptions },
|
||||
)
|
||||
}
|
||||
|
||||
export async function disableGhostCursor(options: { page: Page }): Promise<void> {
|
||||
const { page } = options
|
||||
await page.evaluate(() => {
|
||||
const api = (globalThis as { __playwriterGhostCursor?: GhostCursorBrowserApi }).__playwriterGhostCursor
|
||||
api?.disable()
|
||||
})
|
||||
}
|
||||
|
||||
export async function applyGhostCursorMouseAction(options: {
|
||||
page: Page
|
||||
event: MouseActionEvent
|
||||
}): Promise<void> {
|
||||
const { page, event } = options
|
||||
|
||||
const applied = await page.evaluate(
|
||||
({ serializedEvent }) => {
|
||||
const api = (globalThis as { __playwriterGhostCursor?: GhostCursorBrowserApi }).__playwriterGhostCursor
|
||||
if (!api) {
|
||||
return false
|
||||
}
|
||||
|
||||
api.applyMouseAction(serializedEvent)
|
||||
return true
|
||||
},
|
||||
{ serializedEvent: event },
|
||||
)
|
||||
|
||||
if (applied) {
|
||||
return
|
||||
}
|
||||
|
||||
await ensureGhostCursorInjected({ page })
|
||||
await page.evaluate(
|
||||
({ serializedEvent }) => {
|
||||
const api = (globalThis as { __playwriterGhostCursor?: GhostCursorBrowserApi }).__playwriterGhostCursor
|
||||
api?.applyMouseAction(serializedEvent)
|
||||
},
|
||||
{ serializedEvent: event },
|
||||
)
|
||||
}
|
||||
@@ -4,9 +4,10 @@
|
||||
* and locator-initiated actions like page.locator().click().
|
||||
*/
|
||||
import { chromium } from '@xmorse/playwright-core'
|
||||
import type { MouseActionEvent } from '@xmorse/playwright-core'
|
||||
import type { MouseActionEvent, Page } from '@xmorse/playwright-core'
|
||||
import { describe, it, expect, beforeAll, afterAll } from 'vitest'
|
||||
import { getCdpUrl } from './utils.js'
|
||||
import { enableGhostCursor, applyGhostCursorMouseAction, disableGhostCursor } from './ghost-cursor.js'
|
||||
import {
|
||||
setupTestContext,
|
||||
cleanupTestContext,
|
||||
@@ -113,6 +114,62 @@ describe('onMouseAction callback', () => {
|
||||
await safeCloseCDPBrowser(directBrowser)
|
||||
}, 30000)
|
||||
|
||||
it('should animate ghost cursor from onMouseAction callback', async () => {
|
||||
const browserContext = testCtx!.browserContext
|
||||
|
||||
const directBrowser = await chromium.connectOverCDP(getCdpUrl({ port: TEST_PORT }))
|
||||
let targetPage: Page | null = null
|
||||
try {
|
||||
const contexts = directBrowser.contexts()
|
||||
const pages = contexts[0].pages()
|
||||
targetPage = pages.find((p) => p.url().startsWith('data:'))!
|
||||
expect(targetPage).toBeDefined()
|
||||
const pageForTest = targetPage
|
||||
|
||||
await enableGhostCursor({ page: pageForTest })
|
||||
pageForTest.onMouseAction = async (event) => {
|
||||
await applyGhostCursorMouseAction({ page: pageForTest, event })
|
||||
}
|
||||
|
||||
await pageForTest.mouse.click(140, 120)
|
||||
|
||||
const cursorState = await pageForTest.evaluate(() => {
|
||||
const cursorElement = document.getElementById('__playwriter_ghost_cursor__')
|
||||
if (!cursorElement) {
|
||||
return { exists: false, transform: '' }
|
||||
}
|
||||
return {
|
||||
exists: true,
|
||||
transform: cursorElement.getAttribute('style') || '',
|
||||
}
|
||||
})
|
||||
|
||||
expect(cursorState.exists).toBe(true)
|
||||
const translateMatch = cursorState.transform.match(/translate3d\(([-\d.]+)px, ([-\d.]+)px, 0px\)/)
|
||||
expect(translateMatch).toBeTruthy()
|
||||
const translateX = Number(translateMatch![1])
|
||||
const translateY = Number(translateMatch![2])
|
||||
// Screen Studio cursor uses a hotspot offset, so CSS position is slightly above/left of click target.
|
||||
expect(translateX).toBeLessThanOrEqual(140)
|
||||
expect(translateY).toBeLessThanOrEqual(120)
|
||||
expect(translateX).toBeGreaterThan(120)
|
||||
expect(translateY).toBeGreaterThan(100)
|
||||
|
||||
await disableGhostCursor({ page: pageForTest })
|
||||
|
||||
const hasGhostCursor = await pageForTest.evaluate(() => {
|
||||
return Boolean(document.getElementById('__playwriter_ghost_cursor__'))
|
||||
})
|
||||
expect(hasGhostCursor).toBe(false)
|
||||
} finally {
|
||||
if (targetPage) {
|
||||
targetPage.onMouseAction = null
|
||||
await disableGhostCursor({ page: targetPage })
|
||||
}
|
||||
await safeCloseCDPBrowser(directBrowser)
|
||||
}
|
||||
}, 30000)
|
||||
|
||||
it('should not fire when onMouseAction is set to null', async () => {
|
||||
const browserContext = testCtx!.browserContext
|
||||
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
/**
|
||||
* Encapsulates ghost cursor lifecycle for recording sessions.
|
||||
* Keeps onMouseAction chaining/restoration isolated from executor logic.
|
||||
*/
|
||||
|
||||
import type { BrowserContext, Page } from '@xmorse/playwright-core'
|
||||
import { applyGhostCursorMouseAction, disableGhostCursor, enableGhostCursor, type GhostCursorClientOptions } from './ghost-cursor.js'
|
||||
|
||||
interface RecordingGhostCursorLogger {
|
||||
error: (...args: unknown[]) => void
|
||||
}
|
||||
|
||||
interface RecordingTargetOptions {
|
||||
page?: Page
|
||||
sessionId?: string
|
||||
}
|
||||
|
||||
export class RecordingGhostCursorController {
|
||||
private readonly previousMouseActionByPage = new WeakMap<Page, Page['onMouseAction']>()
|
||||
private readonly logger: RecordingGhostCursorLogger
|
||||
|
||||
constructor(options: { logger: RecordingGhostCursorLogger }) {
|
||||
this.logger = options.logger
|
||||
}
|
||||
|
||||
resolveRecordingTargetPage(options: {
|
||||
context: BrowserContext
|
||||
defaultPage: Page
|
||||
target?: RecordingTargetOptions
|
||||
}): Page {
|
||||
const { context, defaultPage, target } = options
|
||||
|
||||
if (target?.page) {
|
||||
return target.page
|
||||
}
|
||||
|
||||
if (target?.sessionId) {
|
||||
const pageForSession = context.pages().find((candidatePage) => {
|
||||
return candidatePage.sessionId() === target.sessionId
|
||||
})
|
||||
|
||||
if (pageForSession) {
|
||||
return pageForSession
|
||||
}
|
||||
}
|
||||
|
||||
return defaultPage
|
||||
}
|
||||
|
||||
async enableForRecording(options: { page: Page }): Promise<void> {
|
||||
const { page } = options
|
||||
|
||||
try {
|
||||
await enableGhostCursor({ page })
|
||||
if (!this.previousMouseActionByPage.has(page)) {
|
||||
this.previousMouseActionByPage.set(page, page.onMouseAction)
|
||||
}
|
||||
|
||||
const previousMouseAction = this.previousMouseActionByPage.get(page)
|
||||
page.onMouseAction = async (event) => {
|
||||
void applyGhostCursorMouseAction({ page, event }).catch((error) => {
|
||||
this.logger.error('[playwriter] Failed to apply ghost cursor action', error)
|
||||
})
|
||||
|
||||
if (!previousMouseAction) {
|
||||
return
|
||||
}
|
||||
|
||||
await previousMouseAction(event)
|
||||
}
|
||||
} catch (error) {
|
||||
page.onMouseAction = this.previousMouseActionByPage.get(page) ?? null
|
||||
this.previousMouseActionByPage.delete(page)
|
||||
this.logger.error('[playwriter] Failed to enable ghost cursor', error)
|
||||
}
|
||||
}
|
||||
|
||||
async disableForRecording(options: { page: Page }): Promise<void> {
|
||||
const { page } = options
|
||||
page.onMouseAction = this.previousMouseActionByPage.get(page) ?? null
|
||||
this.previousMouseActionByPage.delete(page)
|
||||
|
||||
try {
|
||||
await disableGhostCursor({ page })
|
||||
} catch (error) {
|
||||
this.logger.error('[playwriter] Failed to disable ghost cursor', error)
|
||||
}
|
||||
}
|
||||
|
||||
async show(options: { page: Page; cursorOptions?: GhostCursorClientOptions }): Promise<void> {
|
||||
const { page, cursorOptions } = options
|
||||
await enableGhostCursor({ page, cursorOptions })
|
||||
}
|
||||
|
||||
async hide(options: { page: Page }): Promise<void> {
|
||||
const { page } = options
|
||||
await disableGhostCursor({ page })
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,15 @@ const __dirname = path.dirname(__filename)
|
||||
|
||||
export const RELAY_PORT = Number(process.env.PLAYWRITER_PORT) || 19988
|
||||
|
||||
export type ExtensionStatus = {
|
||||
extensionId: string
|
||||
stableKey?: string
|
||||
browser: string | null
|
||||
profile: { email: string; id: string } | null
|
||||
activeTargets: number
|
||||
playwriterVersion: string | null
|
||||
}
|
||||
|
||||
export async function getRelayServerVersion(port: number = RELAY_PORT): Promise<string | null> {
|
||||
try {
|
||||
const response = await fetch(`http://127.0.0.1:${port}/version`, {
|
||||
@@ -46,33 +55,81 @@ export async function getExtensionStatus(
|
||||
}
|
||||
}
|
||||
|
||||
export async function getExtensionsStatus(port: number = RELAY_PORT): Promise<ExtensionStatus[]> {
|
||||
try {
|
||||
const response = await fetch(`http://127.0.0.1:${port}/extensions/status`, {
|
||||
signal: AbortSignal.timeout(2000),
|
||||
})
|
||||
if (!response.ok) {
|
||||
const fallback = await fetch(`http://127.0.0.1:${port}/extension/status`, {
|
||||
signal: AbortSignal.timeout(2000),
|
||||
})
|
||||
if (!fallback.ok) {
|
||||
return []
|
||||
}
|
||||
|
||||
const fallbackData = (await fallback.json()) as {
|
||||
connected: boolean
|
||||
activeTargets: number
|
||||
browser: string | null
|
||||
profile: { email: string; id: string } | null
|
||||
playwriterVersion?: string | null
|
||||
}
|
||||
|
||||
if (!fallbackData?.connected) {
|
||||
return []
|
||||
}
|
||||
|
||||
return [
|
||||
{
|
||||
extensionId: 'default',
|
||||
stableKey: undefined,
|
||||
browser: fallbackData.browser,
|
||||
profile: fallbackData.profile,
|
||||
activeTargets: fallbackData.activeTargets,
|
||||
playwriterVersion: fallbackData.playwriterVersion || null,
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
const data = (await response.json()) as {
|
||||
extensions: ExtensionStatus[]
|
||||
}
|
||||
|
||||
return data.extensions || []
|
||||
} catch {
|
||||
return []
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for the extension to connect to the relay server.
|
||||
* Returns true if connected within timeout, false otherwise.
|
||||
* Wait for at least one extension to appear in extensions status.
|
||||
* Returns connected extension entries, or [] on timeout.
|
||||
*/
|
||||
export async function waitForExtension(
|
||||
export async function waitForConnectedExtensions(
|
||||
options: {
|
||||
port?: number
|
||||
timeoutMs?: number
|
||||
pollIntervalMs?: number
|
||||
logger?: { log: (...args: any[]) => void }
|
||||
} = {},
|
||||
): Promise<boolean> {
|
||||
const { port = RELAY_PORT, timeoutMs = 5000, logger } = options
|
||||
): Promise<ExtensionStatus[]> {
|
||||
const { port = RELAY_PORT, timeoutMs = 5000, pollIntervalMs = 200, logger } = options
|
||||
const startTime = Date.now()
|
||||
|
||||
logger?.log(pc.dim('Waiting for extension to connect...'))
|
||||
|
||||
while (Date.now() - startTime < timeoutMs) {
|
||||
const status = await getExtensionStatus(port)
|
||||
if (status?.connected) {
|
||||
const extensions = await getExtensionsStatus(port)
|
||||
if (extensions.length > 0) {
|
||||
logger?.log(pc.green('Extension connected'))
|
||||
return true
|
||||
return extensions
|
||||
}
|
||||
await sleep(200)
|
||||
await sleep(pollIntervalMs)
|
||||
}
|
||||
|
||||
logger?.log(pc.yellow('Extension did not connect within timeout'))
|
||||
return false
|
||||
return []
|
||||
}
|
||||
|
||||
async function killRelayServer(options: { port: number; waitForFreeMs?: number }): Promise<void> {
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
|
||||
import os from 'node:os'
|
||||
import path from 'node:path'
|
||||
import type { Page } from '@xmorse/playwright-core'
|
||||
import type { BrowserContext, Page } from '@xmorse/playwright-core'
|
||||
import type {
|
||||
StartRecordingResult,
|
||||
StopRecordingResult,
|
||||
@@ -16,6 +16,7 @@ import type {
|
||||
CancelRecordingResult,
|
||||
} from './protocol.js'
|
||||
import { EXTENSION_IDS } from './utils.js'
|
||||
import { RecordingGhostCursorController } from './recording-ghost-cursor.js'
|
||||
|
||||
/**
|
||||
* Generate a shell command to quit and restart Chrome with flags that allow automatic tab capture.
|
||||
@@ -86,6 +87,132 @@ export interface RecordingState {
|
||||
tabId?: number
|
||||
}
|
||||
|
||||
export interface ExecutionTimestamp {
|
||||
start: number
|
||||
end: number
|
||||
}
|
||||
|
||||
interface RecordingTargetOptions {
|
||||
page?: Page
|
||||
sessionId?: string
|
||||
}
|
||||
|
||||
interface CreateRecordingApiOptions {
|
||||
context: BrowserContext
|
||||
defaultPage: Page
|
||||
relayPort: number
|
||||
ghostCursorController: RecordingGhostCursorController
|
||||
onStart: () => void
|
||||
onFinish: () => void
|
||||
getExecutionTimestamps: () => ExecutionTimestamp[]
|
||||
}
|
||||
|
||||
interface StartRecordingWithDefaultsOptions extends Omit<StartRecordingOptions, 'relayPort'> {}
|
||||
interface StopRecordingWithDefaultsOptions extends Omit<StopRecordingOptions, 'relayPort'> {}
|
||||
interface IsRecordingWithDefaultsOptions {
|
||||
page?: Page
|
||||
sessionId?: string
|
||||
}
|
||||
interface CancelRecordingWithDefaultsOptions {
|
||||
page?: Page
|
||||
sessionId?: string
|
||||
}
|
||||
|
||||
function resolveRecordingTargetPage(options: {
|
||||
context: BrowserContext
|
||||
defaultPage: Page
|
||||
ghostCursorController: RecordingGhostCursorController
|
||||
target?: RecordingTargetOptions
|
||||
}): Page {
|
||||
return options.ghostCursorController.resolveRecordingTargetPage({
|
||||
context: options.context,
|
||||
defaultPage: options.defaultPage,
|
||||
target: options.target,
|
||||
})
|
||||
}
|
||||
|
||||
function withRecordingDefaults<T extends { page?: Page; sessionId?: string }, R>(options: {
|
||||
relayPort: number
|
||||
defaultPage: Page
|
||||
fn: (opts: T & { relayPort: number; sessionId?: string }) => Promise<R>
|
||||
}): (input?: T) => Promise<R> {
|
||||
const { relayPort, defaultPage, fn } = options
|
||||
return async (input: T = {} as T) => {
|
||||
const targetPage = input.page || defaultPage
|
||||
const sessionId = input.sessionId || targetPage.sessionId() || undefined
|
||||
return fn({ page: targetPage, sessionId, relayPort, ...input })
|
||||
}
|
||||
}
|
||||
|
||||
export function createRecordingApi(options: CreateRecordingApiOptions): {
|
||||
start: (opts?: StartRecordingWithDefaultsOptions) => Promise<RecordingState>
|
||||
stop: (opts?: StopRecordingWithDefaultsOptions) => Promise<{ path: string; duration: number; size: number; executionTimestamps: ExecutionTimestamp[] }>
|
||||
isRecording: (opts?: IsRecordingWithDefaultsOptions) => Promise<RecordingState>
|
||||
cancel: (opts?: CancelRecordingWithDefaultsOptions) => Promise<void>
|
||||
} {
|
||||
const { context, defaultPage, relayPort, ghostCursorController, onStart, onFinish, getExecutionTimestamps } = options
|
||||
|
||||
const startWithDefaults = withRecordingDefaults<StartRecordingWithDefaultsOptions, RecordingState>({
|
||||
relayPort,
|
||||
defaultPage,
|
||||
fn: startRecording,
|
||||
})
|
||||
const stopWithDefaults = withRecordingDefaults<StopRecordingWithDefaultsOptions, { path: string; duration: number; size: number }>({
|
||||
relayPort,
|
||||
defaultPage,
|
||||
fn: stopRecording,
|
||||
})
|
||||
const isRecordingWithDefaults = async (opts: IsRecordingWithDefaultsOptions = {}): Promise<RecordingState> => {
|
||||
const targetPage = opts.page || defaultPage
|
||||
const sessionId = opts.sessionId || targetPage.sessionId() || undefined
|
||||
return isRecording({ page: targetPage, sessionId, relayPort })
|
||||
}
|
||||
|
||||
const cancelWithDefaults = async (opts: CancelRecordingWithDefaultsOptions = {}): Promise<void> => {
|
||||
const targetPage = opts.page || defaultPage
|
||||
const sessionId = opts.sessionId || targetPage.sessionId() || undefined
|
||||
await cancelRecording({ page: targetPage, sessionId, relayPort })
|
||||
}
|
||||
|
||||
const start = async (opts?: StartRecordingWithDefaultsOptions): Promise<RecordingState> => {
|
||||
const targetPage = resolveRecordingTargetPage({ context, defaultPage, ghostCursorController, target: opts })
|
||||
const result = await startWithDefaults(opts)
|
||||
onStart()
|
||||
await ghostCursorController.enableForRecording({ page: targetPage })
|
||||
return result
|
||||
}
|
||||
|
||||
const stop = async (
|
||||
opts?: StopRecordingWithDefaultsOptions,
|
||||
): Promise<{ path: string; duration: number; size: number; executionTimestamps: ExecutionTimestamp[] }> => {
|
||||
const targetPage = resolveRecordingTargetPage({ context, defaultPage, ghostCursorController, target: opts })
|
||||
try {
|
||||
const result = await stopWithDefaults(opts)
|
||||
return { ...result, executionTimestamps: [...getExecutionTimestamps()] }
|
||||
} finally {
|
||||
onFinish()
|
||||
await ghostCursorController.disableForRecording({ page: targetPage })
|
||||
}
|
||||
}
|
||||
|
||||
const cancel = async (opts?: CancelRecordingWithDefaultsOptions): Promise<void> => {
|
||||
const targetPage = resolveRecordingTargetPage({ context, defaultPage, ghostCursorController, target: opts })
|
||||
try {
|
||||
await cancelWithDefaults(opts)
|
||||
} finally {
|
||||
onFinish()
|
||||
await ghostCursorController.disableForRecording({ page: targetPage })
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
start,
|
||||
stop,
|
||||
isRecording: isRecordingWithDefaults,
|
||||
cancel,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Start recording the page.
|
||||
* The recording is handled by the extension, so it survives page navigation.
|
||||
|
||||
+28
-11
@@ -861,13 +861,15 @@ await screenshotWithAccessibilityLabels({ page: state.page })
|
||||
|
||||
Labels are color-coded: yellow=links, orange=buttons, coral=inputs, pink=checkboxes, peach=sliders, salmon=menus, amber=tabs.
|
||||
|
||||
**startRecording / stopRecording** - record the page as a video at native FPS (30-60fps). Uses `chrome.tabCapture` in the extension context, so **recording survives page navigation**. Video is saved as mp4.
|
||||
**recording.start / recording.stop** - record the page as a video at native FPS (30-60fps). Uses `chrome.tabCapture` in the extension context, so **recording survives page navigation**. Video is saved as mp4.
|
||||
|
||||
While recording is active, Playwriter automatically overlays a smooth ghost cursor that follows automated mouse actions (`page.mouse.*`, `locator.click()`, hover flows) using `page.onMouseAction` from the Playwright fork.
|
||||
|
||||
**Note**: Recording requires the user to have clicked the Playwriter extension icon on the tab. This grants `activeTab` permission needed for `chrome.tabCapture`. Recording works on tabs where the icon was clicked - if you need to record a new tab, ask the user to click the icon on it first.
|
||||
|
||||
```js
|
||||
// Start recording - outputPath must be specified upfront
|
||||
await startRecording({
|
||||
await recording.start({
|
||||
page: state.page,
|
||||
outputPath: './recording.mp4',
|
||||
frameRate: 30, // default: 30
|
||||
@@ -881,7 +883,7 @@ await state.page.waitForLoadState('domcontentloaded')
|
||||
await state.page.goBack()
|
||||
|
||||
// Stop and get result
|
||||
const { path, duration, size } = await stopRecording({ page: state.page })
|
||||
const { path, duration, size } = await recording.stop({ page: state.page })
|
||||
console.log(`Saved ${size} bytes, duration: ${duration}ms`)
|
||||
```
|
||||
|
||||
@@ -889,17 +891,32 @@ Additional recording utilities:
|
||||
|
||||
```js
|
||||
// Check if recording is active
|
||||
const { isRecording, startedAt } = await isRecording({ page: state.page })
|
||||
const { isRecording, startedAt } = await recording.isRecording({ page: state.page })
|
||||
|
||||
// Cancel recording without saving
|
||||
await cancelRecording({ page: state.page })
|
||||
await recording.cancel({ page: state.page })
|
||||
```
|
||||
|
||||
**ghostCursor.show / ghostCursor.hide** - manually show or hide the in-page cursor overlay. Useful for screenshots and demos even when recording is not running.
|
||||
|
||||
```js
|
||||
// Show cursor in the center (or keep current position if already visible)
|
||||
await ghostCursor.show({ page: state.page })
|
||||
|
||||
// Optional styles: 'minimal' (default triangular pointer), 'dot', 'screenstudio'
|
||||
await ghostCursor.show({ page: state.page, style: 'minimal' })
|
||||
|
||||
// Hide cursor overlay
|
||||
await ghostCursor.hide({ page: state.page })
|
||||
```
|
||||
|
||||
`startRecording`, `stopRecording`, `isRecording`, and `cancelRecording` remain available as backward-compatible aliases.
|
||||
|
||||
**Key difference from getDisplayMedia**: This approach uses `chrome.tabCapture` which runs in the extension context, not the page. The recording persists across navigations because the extension holds the `MediaRecorder`, not the page's JavaScript context.
|
||||
|
||||
**createDemoVideo** - create a polished demo video from a recording by automatically speeding up idle sections (time between execute() calls) while keeping interactions at normal speed. Useful for creating demo videos of agent workflows without long pauses.
|
||||
|
||||
While recording is active, playwriter tracks when each `execute()` call starts and ends. `stopRecording()` returns these timestamps alongside the video file. `createDemoVideo` uses this data to identify idle gaps and speed them up with ffmpeg in a single pass.
|
||||
While recording is active, playwriter tracks when each `execute()` call starts and ends. `recording.stop()` returns these timestamps alongside the video file. `createDemoVideo` uses this data to identify idle gaps and speed them up with ffmpeg in a single pass.
|
||||
|
||||
A 1-second buffer is preserved around each interaction so viewers see context before and after each action.
|
||||
|
||||
@@ -907,7 +924,7 @@ Requires `ffmpeg` and `ffprobe` installed on the system.
|
||||
|
||||
```js
|
||||
// Start recording
|
||||
await startRecording({ page: state.page, outputPath: './recording.mp4' })
|
||||
await recording.start({ page: state.page, outputPath: './recording.mp4' })
|
||||
```
|
||||
|
||||
```js
|
||||
@@ -917,13 +934,13 @@ await startRecording({ page: state.page, outputPath: './recording.mp4' })
|
||||
|
||||
```js
|
||||
// Stop recording — executionTimestamps is included in the result
|
||||
const recording = await stopRecording({ page: state.page })
|
||||
const recordingResult = await recording.stop({ page: state.page })
|
||||
|
||||
// Create demo video — idle gaps are sped up 4x (default)
|
||||
const demoPath = await createDemoVideo({
|
||||
recordingPath: recording.path,
|
||||
durationMs: recording.duration,
|
||||
executionTimestamps: recording.executionTimestamps,
|
||||
recordingPath: recordingResult.path,
|
||||
durationMs: recordingResult.duration,
|
||||
executionTimestamps: recordingResult.executionTimestamps,
|
||||
speed: 5, // optional, default 5x for idle sections
|
||||
// outputFile: './demo.mp4', // optional, defaults to recording-demo.mp4
|
||||
})
|
||||
|
||||
@@ -6,5 +6,5 @@
|
||||
"types": ["node", "chrome"]
|
||||
},
|
||||
"include": ["src"],
|
||||
"exclude": ["src/a11y-client.ts"]
|
||||
"exclude": ["src/a11y-client.ts", "src/ghost-cursor-client.ts"]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user