Computer-Use and Browser Agent Systems

Treat the interface as a feedback-controlled environment

A computer-use run is not a generated script that can be trusted from start to finish. It is a sequence of bounded decisions over a changing environment.

Keep four records separate

• Intent: what the user directly asked the system to accomplish.
• Proposal: what the model wants the executor to do next.
• Decision: why trusted policy allowed, denied, or paused that proposal.
• Observation: what the environment actually showed or changed after execution.

Page text, emails, documents, and tool results can influence a proposal, but they are third-party content rather than user permission.

Observation is evidence with blind spots

No single representation is best for every interface. A production harness usually combines several channels and records which one grounded the target.

Freshness and identity matter more than coordinates

• Re-observe immediately before consequential actions; do not reuse an old screenshot after navigation or a long model turn.
• Prefer a role and accessible name, explicit test identifier, or application contract over a long CSS path or guessed coordinate.
• Require one unique target. Multiple matches are ambiguity, not an invitation to choose the first element.
• Record the window, frame, URL, account, and observation version alongside the target.
• Use pixels when they add evidence, not as proof that the selected element owns the intended business action.

Playwright's official guidance recommends user-facing locators and re-resolves a locator before each action. Its actionability checks also verify conditions such as visibility, stability, enabled state, and whether the element can receive events.

Ground one next action before committing to a trajectory

The planning decision is not merely which button to click. The harness must decide whether the available observation channels identify the target, whether the state can change before execution, and how soon to verify.

Use the workbench to compare a semantic form, a volatile checkout, and a canvas application. Notice that a three-action macro can turn good evidence into a stale plan before its final step.

Express the next step as an action contract

An action contract gives deterministic code enough information to reject an unsafe or ungrounded proposal. Keep it smaller than a full task plan.

Proposal fields

• Observation ID and environment identity used to ground the decision.
• Action type, unique target description, and bounded input.
• Expected postcondition stated in observable terms.
• Risk class, data sensitivity, and estimated external effect.
• Maximum wait and the recovery rule for an ambiguous result.

Gate checks

• The observation is still current and belongs to the expected window, account, and origin.
• The target resolves uniquely and passes actionability checks.
• Direct user intent covers the target, data, and external effect.
• The policy allows this action class in the current sandbox and network scope.
• Approval is obtained at the point of risk, not inferred from page content or an earlier broad confirmation.

The runnable example rejects stale observations, non-unique targets, unsupported macros, and missing approval before returning one executable action.

Separate the agent loop into owned boundaries

Each stage should have a clear owner and durable evidence. The model can help interpret observations and propose actions, but it should not silently absorb policy or execution responsibilities.

1. 1

   Observe

   Capture the environment

   Collect a screenshot and available semantics with a monotonic observation ID, timestamp, window identity, URL, account, and redaction policy.

2. 2

   Propose

   Select one bounded action

   Identify the target, supporting evidence, expected postcondition, and what would make the proposal invalid before execution.

3. 3

   Decide

   Evaluate authority

   Apply allowlists, identity, origin, data sensitivity, action risk, budgets, and current user approval outside the model.

4. 4

   Attempt

   Execute in isolation

   Use a browser, container, or VM with minimal privileges, restricted network access, scoped credentials, and controlled file exchange.

5. 5

   Verify

   Check the postcondition

   Re-observe the interface and query authoritative state for consequential writes. Record completed, rejected, failed, or ambiguous.

6. 6

   Resolve

   Continue or recover

   Advance only from verified evidence. Otherwise re-plan, reconcile, ask the user, compensate, or stop according to the failure boundary.

Bound the loop with maximum steps, wall-clock time, model spend, retries, downloads, uploads, and external writes. A loop that cannot explain why it is still running should stop.

Permission follows the external effect

Permission is specific to an actor, target, data item, action, and time. A user asking an agent to find a draft does not automatically authorize sending it, and a page telling the agent to upload a credential is not user intent.

Ask late enough to be specific

• Complete safe preparation first, then pause immediately before the external effect.
• State the target, recipient, data, amount, permanence, and reason for the confirmation.
• Reconfirm when those details change after approval.
• Never treat a screenshot, website, email, PDF, chat, or tool result as permission.
• Prefer user takeover when policy requires direct human interaction with the final control.

Resolve ambiguity before choosing a recovery action

A timeout or lost screenshot says the harness did not receive enough evidence. It does not prove the action failed. Recovery must consider permission, reversibility, and an authoritative source of truth.

The lab starts with an unsafe immediate retry. Compare it with reconciliation, a precise user question, and compensation. The correct response changes with the failed boundary.

Model ambiguous completion as a state, not an exception string

Use at least four terminal states for each attempt:

• Completed: authoritative evidence proves the intended effect happened once.
• Rejected: policy or the environment prevented execution before the effect.
• Failed: evidence proves the effect did not happen and the failure is safe to retry.
• Ambiguous: execution may have crossed the commit boundary, but confirmation was lost or contradictory.

For ambiguous writes, reconcile by operation ID, draft ID, message ID, transaction state, or another system-of-record key before retrying. Use idempotency keys when the dependency supports them, but do not assume a graphical click is idempotent merely because the button looks unchanged.

The example below simulates a send that commits before its response is lost. Reconciliation finds the committed operation and prevents a duplicate send.

Evaluate trajectories and control decisions separately

End-to-end task success alone can hide unsafe near misses, unnecessary steps, and lucky outcomes. Evaluate repeated executions from controlled initial states and score both the result and the control plane.

Offline evaluation slices

• Stable and changing layouts, overlays, scroll regions, pop-ups, multiple windows, and native dialogs.
• Pixel-only, semantic-only, and hybrid interfaces with ambiguous or duplicate targets.
• Benign content, prompt injection, phishing, unexpected warnings, and cross-origin navigation.
• Approval required, approval expired, target changed after approval, and missing direct intent.
• Failure before commit, failure after commit, duplicate delivery, stale observation, and compensation failure.

Production signals

Also track steps per completed task, stale-observation rejections, policy denials, prompt-injection stops, duplicate side effects, user takeovers, latency by loop stage, and retained-data volume. Segment all metrics by application, task class, model and prompt version, observation strategy, and risk tier.

Production readiness review

• Define the exact sites, applications, accounts, files, actions, and network destinations the agent may reach.
• Keep proposal, policy decision, execution, and verification in separate records with stable task and action IDs.
• Re-observe before each action and reject stale, non-unique, hidden, disabled, or origin-mismatched targets.
• Require current approval immediately before transmitting data or creating a consequential external effect.
• Prefer structured APIs for durable writes and authoritative verification when available.
• Make ambiguous completion explicit; reconcile before retrying and use idempotency where the dependency supports it.
• Isolate the runtime, remove ambient credentials, redact artifacts, and expire the environment after the task.
• Enforce step, time, token, spend, download, upload, and side-effect budgets with a kill switch.
• Test prompt injection, layout changes, expired sessions, changed approval scope, lost responses, and repeated execution.
• Give the user a readable trace of what was observed, proposed, approved, executed, and verified.

Primary sources and version notes

Vendor tool envelopes and model capabilities evolve quickly. The architecture in this lesson is provider-neutral; verify current action schemas and safety behavior against the runtime you deploy.

• OpenAI computer-use guide for screenshot-action loops, isolation, untrusted content, and confirmation at the point of risk.
• Anthropic computer-use tool documentation for the client-side agent loop, sandboxing guidance, and prompt-injection precautions.
• Playwright locator guidance for user-facing target contracts and fresh locator resolution.
• Playwright actionability checks for visibility, stability, enabled state, and event-receiving checks before UI actions.
• OSWorld, NeurIPS 2024 for execution-based evaluation of open-ended tasks in real computer environments.