CUA Architecture

Computer Use Agent — screenshot, reason, act, verify. Every browser action passes through deontic safety gates with human-in-the-loop confirmation before form submission.

Overview

The Computer Use Agent (CUA) automates browser-based tasks on EU institutional portals — such as TED eProcurement, EUR-Lex, and national law databases — while maintaining strict safety controls. Unlike unguarded browser automation, every CUA action passes through deontic safety gates before execution. Destructive actions (form submissions, data entry) require explicit human approval.

The CUA operates in a continuous loop: it observes the current browser state via screenshot, reasons about the next action using an AI model, executes the action (subject to safety gates), and verifies the result. This loop continues until the task is complete or the user intervenes.

The CUA loop

Step-by-step

1. Screenshot. The CUA captures a screenshot of the current browser viewport. This is the agent’s “eyes” — it has no DOM access, only pixel-level observation.
2. Reason. An AI model analyses the screenshot together with the task description and action history. It determines the next action: what to click, what to type, where to scroll, or whether the task is complete.
3. Deontic safety gate. Before execution, the proposed action is classified against the four deontic modalities (see below). Prohibited actions are blocked. Obligatory confirmations pause for human approval.
4. Act. The action is executed in the browser: a click at specific coordinates, text entry into a field, scrolling, navigation, or form submission.
5. Verify. A new screenshot is taken. The CUA compares the result against expected state. If the action failed or produced unexpected results, the loop can retry or escalate to the user.

Deontic safety gates

Every action passes through deontic safety gates before execution, following IEC 62443-3-3 zone-based security principles. The four modalities control what the CUA may, must, must not, and need not do.

Human-in-the-loop flow

When a CUA action triggers an M2 Obligation gate (typically form submission), the system pauses and presents a confirmation modal. The user sees:

• All fields the CUA has filled, with their values
• The target form and portal
• Three options: Approve (proceed with submission), Edit (modify field values before submission), or Cancel (abort the action)

The CUA does not resume until the user makes an explicit choice. This ensures no data is submitted to external portals without human review.

Action types

The CUA supports 7 browser action types. Each maps to a specific deontic gate level.

Gate classification can be elevated by context. For example, a click on a “Submit” button is reclassified from M3 to M2. A navigate to a domain not on the allowlist is reclassified from M3 to M1 (blocked).

Action lifecycle

Each action progresses through a defined state machine with 7 possible statuses.

Architecture components

CuaActionOverlay

CuaSessionPanel

CuaConfirmationModal

CuaReplayViewer

CuaDashboard

SSE event stream

The CUA executor streams real-time events to the frontend via Server-Sent Events (SSE). The connection reconnects automatically with exponential backoff (maximum 5 attempts).

Event types

Connection example

const source = new EventSource('/v1/cua/stream?session=' + sessionId);

source.addEventListener('action_pending', (e) => {
  const action = JSON.parse(e.data);
  // Show CuaActionOverlay at action.coordinates
});

source.addEventListener('form_confirmation', (e) => {
  const form = JSON.parse(e.data);
  // Show CuaConfirmationModal with form.fields
});

source.addEventListener('action_completed', (e) => {
  const result = JSON.parse(e.data);
  // Update CuaSessionPanel with completed step
});

Session model

Each CUA session targets a specific portal and task. Sessions maintain full state across the action lifecycle.

Session properties

Session example

{
  "id": "cua_sess_a1b2c3d4",
  "status": "active",
  "portal": "TED eProcurement",
  "taskDescription": "Search for procurement notices matching CPV 72000000 in Finland",
  "actions": [
    {
      "step": 0,
      "type": "navigate",
      "target": "https://ted.europa.eu/en/search",
      "status": "completed",
      "timestamp": "2026-03-10T14:00:01Z",
      "screenshot": "screenshots/step-0.png"
    },
    {
      "step": 1,
      "type": "type",
      "target": "input#search-query",
      "value": "CPV 72000000",
      "coordinates": { "x": 540, "y": 312 },
      "status": "completed",
      "timestamp": "2026-03-10T14:00:03Z",
      "screenshot": "screenshots/step-1.png"
    },
    {
      "step": 2,
      "type": "click",
      "target": "Country filter: Finland",
      "coordinates": { "x": 180, "y": 480 },
      "status": "pending",
      "timestamp": "2026-03-10T14:00:05Z"
    }
  ],
  "currentStep": 2,
  "createdAt": "2026-03-10T14:00:00Z",
  "updatedAt": "2026-03-10T14:00:05Z"
}

Pause and resume

Sessions can be paused at any time. When paused, the CUA stops proposing new actions but retains full state. Resuming continues from the current step. This is useful when the user needs to inspect intermediate results or perform manual steps.

Audit trail

Every CUA action is logged with full provenance. The audit trail is immutable — entries cannot be modified or deleted after creation.

Audit record fields

Audit records are stored in EU jurisdiction and retained per GDPR data retention policies. Screenshots are stored alongside action metadata for complete session reconstruction.

Rollback capability

The CUA supports rollback to any previously completed step within a session. When a rollback is triggered:

1. All actions after the target step are marked as rolled_back
2. The browser navigates back to the state captured in the target step’s screenshot
3. The session’s currentStep is reset to the target step
4. The CUA loop resumes from that point, proposing new actions based on the restored state

Rollback is non-destructive: rolled-back actions remain in the audit trail with their original timestamps and screenshots. The audit record shows that a rollback occurred, when it was triggered, and which step was the target.

Rollback limitations

• Submitted forms cannot be un-submitted. If a form was submitted (M2 approved) and the user rolls back past that point, the CUA will flag that the portal state may be inconsistent.
• External state changes persist. Rollback restores the CUA’s internal session state but cannot undo changes already made on external portals.

Support

Technical: support@pauhu.eu

Sales: sales@pauhu.eu