AutonomousScreenAgent: a self-hosted agent that sees, acts, and verifies, on homelab hardware.

A self-hosted peer to the big computer-use agents — where the differentiator is trust.

The frontier labs have shown that a model can drive a screen. The open question is not whether it can click, but whether you can trust it to act on a machine you care about. That is an engineering problem — verification, safety boundaries, and etiquette on a live system — and it is the part this project is built around.

One loop, two muscles, a ratified safety spine.

The agent runs a closed sensorimotor loop — perceive → plan → act → verify → heal — over eyes and hands that are deliberately separate subsystems. The safety rules below are not aspirations; they were argued through my own multi-model Council and are recorded, with their ratification jobs, in the Forge dossier.

• Independent verification channel — the eye that verifies is never the hand that acted; you never grade your own homework.
• Verifier output is never a training label — a signal used to check work is not fed back as ground truth to learn from.
• A verifier hit is never terminal for an irreversible action — nothing destructive rides on a single automated “looks done”.
• No blind retry for irreversibles; ack-before-commit, idempotency and de-dupe on the hands lane.
• VNC-RFB is barred from irreversible actions at the muscle level — the boundary is structural, not a runtime check.
• Multi-scale NCC targeting plus a deterministic staged verification cascade (actuation-ack → pre-state → localized delta → expected-post).

One recorded loop, timed honestly.

Run 20260921_001552, host Miner360 (192.168.5.6:5900), 1920×1080. Recorded live with script so it can be replayed at any speed. Every phase below is measured with perf_counter; nothing is rounded into a nicer story.

Side measurements from the same run: hand connect 4.49 ms; independent VLM readback 1298.74 ms. On the JPEG operational path the agent's own work is roughly ~19 ms (an SVA JPEG capture measures ~8.5 ms); the 92.61 ms above is the run's own lossless-PNG path.

target: miner360 (192.168.5.6:5900)
probe : 'HANDFORGE closed-loop reproduction 20260921-001552 - SVA eyes and asyncvnc hands - safe to discard'  (98 chars)
[hand ] asyncvnc connected  fb=1920x1080  (4 ms)
[percv] SVA capture 45.8 ms (wall 66, frame_age 3.30s) bright=388  -> ..._1_before.png
[act  ] inject(focus+newtab+insert+type 98) = 1.05 ms  | deliberate UI-settle waits = 2903 ms
[vrfy ] probe visible after 1538 ms (5 polls)  bright 388->667 (thr 600)  confirmed=True  -> ..._2_after.png
[vlm  ] independent readback 1299 ms  token_match=True
         -> "HANDFORGE closed-loop reproduction 20260921-001552 - SVA eyes and asyncvc hands - sa"
[disc ] :q! sent  bright 667->388 (probe gone)  tab-bar Δ(before->restored)=0.020  clean_restore=True
=== MEASURED ===  machine_work_ms = 92.61   end_to_end_wall_ms = 4533.98   observe_latency_ms = 1538.19
Script done on 2026-09-21 00:16:01+03:00 [COMMAND_EXIT_CODE="0"]

Raw recording for scriptreplay: ./typescript  ·  ./timing

A hallucination cannot reproduce that timestamp.

The probe carries a unique run timestamp, 20260921-001552, generated at launch. Three independent readings all agree on that exact string — so the probe genuinely reached the screen; no single reader is being taken on faith.

• Deterministic content check. A calibrated bright-pixel count in the probe-line region: baseline ~388 rose to ~667 with the probe present (threshold 600), and fell back to ~388 after :q! removed it. Calibration scale: empty-editor placeholder ~177, VS Code Release-Notes ~388, a 98-char probe ~1300.
• Local 8B VLM readback (~1.3 s) — read the region back and quoted the line; one minor character misread (“asyncvc”), timestamp exact.
• Cloud model readback (authoritative) — quoted the line, timestamp exact.

An earlier verifier was confidently wrong — and a strong reader caught it.

This is the part I would most want a reviewer to see, because it is where the engineering actually happened. The first version of this harness verified the action with a naive pixel-delta — “did the editor region change?” It reported success before and independently of whether the probe had landed. Here is exactly why it lied, and exactly how it was fixed.

• Pixel-motion is not the expected content. Opening a new tab makes the editor region change on its own — VS Code paints its empty-editor “Generate code” placeholder — so a bare delta fires whether or not the probe was typed. It cannot tell the probe from the ghost text. A strong VLM reading the saved frame is what exposed the false pass.
• Frame staleness. TightVNC's mirror driver emits framebuffer updates only on change, so a served frame can be seconds old on a near-static screen. Measured here, the probe takes ~1.25–1.7 s to render in VS Code and propagate into a served frame — so a fixed short settle reads a pre-render frame and “confirms” nothing.

• Confirm expected content, not motion: a calibrated bright-pixel count that separates real typed text from the faint placeholder, cross-checked by a VLM that must read the probe's unique token back.
• Poll until genuinely visible instead of sleeping a fixed guess — which, as a side effect, measures the real observation latency (the 1.54 s above) honestly, per run, rather than hiding it.

Three pieces, verbatim.

Lifted unchanged from the run's harness (milestone_loop_harness.py, v3) — the hand, the eyes, and the verify. Minimal and readable on purpose.

# ACT (asyncvnc hand): inject cost separated from deliberate UI-settle waits
inj = 0.0
t_seq = time.perf_counter()
for keys, wait in ((("Ctrl", "1"), 0.60), (("Ctrl", "n"), 1.50), (("i",), 0.80)):
    t = time.perf_counter(); hand.keyboard.press(*keys); inj += time.perf_counter() - t
    await asyncio.sleep(wait)
t = time.perf_counter(); hand.keyboard.write(probe); inj += time.perf_counter() - t
act_seq = (time.perf_counter() - t_seq) * 1000.0
t_ms["act_inject_ms"] = inj * 1000.0
t_ms["act_ui_settle_ms"] = act_seq - inj * 1000.0
t_ms["act_sequence_ms"] = act_seq

def sva_capture(host):
    """SVA EYES: grab a frame from the warm persistent VNC session.
    -> (png_bytes, capture_ms, wall_ms, fb, frame_age_s)."""
    t0 = time.perf_counter()
    d = _post("/api/vnc/capture", {"host": host, "image_format": "png", "return_b64": True}, 20)
    wall = (time.perf_counter() - t0) * 1000.0
    png = base64.b64decode(d["jpeg_b64"])
    age = float((d.get("meta") or {}).get("session_last_frame_age_s") or 0.0)
    return png, float(d.get("capture_ms") or 0.0), wall, tuple(d.get("framebuffer_size") or []), age

def bright_count(png, box):                      # BRIGHT=190, PROBE_VISIBLE=600
    g = np.asarray(Image.open(io.BytesIO(png)).convert("L"))
    x1, y1, x2, y2 = box                          # ROI = (358, 88, 1260, 112)
    return int((g[y1:y2, x1:x2] > BRIGHT).sum())

# VERIFY: poll until the probe is genuinely visible (and MEASURE observe latency)
t0 = time.perf_counter(); after_png = before_png; bright_after = 0; polls = 0
while (time.perf_counter() - t0) < OBSERVE_TIMEOUT:
    after_png, cap_after, _, _, age_after = sva_capture(HOST); polls += 1
    bright_after = bright_count(after_png, ROI)
    if bright_after > PROBE_VISIBLE:
        break
    await asyncio.sleep(0.30)
t_ms["observe_latency_ms"] = (time.perf_counter() - t0) * 1000.0
confirmed = bright_after > PROBE_VISIBLE

It works alongside a person, and leaves no trace.

The operating mentality is to learn the environment and work alongside it non-destructively — never halt in fear, never trample the operator's state. In this run the agent opened a scratch Untitled tab, typed the probe, and discarded it with Vim :q!. The operator's Welcome / VS Code release-notes tabs, live PowerShell, and Vim buffer were untouched.

What this is not — stated plainly.

The strongest version of this work is the one that is honest about its edges. None of the following is hidden.

• VNC-RFB is a blind muscle. No application-level acknowledgement, so it is ineligible for irreversible actions (submit, send, pay, delete) and cannot reach the Windows UAC secure desktop. Reversible and normal-desktop only.
• The act waits are conservative, not optimised. The ~2.9 s of inter-keystroke settle is tuned for reliability while proving the loop, not for speed. It is tunable.
• The observation latency is a property of this transport. The ~1.5 s is how long TightVNC's mirror driver plus the capture channel take to propagate a change. A native / CDP / UIA channel would cut it drastically — that path is design-approved but gated on a demonstration, and is the next build.
• Irreversible actions are not yet reached. No real trade order, send, or payment has been actuated. That deliberately waits behind the native, ack-carrying path and the full safety gate.
• One clean run is shown here. The technique was exercised repeatedly this session; this page documents a single fully-instrumented, recorded instance rather than a statistical claim.

How this was engineered.

AutonomousScreenAgent did not appear from nowhere. It is the actuation stage of a vision line that has been building for over a year, and it was engineered through the same dossier-and-Council process as the rest of the Forge.

Who built this.

Talk to me.