Skip to content

Runs on QNX 8.0  ·  a real stop, an injected freeze, replayed at 1×

The car stopped.The speedometer didn't.Telltale noticed.

Monitor off

48frozen

wrong for 8.6 s

still counting · replay clock

Telltale armed

0verifiedDISPLAY FAULT · SHOWING TRUE SPEED

true again after 53.2 ms

measured · TT-FRZ-0001

QEMU (TCG) timings on an x86_64 guest, not hardware. One real stop, one injected freeze; the left glass is redrawn.

A frozen instrument cluster looks healthy: a live process, crisp digits, the wrong speed. Telltale is a separate QNX 8.0 process that reads back the pixels the driver sees, every 20 ms, and puts the truth back on the glass. On this stop it was true again 53.2 ms after the last true frame. Across 200 injected faults, it missed 0.

C · QNX Screen · resource managers · secpol · Apache-2.0

Speed over time on a real stop: the true speed falls from 49.01 to 0 km/h; with the monitor off the glass stays at 48; with Telltale armed the glass is caught 20.3 ms after the last true frame and the true value is back on the glass 53.2 ms after it.0204060km/hcaught 20.3 ms · truth back on glass 53.2 msmonitor off: the glass stays 48truth: 0 km/h
t = 21.0 s1×
Speed over time on a real stop: the true speed falls from 49.01 to 0 km/h; with the monitor off the glass stays at 48; with Telltale armed the glass is caught 20.3 ms after the last true frame and the true value is back on the glass 53.2 ms after it.0204060km/h
One real stop from the comma2k19 dataset: 49.01 → 0 km/h in 4.8 s, with a freeze injected. Right: incident TT-FRZ-0001, recorded on this stop at build 985ab29. Left: the same freeze with the monitor off, redrawn; the ablation (40 runs on the bench drive) found the glass wrong a median 29.9 s of 30. QEMU (TCG), not hardware.

Inside the green tick · ×20 slower

At 1× it is a hairline. Slowed down, it is a procedure.

Cover the lie with the truth first, then repair. Each mark is a timestamp from incident TT-FRZ-0001, counted from the last frame verified true.

incidents.csv · TICK_STALL, cause hung · shown 48, truth 46 km/h · QEMU TCG. Counted from the last true frame; from the fault command this incident was detected at 165 ms (the bench FREEZE p95, from injection, is 129.8 ms).

glass unverifiedsafe layer: true speed + bannerhanded backverified
  1. 0 mslast frame verified true
  2. 20.3 mscaught: TICK_STALL, renderer hung
  3. 53.2 mstrue value read back on the glass
  4. 58.7 msSIGKILL to the hung renderer
  5. 296.0 msnew renderer registers
  6. 319.7 msits first verified frame: hand-back
200/200

injected faults caught, 0 missed: 4 classes × 50 on the QNX guest

evidence/bench-faults.csv
0

false alarms in 91,029 checks over 30 minutes of a real drive

evidence/bench-clean.csv
29.9 s

of 30, the glass stays wrong with the monitor off (median, n = 40)

evidence/bench-ablation.csv
163/0

on-target test cases passed / failed, incl. 22 secpol checks

evidence/test-target/…/summary.txt

What the monitor saw

Read back from the glass, never asked of the renderer.

The cluster's own pixels, read from the composed display with screen_read_display on the QNX guest. This run used the engineering view, which labels each decision.

Read-back of the QNX cluster during the hold: digits 42 and an amber engineering banner, DISPLAY FAULT CAUGHT, VALUE_MISMATCH, 19 ms. 123
+466 ms after the catch · during the hold · incident TT-VAL-0001 · engineering run t-debug at 985ab29: the fault was healed on command, the renderer was not killed · 1280 × 768
  1. 1speed digits as composed, decoded against the D-DIN glyph atlas
  2. 2the lamp row: 6 lamps checked against the vehicle state
  3. 3drawn by Telltale, not the renderer: banner during the hold, incident card after
telltale.log · same runQEMU TCG
DETECT TT-VAL-0001 VALUE_MISMATCH shown=48 truth=47 lamps_shown=0x01 lamps_truth=0x00 caught_ms=19.1
NORMAL -> SAFE-HOLD
SAFE verified on glass TT-VAL-0001 (21.8 ms after detect, composed check)
CARD shown TT-VAL-0001 at 868,144 404x188 rows="INCIDENT TT-VAL-0001|VALUE_MISMATCH|shown 48 · truth 47 km/h|caught 19 ms · restored 1757 ms|handed back, no restart|service: cat /dev/telltale"
SAFE-HOLD -> NORMAL

Two processes look at one screen. Only one of them can lie.

vehfeed sets the vehicle state in vehicled, the /dev/vehicle resource manager. The cluster renderer reads it and posts its window to QNX Screen. telltale reads the vehicle state and reads the composed display back from Screen; on a lie it posts a safe layer on top, kills the renderer, which slm or a shell loop respawns, and verifies the new renderer before hand-back. vehfeedreal comma2k19 speed vehicledresource manager /dev/vehicle clusterrenderer_tdraw prio 20 QNX Screencomposes what the driver sees telltalemonitor_t · SCHED_FIFO 60read back → decode → compare→ safe layer, kill, verify devctl SET ionotify pulse + devctl GET devctl GET_SINCE, every check screen_post_window screen_read_display safe layer, z-top SIGKILL if hung, wrong or blank → respawned by slm or a shell loop
  1. vehfeed → vehicled /dev/vehicle: devctl SET
  2. vehicled → cluster: ionotify pulse + devctl GET
  3. cluster → QNX Screen: screen_post_window (the path that can lie)
  4. QNX Screen → telltale: screen_read_display
  5. vehicled → telltale: devctl GET_SINCE, every check
  6. telltale → QNX Screen: safe layer, z-top
  7. telltale → cluster: SIGKILL if hung, wrong or blank; slm or a shell loop respawns it

It reads the glass, not the renderer's word

Every 20 ms the monitor reads back the digit, lamp and tick regions and decodes them with the glyph atlas the cluster draws with. In the BLANK and MISDRAW runs a renderer self-report would have passed 60 of 60; the read-back caught all 60.

Crop of a real read-back: the digits 42 on the cluster glass during a hold.
the digit region of a real read-back (run t-debug, during the hold)
screen_read_display

The truth is a pathname

vehicled serves the vehicle state at /dev/vehicle. The cluster is woken by an ionotify pulse; the monitor asks for everything since its last check.

resource manager · devctl

Its own type

Feeder, vehicled, renderer and monitor each run in their own secpol type; 22 of 22 policy checks pass on the target, 13 of them denials. The bench ran without the policy loaded.

secpol

Hung or starved?

The monitor runs at FIFO 60, the renderer's draw thread at 20. On a stall it reads /proc to tell a hung renderer from a starved one, and leaves a starved one running while its window is readable: 0 of 50 STARVE runs were restarted.

SCHED_FIFO · /proc

Cover first, then repair

A safe layer with the true values goes on top, then SIGKILL, respawn, and a verified frame before the glass is handed back.

screen_post_window · slm

No machine learning, on purpose: the check is a deterministic glyph decode, so the same frame always gets the same verdict.

Six ways for a cluster to lie. 260 tries. None missed.

Every detection, one dot each

Milliseconds from the injected fault to the monitor's verdict, with each class's p95.

Detection time of every injected fault, by class, with each class p95: BLANK 24.8, CRASH 31.1, MISDRAW 55.0, FREEZE 129.8, STARVE 175.8, STALE 271.2 ms; 0 missed.050100150200250300ms from fault to detection · QEMU TCG100 ms stretch197.3 ms boundBLANKn = 50p95 24.8CRASHn = 50p95 31.1MISDRAWn = 10p95 55.0FREEZEn = 50p95 129.8STARVEn = 50p95 175.8STALEn = 50p95 271.2Detection time of every injected fault, by class, with each class p95: BLANK 24.8, CRASH 31.1, MISDRAW 55.0, FREEZE 129.8, STARVE 175.8, STALE 271.2 ms; 0 missed.0100200300ms from fault to detection · QEMU TCG100 ms197.3 msBLANKn = 50p95 24.8CRASHn = 50p95 31.1MISDRAWn = 10p95 55.0FREEZEn = 50p95 129.8STARVEn = 50p95 175.8STALEn = 50p95 271.2evidence/bench-faults.csv · bench-crash.csv · bench-misdraw.csv

Switch it off and the lie stays

The four fault classes again, 10 runs each, with the monitor disarmed: the glass was wrong a median 29.9 s of a 30 s window.

Monitor off: seconds the glass was wrong in each of the 40 runs of a 30-second window; median 29.9 s, every run still wrong when the window closed.0102030 s40 runs, monitor disarmed, sortedseconds wrong of a 30 s window · QEMU TCG
40/40runs still wrong when the 30 s window closed
evidence/bench-ablation.csv

200 injections in the four B1 classes plus 50 CRASH and 10 MISDRAW from separate runs. Not every target held: among the B1 classes the 197.3 ms formula bound held for BLANK, FREEZE and STARVE, the under-100 ms stretch only for BLANK, and STALE missed its 200 ms target (p95 271.2 ms). The fault and ablation CSVs name the build of every row: faults at 7592e40, CRASH at 5edab0a, MISDRAW at f15744a, the ablation at d3d5892; a 25-run regression after the code review (7a4367f) detected 25 of 25. The 163/0 on-target suite and the incident above are at 985ab29.

The 31-minute real drive drawn like a tachograph disc; the clean run over its first 30 minutes made 91,029 checks with 0 false alarms.051015202530
0false alarms

The 31-minute bench drive, drawn the way a tachograph disc records a shift: minutes around the rim, speed outward (rings at 0, 40, 80, 120 km/h). The clean run covered its first 30 minutes.

The quiet half of the job

Half an hour of real driving. Not one false alarm.

A monitor that cries wolf gets switched off. Armed for 30 minutes of a real drive, Telltale checked the glass 91,029 times and raised 0 false alarms.

evidence/bench-clean.csv · read-back cost p95 6.28 ms per check · QEMU TCG

Honest by design

Three of the README's limits, word for word.

  1. All timings are QEMU (TCG) on a loaded Mac, not hardware; QEMU ran without a visible display, so the glass is read back and captured to images by ttshot.
    README · Honest limits
  2. The monitor is not supervised: nothing restarts telltale if it dies.
    README · Honest limits
  3. The 200 injections are a declared population in one real 31-minute drive, not general coverage.
    README · Honest limits · full list in ENGINEERING.md

Codelab

Learn the primitive it stands on.

Your first QNX 8.0 resource manager on an Apple-Silicon Mac: a counter at a pathname with read, write, devctl and ionotify pulses. Every command block except the QEMU boot shows the output recorded on the guest.

excerpts · codelab/srcC · QNX 8.0
/* counter.h */
#define COUNTER_DEVPATH "/dev/counter"
/* counter.c: the server owns the pathname */
resmgr_attach(dpp, &rattr, COUNTER_DEVPATH, _FTYPE_ANY, 0,
              &connect_funcs, &io_funcs, &attr)
/* counterctl.c: a client is plain POSIX + devctl() */
int fd = open(COUNTER_DEVPATH, O_RDWR);
devctl(fd, DCMD_COUNTER_GET, &v, sizeof v, NULL)

Questions a QNX engineer would ask.

Does this really run on QNX?

Yes. C built with qcc from QNX SDP 8.0, running on the QNX Everywhere Quick Start image: an x86_64 guest under QEMU (TCG) on an Apple-Silicon Mac. No hardware was used, so every timing here is QEMU timing.

Why read pixels instead of trusting the renderer?

Because the renderer is the part that fails. In the BLANK and MISDRAW runs, a self-report of "intended vs truth" would have passed 50 of 50 and 10 of 10; reading back the composed display caught all 60.

What does QNX give it that a plain process would not?

A monitor is only worth something if a broken renderer cannot take it down. Telltale is its own process in its own secpol type at FIFO 60; it reads the truth from a resource manager and the glass from Screen, and restarts the renderer while everything else keeps running. The policy is exercised by the on-target suite; the bench and the incident above ran without it.

What is real, and what is replayed or injected?

Real: the QNX target, Screen composition, read-back, decode and recovery. Replayed: real CAN speed from comma2k19, fed into /dev/vehicle. Injected: the faults, by the project's own injector. Not verified yet: the gear indicator.

Is there AI in it?

No, on purpose. The check is a deterministic glyph decode: the same frame always gets the same verdict.

How do I reproduce the numbers?

With the SDP installed: make target, boot the guest with scripts/run_qemu.sh, then make test-target and scripts/bench_all.sh run. scripts/bench_all.sh report regenerates every table from the committed CSVs; see DEMO.md.