nfsmw-online/docs/BETA_TELEMETRY_PLAN.md

Closed beta: diagnostics, crash reports and tester workflow

Plan for shipping a preview build to a small group of testers and getting back reports we can actually act on. Written 2026-09-21, the day the first playable build appeared.

Decisions already taken (owner's call):

• Delivery: a "send report" button using the system share sheet. No backend, no background upload. The tester sees the file and sends it. This keeps us out of collecting data from other people's devices, which would otherwise need consent handling, storage and a retention policy.
• Scope: crashes and bug reports, performance counters, unhonoured shim contracts, device model and OS version.

Why this needs building at all

Everything the engine currently reports goes to __android_log_write and nowhere else (util.cpp's Log()). That fails for beta in three separate ways, each of which already bit us during development:

1. logcat wraps. Twice on 2026-09-21 a measurement was lost to it - once the whole startup sequence, once the frame-counter history, which made a frame-rate figure look far better than it was.
2. Testers cannot retrieve logcat. It needs a cable and a developer setup.
3. A hang produces no log at all. The black-screen bug that day left the process alive and silent; the only evidence was three lines that had already scrolled past.

Part 1 - what the build must produce

1.1 In-memory log ring (the foundation)

Log() keeps writing to logcat, and additionally appends into a fixed-size in-memory ring buffer (suggest 2 MB, tunable). Nothing is written to disk during normal play - the log rate is high (per-frame GLES sampling alone produces thousands of lines) and per-line file I/O would show up in the frame time we just spent the day reducing.

The ring is allocated once at startup. It must be usable from a signal handler, so: a plain pre-allocated byte array, no malloc, no locks that a crashed thread might hold. A per-record sequence number plus a short spinlock-free write is enough; a torn last record in a crash dump is acceptable.

1.2 Native crash handler

A handler for SIGSEGV, SIGBUS, SIGABRT, SIGILL, SIGFPE that writes a report file and then chains to the previous handler so the normal tombstone still happens.

Async-signal-safety is not optional here and we have a recorded incident: calling mmap() inside a SIGSEGV handler has deadlocked on bionic in this project before. The handler must use only a pre-allocated buffer, a pre-opened-or-open()ed fd, and write(). No malloc, no stdio, no Log(), no C++ allocation, no locks.

What to capture - this is the part specific to a translation layer, and it is what made today's crash solvable in one reproduction:

• The decoded guest address. With the flat mapping, a host fault address is flat_map_base + guest_address. On 2026-09-21 the fault at 0x6f464c459b decoded to guest 0x464c459b by subtracting x28. The handler should do that subtraction itself and print the guest address, since nobody reading a report will do it by hand.
• Which arena the guest address belongs to. guest_engine.cpp already has the classifier ("thread-stacks arena" and friends) - reuse it. "Fault in the heap arena" and "fault 1.2 GB past the end of everything" are completely different bugs and the report should say which.
• Full guest register set, and the host registers from ucontext.
• The last N KB of the log ring.
• Build stamp and device identity (below).

1.3 Hang detector

The black-screen bug was a hang, not a crash - no signal, no tombstone, process alive at 2% CPU. A crash handler would have caught nothing.

A watchdog thread checks the onDrawFrame counter. If it has not advanced for ~10 seconds while the activity is resumed, it writes the same report the crash handler would, tagged HANG, including a snapshot of every thread's state and stack pointer. It should fire once per hang, not repeatedly.

1.4 Unhonoured-contract registry

Today's crash was found because a shim logged that it could not honour a request, one line above the fault. That should be a first-class, structured record rather than a log line we grep for.

A small registry: ReportUnhonouredContract(area, detail), deduplicated by string, counting occurrences. rtti_shims.cpp's use_facet, jni_shim.cpp's silent zero-returns (task #51) and every other "returning NULL because we do not implement this" path calls it. The report carries the full deduplicated list.

This turns beta into a gap-discovery mechanism: the union of these lists across testers is a prioritised work queue for what the game actually needs, discovered from real play instead of guessed.

1.5 Session counters

Sampled once a second into a compact rolling summary, not one line per sample:

• frames per second - min, median, 10th percentile, and where the low ones happened (which is what the median alone hides, as it did today)
• time from launch to first frame, and each level-load duration
• CPU time consumed by the process, and by the GLThread specifically
• peak CPU temperature
• guest heap: live, peak, arena exhaustion events
• thread-stack arena: peak in use, exhaustion events (the black-screen cause - this must never again be discovered by reading a log tail)

1.6 Build and device identity

Every report starts with: build stamp (version plus a short git hash or build timestamp baked in at compile time), device model, SoC, Android version, ABI, available RAM, and screen size.

The build stamp matters more than it sounds. Twice on 2026-09-21 the wrong APK was nearly measured - once a three-week-old release build installed by mistake. With testers there is no chance to check by hand; the report must say which build produced it.

1.7 The share button

A screen reachable from the pause menu: "Report a problem". It bundles the newest reports plus the current log ring into a single zip in the app's own files directory and hands it to ACTION_SEND.

Before sharing it shows the tester a short plain-language summary of what is in the file - log lines, device model, no personal data, no game account details. They are sending it themselves; they should know what it is.

If the previous session ended in a crash or hang, offer to send that report on the next launch, since the tester will not go looking for it.

Part 2 - the tester-facing report form

Free-text bug reports from testers are usually unusable not because testers are careless but because nobody told them which three facts matter. Keep it short - a long form gets skipped.

In-app, attached automatically: build stamp, device, the log bundle. The tester never types any of this.

What we ask the tester for, in this order:

1. What were you doing? One line. "Entered a race from the city map."
2. What happened? One line. "Black screen, music kept playing."
3. What did you expect? Only when it is not obvious.
4. Can you make it happen again? Every time / sometimes / happened once. This single question decides whether we can chase it at all.
5. Did you play for a while before it happened? Yes/no. Specifically included because the whole class of resource-exhaustion bugs - the thread-stack arena, the guest heap - only shows up after a long session, and testers do not think to mention it.

Severity, defined by consequence rather than by feeling, so it is not argued about:

• Blocker - cannot continue playing; progress lost.
• Major - a feature does not work, but the session survives.
• Minor - visual or audio defect, gameplay unaffected.

Ask them explicitly to send the report even when the game recovers. A hang that resolved itself still wrote a HANG report, and that is often the easier one to diagnose.

Part 3 - what we do with reports

Triage order, informed by what has actually been expensive to find:

1. Unhonoured-contract list first, before reading the crash. Today the answer was in that list. It is cheap to check and frequently decisive.
2. Decoded guest address and its arena. Distinguishes a wild pointer from arena exhaustion from a real logic bug, without any further work.
3. Exhaustion counters. If a thread-stack or heap arena hit its ceiling, the crash is a symptom and the ceiling is the bug.
4. Only then the register dump and the log tail.

Group reports by build stamp before comparing anything. Mixing builds is how a fixed bug looks like it is still present.

Suggested order of work

Each step is independently useful, so the beta does not wait on the whole set.

1. Log ring + report file + share button, with device and build identity. Minimum shippable - a tester can send something useful.
2. Crash handler with guest-address decoding.
3. Unhonoured-contract registry, with use_facet and the JNI zero-returns as the first callers.
4. Hang detector.
5. Session counters.

Deliberately out of scope

• No backend, no automatic upload. Chosen above. Revisit only if the manual path proves too lossy in practice.
• No unique device or user identifier. Grouping by build stamp and device model is enough at this scale and avoids tracking individuals.
• No gameplay telemetry - what cars, which races, how long played. It is not needed to fix defects, and collecting it would change what this file is.

---

Status 2026-09-21: the crash handler is built (plan section 1.2 + 1.7 partial)

Implemented and verified end to end on the Pixel 6a.

Native (mpcore/src/main/cpp/crash_handler.cpp). Hooks SIGSEGV, SIGBUS, SIGABRT, SIGILL, SIGFPE with SA_SIGINFO | SA_ONSTACK, on a pre-allocated alternate stack so a stack-overflow crash is still reportable. Everything the handler needs - the output path, the build stamp - is built at install time; inside the handler only open/write/close and hand-written integer formatters run. No malloc, no snprintf, no JNI. It chains to the previous handler afterwards, so Android still writes its own tombstone.

The report decodes the fault address: a host address inside the guest window is also printed as the guest address, and flagged when it is past the end of the mapped region ("a wild pointer, not a real guest object"). That is the number worth reading, and nobody will subtract the base by hand from a tester's report.

Java. CrashReportActivity renames the pending report (the handler writes a fixed name, since it cannot safely format a timestamp), zips it with device and build details, shows it, and offers ACTION_SEND through a FileProvider scoped to the crash directory only. Reports live in Android/data/<pkg>/files/crashes - reachable over USB with no permission.

Verified: handler installs; kill -11 produces a report with the right signal, registers and a correct "outside the guest window" verdict; the next launch detects it, renames it, builds the zip, and CrashReportActivity becomes the resumed activity. Files land where intended.

Not verified: what the screen actually looks like. The test device locked itself, so every screenshot was of a sleeping or locked display - which is also why an early "black screen" reading was wrong and led to one unnecessary fix (explicit colours, harmless and kept). The layout needs a human to unlock the phone and look.

Three failures on the way, all worth keeping

1. Installed too early. The call sat at the top of onCreate, but libmpcore.so is only loaded later by loadCore() - UnsatisfiedLinkError, caught and logged. Moved to immediately after loadCore().
2. Missing extern "C". The JNI function was C++-mangled (_Z65Java_...), so the JVM could not find it. The symptom was identical to the load-order bug above, which cost a wrong fix before the symbol table was actually read.
3. Blocked activity start. Checking for a pending report in GameActivityMain.onCreate - which starts the report screen and finishes itself - was refused by the platform (BAL_ALLOW_GRACE_PERIOD) and dumped the tester on the home screen. The check belongs in PermissionsActivity, the visible launcher entry.

---

Status 2026-09-22: game data ships inside the APK

A tester now installs one file and plays. No separate .obb download, no file manager, no instructions about where to put anything.

How. The ~595 MB archive ships as assets/game_data.obb, and androidResources { noCompress += "obb" } keeps it stored rather than deflated - it is already compressed, so re-compressing would cost build and install time for nothing. On first launch GameDataUnpackActivity copies it to getObbDir()/main.<versionCode>.<package>.obb, which is exactly the path GameActivityMain.obbFullPath already builds, so no other code knows this happened.

The copy writes to a .part file and renames only on success. A half-written archive that merely exists would pass a naive check and send the game off to read truncated data - failing far from the cause, which is the failure mode this project keeps paying for. Free space is checked before starting rather than 500 MB in.

Measured on the Pixel 6a, with the real OBB renamed aside to simulate a clean device:

APK size	615 MB (was 22 MB)
build time	16 s - aapt2 handles the stored asset without trouble
adb install	31 s
unpack	under 8 s - it finished before the first progress poll
result	md5 identical to the original OBB
game afterwards	2,893 frames, 0 faults, mAssetLocationType=OBB

Costs worth stating. The device needs the APK plus the unpacked copy at once: about 1.2 GB free at install time, ~600 MB after. And the data exists twice on disk permanently, since Android keeps the APK.

The alternative not taken. Because the asset is stored uncompressed, its bytes sit contiguously in the APK, so the engine's own Shim_open/Shim_read could serve the OBB path straight out of the APK at an offset - no copy, no duplication. That is a real option if the 600 MB ever matters, but it adds a new failure surface in file I/O right before a beta, and the ask here was explicitly for self-extraction.