Measured on both hosts, with the method and the reason a median is the wrong
statistic for the Pixel 6a (bimodal 60/30 - the median sits in the upper lobe
and reports 59.8 where the mean is 39.5).
Also retires the old "51 fps" from task #61: it was not measured in a race
and is not comparable. That is the second time a figure from elsewhere in the
game was nearly quoted as in-race performance, so it is written down plainly.
The consequence is a priority change - the 6a misses the frame budget by a few
percent, so #63 and #64 are worth more than their profile shares suggest, and
the hot path has very little margin left to spend on new work.
Co-Authored-By: Claude <noreply@anthropic.com>
master was stale (predates both feature branches). native-arm32-trace-harness
is a strict superset of lan-event-injection-poc (already merged in 1a771ae)
plus the Unicorn/GuestFn emulation layer, and file-open-hook-poc /
track-hook-toggle-flag / arm64-translation-poc contribute nothing unique
beyond it. This merge makes master the actual trunk; going forward both
mod-hooks and ARM64-emulation work happens on feature branches merged
back via PR.
Brings opponent_substitution.h and cop_traffic_disable.h - the
native32/launcher branch's proven, live-confirmed multiplayer-mod hooks
(opponent car/color substitution, cop dispatcher removal with a live
before/after roadblock proof, full traffic elimination) - into this repo
as source, plus DebugFeatures.kt/DebugMenuOverlay.kt (debug-only money/menu
scaffold) and this session's GameActivityMain.kt thread-safety fix
(BroadcastReceiver.onReceive wrapping native calls in
gameGLSurfaceView.queueEvent{} instead of calling straight into JNI from
the main thread - a real cross-thread race, independently useful on both
branches).
Two real conflicts, both resolved to preserve architecture rather than
naively keep both sides:
- main.cpp: kept this branch's own structure. The new .h files, like
lan_event_injection.h before them, are NOT #include'd or wired into
JNI_OnLoad - every hook in both files resolves and calls raw
"libapp_base + OFFSET" function pointers, which on a real dlopen'd
libapp.so is a valid jump target but here is only Unicorn-backed guest
memory - including them unmodified would crash immediately, not just
misbehave. Left as source for the same GuestFn/CallGuestFunction porting
session already planned for lan_event_injection.h.
- GameActivityMain.kt: trivial, a code comment disagreeing on this build's
own package id (nfs13_arm here vs nfs13_mod on native32) - merged to
document both.
Verified after resolving: compileTranslatedDebugKotlin,
compileNative32DebugKotlin, and externalNativeBuildDebug (arm64-v8a +
x86_64, confirmed via a fresh main.cpp.o/mpcore.so rebuild, not a cache
hit) all succeed - this merge compiles clean on both Kotlin flavors and
the native side, not just resolves textually.
Co-Authored-By: Claude <noreply@anthropic.com>
ANALYSIS.md/ARCHITECTURE.md/PROGRESS.md and the rest of this project's
living docs have always lived one directory above this repo's root
(NFSMW_Online_Claude_workdir/*.md), so they were never actually part of
this git history despite being the authoritative record of every hook,
offset, and RE finding this branch's code is built on.
Mirrors the docs/ layout already used on native-arm32-trace-harness so
both branches reference the same file set by name, pending the actual
code merge (see PROGRESS.md's own "Repo merge pending" note).
Co-Authored-By: Claude <noreply@anthropic.com>
These files had never been tracked anywhere - they lived in a plain directory
with no git at all, which is also where the whole reverse-engineering record
sat. Code already committed refers to them by name (opponent_substitution.h
cites "ANALYSIS.md section 6hh", DebugMenuOverlay.kt cites "DEBUG_MENU.md
section 3"), so until now a fresh clone carried references to documents it did
not contain.
ANALYSIS.md the RE record, and the reason the rest works
ARCHITECTURE.md how the mod's pieces fit together
ARM64_TRANSLATION_LAYER.md the translation layer's running log
PROGRESS.md chronological progress across both chats
BETA_TELEMETRY_PLAN.md how crash/telemetry reporting is meant to work
LOBBY_UI_DESIGN.md + .html lobby design and its clickable prototype
DEBUG_MENU.md debug panel design
STATIC_RECOMPILATION_FALLBACK.md the plan if translation had not panned out
evidence/ font atlas capture from the glyph-corruption bug
save_backups/ saves at known milestones, for reproducing state
Co-Authored-By: Claude <noreply@anthropic.com>
Subtask 2.1 - opponent_substitution.h. Hooks
OpponentCollection::PopulateFromProperties (sub_2B649C) and overwrites
Opponent.CarDescriptionName/ColourIndex at map-load time so a real lobby
player's car and colour take an AI opponent's slot. Confirmed live: a Ford
Focus RS500 was forced into a slot that held a different car and colour
before the hook, on a real (non-synthetic) replay. The starting-grid
placement code walks a different vector entirely, so there is an untraced
intermediate spawn/resolve step - it was not needed, because whatever it is,
it reads these same fields. The header records that honestly rather than
claiming the whole chain is understood.
Subtask 2.4 - cop_traffic_disable.h. Removes police from multiplayer races.
This corrects an earlier hook that only covered sub_F7E9C: tracing callers
showed sub_F5EA4 is the per-tick dispatcher and branches on a live flag into
two schedulers, both reaching the same SpawnCopCar. Hooking one leaf left the
other unblocked, which is why cops were still appearing while the hook logged
nothing. Hooking the dispatcher covers both leaves and skips only the
spawn decision, leaving the CopManager's other per-tick bookkeeping alone.
Civilian traffic and AI opponents are untouched - both are load-bearing for
multiplayer.
Debug menu scaffold: DebugFeatures.kt is the single switch deciding whether
debug UI is built at all; DebugMenuOverlay.kt is the Compose panel. Its Apply
deliberately touches no game memory yet - no balance getter/setter has been
located (DEBUG_MENU.md section 3) - so it edits local state only rather than
pretending to work.
Stays on this feature branch, not master: toggle-flag and experimental work
does not belong on a release-ready branch.
Co-Authored-By: Claude <noreply@anthropic.com>
The run of work that took the translated build from "boots" to "playable".
Engine:
- Flat guest mapping replaces the software MMU on aarch64 hosts. A 4 GiB
PROT_NONE reservation lets a 32-bit guest address zero-extend safely, so
tcg_out_qemu_ld/st short-circuit before tcg_out_tlb_read and the prologue
materialises the base into X28. Measured 1.24x (51 vs 41 fps, interleaved
A/B). Note the base must be set BEFORE UC_INIT - Unicorn inits lazily, and
setting it after uc_open silently falls back to softmmu.
- num_get<char> facet implemented, which was the real cause of the crash after
the prologue race; a full race is now playable end to end.
- Thread-stack free list + ReleaseThreadEngine, fixing the arena leak that
showed up as a black screen when entering a race. kMaxGuestThreads 16 -> 64.
- Real ARM32 FMOD now runs in-engine via the Java FMODAudioDevice bridge, with
a per-thread JNIEnv. Two of the three blockers were our own single-image-era
guards.
Host/app:
- Native crash handler: async-signal-safe, decodes the host fault back to a
guest address, writes a report file and nothing else. CrashReportActivity
picks it up on the NEXT launch, zips it, and offers to share. No backend, no
automatic upload.
- Game data ships inside the APK and self-extracts on first launch, so a tester
installs one file and plays. Copy-to-.part-then-rename, with a free-space
check up front.
- EGL context preserved across pause, fixing black textures on resume.
- Navigation bar hidden and re-hidden on focus gain; volume keys reported as
system keys, checked before the loading-state gate.
- x86_64 added to abiFilters: the ARM32 guest runs under tcg/i386 with no
houdini in the path. The flat mapping is aarch64-only, so that host falls
back to the software MMU - commented at the abiFilters line.
Ignore rules added for app/translated/ (611 MB of signed release APK, which
also carries the bundled OBB) and ostream_repro/build/.
Co-Authored-By: Claude <noreply@anthropic.com>
notdirty_write called uc->memory_mapping() on every guest write, before
testing whether the region is executable, and used the result for that one
bit. A probe counted over a million such calls per 3 seconds with
exec_region=0 - the full find_memory_mapping -> address_space_translate ->
flatview_translate chain running and the answer then discarded.
The map is static once CreateConfiguredEngine has mapped its five spans, so
a 512-entry direct-mapped page -> MemoryRegion cache removes all but cold
misses.
interleaved A/B, engine start to first OnCarLoaded
cache on: 31.27 / 31.34 / 31.22 s mean 31.28
cache off: 32.27 / 32.12 / 32.44 s mean 32.28
3.1%, ranges fully separate, spreads under 0.35s. The profile confirms the
mechanism rather than just the outcome: flatview_translate 4.46% -> 3.17%,
find_memory_mapping 3.02% -> 2.36%.
THE FIRST VERSION OF THIS CACHE WAS SLOWER, and the measurement is why it
was caught. It used `static __thread`, which on Android resolves through
emulated TLS: __emutls_get_address appeared at 8.11% and pthread_getspecific
at 3.07% - together more than the 7.5% being removed. Storage now lives in
uc_struct, which is already per-thread here (one uc_engine per host thread),
so no TLS is involved. Do not reintroduce __thread on this path.
INVARIANT: the cache assumes mappings and permissions are fixed after engine
setup. Anyone changing them at runtime must clear mr_cache_ready.
Co-Authored-By: Claude <noreply@anthropic.com>
Task #53 ruled out TLB capacity as the cause of flatview_translate 4.46% +
find_memory_mapping 3.02%. A probe on notdirty_write's own first line found
the real reason, and it is a miss in this project's own earlier fix.
notdirty_write calls uc->memory_mapping() UNCONDITIONALLY, before testing
whether the region is executable - and that result is used only to test one
bit. Measured on device, per 3-second window:
hits=3478174 exec_region=1 (startup, all in the 0-1MB bucket)
hits=1184522 exec_region=0
hits=902501 exec_region=0 (10MB and 11MB buckets = .data/.bss)
Over a million calls per 3 seconds where the lookup ran in full and the
answer then did not matter. Task #54 removed the tb_invalidate work for
non-executable spans but left this lookup standing in front of it, so most of
the cost survived that fix.
The map is static once CreateConfiguredEngine has mapped its five spans, so
a small per-thread direct-mapped page -> MemoryRegion cache removes the
lookup for everything but a cold miss. That cache is implemented here but
DEFAULTS OFF (UC_MR_CACHE_ENABLED 0) because it could not be measured:
- the Pixel 6a stopped being measurable (mScreenState=OFF within seconds
despite a 30-minute timeout and svc power stayon true, with
NotificationShade holding focus - it needs physical attention)
- the Galaxy A9 cannot run this build at all: "failed to attach / start
timeout", the engine's startup work exceeds Android's attach deadline on
that hardware
Shipping it enabled would repeat exactly the mistake this project has been
correcting all session. The measurement that motivates it is solid; the fix
is not yet evidence.
INVARIANT if it is ever enabled: anyone changing mappings or permissions at
runtime must clear the cache, or stale permissions will be used. It is
per-thread, so no locking is involved.
Co-Authored-By: Claude <noreply@anthropic.com>
Two results, one positive and one negative.
CONFIRMED - removing the exit-request check from the guest load/store path
(ea079b8) is worth 6.3%. That commit went in on a single A/B pair because
the device wedged mid-measurement; re-run on a freshly rebooted Pixel 6a,
three interleaved pairs, load to first OnCarLoaded:
without the check: 33.42 / 33.10 / 33.32 s mean 33.28
with the check: 35.62 / 35.43 / 35.49 s mean 35.51
Ranges do not overlap and each spread is under a third of a second.
REFUTED - raising CPU_TLB_DYN_DEFAULT_BITS from 8 to 12. The premise checked
out: a probe in tlb_mmu_resize_locked showed it runs only about twice per
second and current_entries stayed at 256 for a whole run, so the adaptive
TLB genuinely never grows for this workload and sits at 1 MB of coverage
against a working set of hundreds of MB.
Raising it took effect (current_entries=4096, verified live) and bought
nothing:
12 bits: 32.36 / 32.26 s
8 bits: 32.86 / 32.32 s 0.28s apart, ranges overlap
The profile moved the WRONG way - tlb_set_page_with_attrs 0.86% -> 2.92%,
since a larger TLB costs more to fill and flush. So the softmmu lookup cost
(flatview_translate + find_memory_mapping, ~7.5%) is not driven by TLB
capacity. Reverted to the upstream default, with the refutation recorded in
place so nobody retries it without a new theory.
Also removes the temporary TLB probe.
Worth noting for future measurements: after the device was rebooted, the
same build measured ~33s where it had measured ~41s before. The earlier
drift was device state, not code.
Co-Authored-By: Claude <noreply@anthropic.com>
Unicorn calls check_exit_request() from tcg_gen_qemu_ld_i32/st_i32/ld_i64/
st_i64 - i.e. on EVERY GUEST MEMORY ACCESS, not per basic block. Upstream
QEMU does not do this. It exists so a uc_emu_stop() issued from a MEMORY hook
callback takes effect immediately instead of at the next block boundary.
Found by taking a profile contradiction seriously. In a clean in-game
profile, helper_check_exit_request_arm was the largest single symbol at
8.71%, against helper_uc_tracecode's 0.98% - yet both are emitted together
at the ARM translator's two hook sites and should have matched. Grepping
every emission site across the tree, rather than just target/arm, found the
six in tcg/tcg-op.c.
This engine never stops emulation from a memory callback: mem_fault_hook_cb
only logs and returns false, and the one accelerator that does call
uc_emu_stop (FnvHashAccelHookCb) is a UC_HOOK_CODE hook, where the check is
still emitted. So on the load/store path it is pure overhead.
helper_check_exit_request_arm 8.71% -> 4.07%
The remainder is the legitimate per-hook emission. Game still loads, reaches
OnCarLoaded, zero MEM FAULTs.
EVIDENCE LEVEL, stated plainly: one clean interleaved A/B pair, 33.13s
without the check vs 35.89s with it (7.7%), plus the profile share above.
That is weaker than this project's usual bar - the follow-up pairs were lost
when the Pixel's screen dozed off mid-run and the engine stopped rendering
(glClear=0, no faults), and later attempts to reset the save between runs
were blocked. The change should be re-A/B'd on a woken, freshly-booted
device before being treated as settled.
Kept as a named constant rather than deleted because this is a vendored
tree: anyone adding a memory hook that calls uc_emu_stop MUST set
kEmitExitCheckOnMemoryAccess back to 1, or that stop will be deferred to the
next block boundary.
Also turns off the task #56 instrumentation (kLogPerfMetrics,
kCountStubDispatches): both took a steady_clock reading per draw and per shim
crossing, which showed up as __kernel_clock_gettime at 3.67% of a profile -
the measurement was costing nearly as much as some of the things being
measured.
Co-Authored-By: Claude <noreply@anthropic.com>
Task #59's idea was that __dynamic_cast - 49% of all shim crossings, called
3x more often by native than this engine can serve (task #58) - could run as
emulated ARM32 instead, since it only reads guest memory and needs nothing
from the host. Emulated-but-not-crossing would then beat
native-but-crossing.
It is implemented, correct, and does remove the crossings. It also makes no
measurable difference to load time.
Interleaved A/B, one run per build, alternating to cancel device drift:
fast path ON 40.67 / 41.13 / 40.98 mean 40.93s
fast path OFF 41.35 / 40.52 / 41.46 mean 41.11s
0.18s apart with fully overlapping spreads. Noise.
THE RESULT THAT MATTERS IS THE NEGATIVE ONE: shim-crossing COUNT is not what
paces loading. Three separate findings now say this and were previously read
as three unrelated disappointments - removing 17 probe hooks bought nothing
(#48), cheapening the crossing bought ~3% (9c4a455), and removing ~41% of all
crossings buys nothing here. That closes a whole line of optimisation.
Kept behind kEnableGuestFastPath, default OFF: it is real added complexity
(hand-assembled guest Thumb, a second registration name, an executable-memory
allocator) for no measured gain, and a wrong dynamic_cast corrupts state
silently rather than crashing. The measurement and the infrastructure are
worth keeping; the behaviour change is not worth defaulting on.
Two things found along the way that ARE keepers:
- AllocGuestCode(): AllocPermanent carves from the control arena, which is
mapped read-write only since task #54, so code placed there faults
immediately with FETCH_PROT at its own entry address. Making the control
arena executable to accommodate it cost a measured ~4s. AllocGuestCode
carves from the trampoline arena, which is already executable and is not
write-hot. Anything generating guest code must use it.
- Device drift is large enough to invalidate cross-session comparisons: the
identical build that measured 35.20s earlier today measured 41.11s a few
hours later. Only interleaved A/B within one sitting is trustworthy. Three
near-miss wrong conclusions today traced back to comparing against a stale
baseline.
Co-Authored-By: Claude <noreply@anthropic.com>
Task #58 asked whether the engine induces the __dynamic_cast flood or the
game does. A native LD_PRELOAD interposer added to trace_agent answers it:
native ARM32 on the A9 calls it up to 824,200/sec, against this engine's
284,986/sec. The game is simply that RTTI-heavy, and native absorbs it
because a real call costs tens of nanoseconds. Our rate is not demand, it is
supply - the shim boundary is throttling the guest.
(Interposing __dynamic_cast collides with libc++abi.a, which the NDK links
statically into the agent; resolved with --allow-multiple-definition scoped
to that target, since our object precedes the archive and wins.)
That makes a guest-side implementation interesting: Shim_dynamic_cast is
already a pure guest-memory walk - it reads the object's vtable, vtable[-1]
(dynamic type) and vtable[-2] (offset-to-top), then compares type_info
records. Nothing comes from the host, so the same algorithm could run as
emulated ARM32 with NO boundary crossing at all.
Whether that is worth doing depends on how big it would have to be, so this
adds a temporary shape survey. Measured over a prologue load, 2.7M calls:
exact (depth 0) 84.2%
one base 0.4%
deeper 0.0% (max depth reached all run: 3)
not found 15.4%
84% of calls need NO hierarchy walk - the object's dynamic type already IS
the target. A guest fast path of roughly six instructions (load vtable,
compare vtable[-1] to the target, return ptr + vtable[-2]) with a fallback
to this shim would eliminate 84% of dynamic_cast crossings, which is ~41% of
ALL shim crossings given dynamic_cast is 49% of them.
Also confirms there is no implementation to reuse inside libapp.so: not only
__dynamic_cast but the type_info vtables themselves (__class_type_info,
__si_class_type_info, __vmi_class_type_info, __pointer_type_info) are all
UND - this engine supplies them from rtti_shims.cpp.
Survey left in the tree behind kSurveyDynamicCast, default off.
Co-Authored-By: Claude <noreply@anthropic.com>
Every shim crossing took a std::mutex in MiscStubDispatch and five separate
uc_reg_read calls in import_stub_dispatch_cb. Measured at ~170,000 crossings
per second during gameplay, from several threads.
Dispatch table is now a fixed array plus an atomic published count. It is
append-only by construction (AllocCodeStub bump-allocates 4 bytes per stub
and refuses past the arena end), entries are never removed or rewritten, and
a writer fills its slot before publishing the count with release ordering -
so an acquiring reader always sees a complete entry. The mutex remains on
the append side, which runs a few hundred times at startup.
Register reads go through one uc_reg_read_batch instead of five API entries.
RESULT, and it is smaller than the reasoning predicted:
time to first OnCarLoaded 36.35s -> 35.20s (35.14 / 35.28 / 35.18)
About 3%, consistent across runs. The estimate that each crossing costs
~6us - derived by dividing a saturated core by the measured crossing rate -
implied far more headroom than this. Either the mutex and register reads
were a small part of the crossing cost, or the thread is not actually
saturated and the estimate is wrong. Recorded rather than explained away;
the change is correct and free either way, but it is not the lever the
arithmetic suggested.
Also adds temporary task #56 instrumentation, all opt-in:
- gles_shim PERF line: fps, draws/frame, shimCalls/frame in ONE line, so the
correlation is readable live instead of joined across logs afterwards.
- guest_engine TOPSHIMS line: the busiest shims once a second, counted per
ImportStubEntry so no shared map or mutex taxes the hot path.
Those two measurements corrected an earlier wrong conclusion. Draw calls
looked like the cause of the in-race slowdown the user demonstrated, but
with the crossing counter in place: draws/frame 178 vs 149 across a 4x fps
gap, while shimCalls/frame went 22,000 vs 5,600 - and fps falls out exactly
as crossings-per-second / crossings-per-frame (148350/4385 = 33.8,
167811/21665 = 7.7). Draws were a bystander.
Co-Authored-By: Claude <noreply@anthropic.com>
A Xiaomi 14 capture showed viewport 2136x960 against a 2670x1200 surface and
raised the question of whether the engine was feeding the game a wrong size -
the shape of task #39, where the UI projection used 2000x1000 instead of the
real 2400x1080.
Logging each DISTINCT viewport with its bound framebuffer and the real EGL
surface size answers it. Pixel 6a:
VIEWPORT [0,0 2400x1080] fb=0 | EGL surface 2400x1080 <- screen, exact
VIEWPORT [0,0 1920x864] fb=3 | EGL surface 2400x1080 <- an FBO
VIEWPORT [0,0 512x512] fb=1 | EGL surface 2400x1080 <- another FBO
The default framebuffer matches the surface exactly; the smaller viewport is
an offscreen target. And the ratio is decisive:
Pixel 6a 1920/2400 = 0.80 864/1080 = 0.80
Xiaomi 14 2136/2670 = 0.80 960/1200 = 0.80
Exactly 0.8 on both devices and both axes - the game's own render scale,
drawing the 3D scene into a reduced FBO and upscaling while the HUD stays at
full resolution. Nothing is broken; the engine reproduces it faithfully.
The probe stays in the tree behind kLogViewportChanges (default off, per the
standing rule that diagnostics are opt-in), because "which framebuffer was
bound" is exactly the context whose absence made the original observation
ambiguous.
Co-Authored-By: Claude <noreply@anthropic.com>
CreateConfiguredEngine mapped the whole guest region with one
uc_mem_map_ptr(UC_PROT_ALL). UC_PROT_ALL includes EXEC, and QEMU's
notdirty_write (qemu/accel/tcg/cputlb.c:1199) does this on every write:
mr = uc->memory_mapping(uc, ...); // region lookup
if (mr && (mr->perms & UC_PROT_EXEC) != 0) {
page_collection_lock(...);
tb_invalidate_phys_page_fast(...);
}
So marking memory executable makes EVERY WRITE to it pay a region lookup
plus a translated-block invalidation check. With the whole region
executable, ordinary stack and data writes all took that path.
Found by elimination rather than guesswork. Two counters were written and
both answered zero: writes into .text (across two sessions) and writes into
the trampoline/stub arenas. Since no guest code and no engine-written stub
bytes were dirtying code pages, the writes driving tb_invalidate had to be
ordinary data writes that merely lived in an executable mapping.
The first split (heap RW, everything above heap_end_ RWX) moved the three
symbols by 0.5 points - i.e. nothing - because it left THREAD STACKS
executable. The heap is 768MB, but stores go overwhelmingly to the stack,
once per call frame. "Biggest region" and "most written region" were not the
same region.
Final layout: image RWX; heap RW; trampoline+stub arenas RWX; control and
thread stacks RW; mmap arena RWX (LoadSecondaryImage maps real code there).
tb_invalidate_phys_page_fast_arm 4.52% -> 0.04%
time to first OnCarLoaded, 3 runs each:
single RWX 40.95 / 39.06 / 38.96 s (mean 39.66)
split 36.31 / 35.87 / 36.88 s (mean 36.35)
Ranges do not overlap: loading is 8.3% faster. Verified end-to-end with an
A/B switch rather than inferred from the profile share, because task #48
established that a share drop is not a speedup. Zero FETCH_PROT/WRITE_PROT
faults, so nothing executes from the now-non-executable spans.
flatview_translate_arm (5.25%) and find_memory_mapping_arm (2.99%) barely
moved, so they have callers beyond notdirty_write - task #53 stays open.
Co-Authored-By: Claude <noreply@anthropic.com>
CallGuestFunction passed count=5000000 to uc_emu_start. Any non-zero count
makes Unicorn install a GLOBAL UC_HOOK_CODE internally (uc.c:1202):
uc_hook_add(uc, &uc->count_hook, UC_HOOK_CODE, hook_count_cb, NULL, 1, 0)
begin=1 > end=0, which HOOK_BOUND_CHECK treats as matching EVERY address. So
helper_uc_tracecode, a gen_set_pc_im PC sync, and check_exit_request were
emitted and executed for EVERY GUEST INSTRUCTION - not per shim call.
That is what the in-race profile had been showing:
helper_uc_tracecode 34.54%
helper_check_exit_request_arm 5.33%
against 0.04% in MiscStubDispatch at ~65,000 stub dispatches/sec, which
worked out to an impossible ~5.4us per dispatch. The arithmetic never fit
because the divisor was wrong: tracecode was per instruction, not per
dispatch.
It also explains task #48's null result. Removing 17 probe hooks shortened
the hook LIST but left this global entry in it, so the per-instruction call
and the per-instruction PC sync - which also prevents TCG from chaining
blocks - both remained.
Measured on the Pixel 6a with the race's own clock, same method throughout:
helper_uc_tracecode 34.54% -> absent from the top symbols
stub dispatches 65k/s -> 76-93k/s (more guest work done)
game time vs wall clock 3.99x -> 2.73x
27s of wall time advanced the race timer 30.13 -> 40.02 (9.89s); the
sub-intervals gave 2.75x and 2.71x. Today's full progression is 7.2x ->
3.99x -> 2.73x.
Trade-off, taken deliberately: a runaway guest loop inside one call now
hangs instead of returning after 5M instructions to be caught by
kMaxCallIterations. That net was already documented in this function as "not
a security boundary", and since the 2026-09-05 stub fix a normal call
completes in a single uc_emu_start anyway. A future watchdog should be one
long-lived thread calling uc_emu_stop, NOT uc_emu_start's `timeout`
argument, which spawns a thread per call (uc.c:1036).
Also turns off the two temporary counters (kCountStubDispatches,
kCountTextWrites) added to find this; the .text write counter answered its
question with a flat zero, killing the self-modifying-code theory behind
tb_invalidate_phys_page_fast_arm.
Co-Authored-By: Claude <noreply@anthropic.com>
Channel::isPlaying answered FALSE for a channel System::playSound had just
handed out with a success code. The guest polled it, concluded the track
had finished, and started the next one - forever. Each cycle reopened the
OBB asset filesystem ON THE GLThread, stalling frames directly.
Found from a user observation ("треки меняются очень быстро в меню паузы"),
then measured: a different track roughly every 1.5s, picked
non-sequentially (DEAD SARA, DEAD SARA, GREEN DAY, SKRILLEX, GREEN DAY,
CROSSES, ICONA POP, JOY FORMIDABLE), so not a walk through playlists.sb but
a fresh selection each time.
Now tracks the channels playSound hands out and reports those as playing
until Channel::stop drops them, so normal transitions still work. A
one-time log states plainly that there is no audio backend and playback is
simulated, instead of leaving that only in a comment.
Measured on the Pixel 6a with the race's own clock, same method both times:
game time vs wall clock 7.69x -> 3.99x
useAssetsFileSystem 1.00/s -> 0.00/s (in-race, fully stopped)
faults 0 -> 0
27s of wall time advanced the race timer 18.82 -> 25.58 (6.76s); the two
sub-intervals gave 4.18x and 3.81x.
Third instance today of one failure shape: a shim reporting SUCCESS while
handing back an empty or negative value. See AndroidBitmap_lockPixels
(task #41) and the synthetic-swap premise (task #47).
Co-Authored-By: Claude <noreply@anthropic.com>
De-registers every diagnostic probe belonging to closed tasks #35, #39 and
#41; the functions stay defined, matching this file's existing convention.
Registrations drop from 20 to 9. What remains is functional only:
MiscStubDispatch, FnvHashAccelHookCb, five zlib_accel hooks, two trampoline
hooks.
Seven of the removed probes sat on the TEXT rendering path (QuadGeometry at
four addresses, TextClipDispatch at two, MeasureAdvance at two), so they
fired once per HUD glyph per frame - continuously during a race.
Motivation was the first in-race profile: helper_uc_tracecode at 63.03% of
CPU. Unicorn's helper (uc.c:2129) walks the entire UC_HOOK_CODE list on
every hit with no early exit, and a hooked instruction also breaks the
translation block.
RESULT - the hypothesis is REFUTED as a performance fix:
helper_uc_tracecode 63.03% -> 41.66%
game time vs wall clock 7.2x -> 7.69x (no improvement)
Measured with the race's own clock: 27s of wall time advanced the race timer
14.97 -> 18.48, i.e. 3.51s; the two sub-intervals gave 7.78x and 7.60x. So
hook-list length was NOT the binding constraint - the freed time simply
redistributed, visible in neighbouring shares rising
(tb_invalidate_phys_page_fast 1.19->3.15%, helper_lookup_tb_ptr 1.58->3.01%,
JIT "unknown" 7.17->9.98%).
Caveat: the before/after races are different sections with different scene
load. That cannot hide a difference of this size, but it is not a controlled
comparison either.
Kept anyway: removing spent diagnostics is worth doing on its own terms, and
a profile no longer dominated 63% by one symbol is readable.
Co-Authored-By: Claude <noreply@anthropic.com>
The engine synthesized an eglSwapBuffers on each detected frame boundary.
That duplicated the present Android's GLSurfaceView already performs after
every onDrawFrame return, so every frame reached the screen twice - the
second time from a buffer still holding the PREVIOUS frame.
The whole mechanism rested on a premise recorded 2026-09-18 and never
verified live: that nativeOnResume runs the guest's persistent loop on the
GLThread and never returns, leaving the framework unable to present. An
entry/exit counter added to GameRenderer.onDrawFrame shows that is false -
it returns every frame (entries=698 exits=697, the gap being the call in
flight). Rates match 1:1:
synthetic swaps 12.98/s (SWAPMARK, one line per swap)
onDrawFrame 12.69/s (so the framework swapped 12.69/s too)
The user saw this as the car jumping forward then back along its path.
Frame-by-frame the race clock stepped 1:10.06 -> 1:10.03 -> 1:10.09 ->
1:10.06, with presents arriving in pairs 10-20ms apart at a 218ms period -
exactly the synthetic swap period.
Letting the framework present is also what the real game does: libapp.so
cannot present at all (only EGL import is eglGetProcAddress; the string
"eglSwapBuffers" is absent from the binary), and on the A9 all 2,736
eglSwapBuffers calls in 46s came from Android's own framework.
Measured after, on the Pixel 6a:
frame interval median 89ms -> 17ms
p90 217ms -> 19ms
max 316ms -> 24ms
synthetic swaps 0 (verified, not merely absent from view)
onDrawFrame 12.69 -> 12.39/s (throughput deliberately unchanged)
User verdict: "Сейчас очень плавно". Throughput is untouched - this fixes
presentation, not the ~7x simulation deficit.
Kept behind kSynthesizeSwap rather than deleted: the 2026-09-18 symptom it
was built for (screen cycling between stale loading screen, black, and the
real scene) was diagnosed under the false premise above, so its actual
cause is still open.
Co-Authored-By: Claude <noreply@anthropic.com>
Shim_AndroidBitmap_lockPixels handed the guest a NULL pixel pointer
whenever the real buffer lived outside the guest region, on the
documented assumption that "texture decode FROM a Bitmap ... reads
pixels via other means, not by writing through this specific pointer".
Measurement disproved that assumption. The glyph atlas path is exactly
a read through this pointer:
AndroidBitmap_lockPixels rc=0 hostAddr=0x... -> guest=0x0
bitmap 1024x1024 stride=4096 format=1 (RGBA_8888)
1024x1024 is precisely the atlas size independently derived from the
glyph UVs (u1 = 35/1024 for a 35px glyph), which is why glyph
positions, sizes, pen advance and UVs all measured correct while the
pixels were coloured noise: BitmapGraphics rendered the glyphs fine,
the guest asked for them, got nothing, and uploaded guest address 0
as the font texture.
Fixed with a bounce buffer - real pixels copied into a guest-visible
buffer on lock, copied back and freed on unlock. The copy-back also
closes the write direction the original comment called a known gap.
Costs one height*stride copy each way, a handful of times per session.
Verified in both directions on the Pixel 6a: text renders correctly,
and the added "no bounce buffer was made" warning fired zero times,
so every lock genuinely got a buffer rather than silently falling
back to the old NULL path.
Co-Authored-By: Claude <noreply@anthropic.com>
First piece of actual verification infrastructure, rather than
symptom-driven debugging. Off by default (kHeapDebugChecks), so a normal
build is byte-for-byte the same arithmetic as before: kRedZoneBytes is 0 and
every guard branch folds away at compile time.
With checks on:
* each block carries a 16-byte guard immediately after the bytes the caller
asked for - deliberately not at the end of the block, where slack left by
a reused larger block would hide a small overflow - verified on Free and
reported with the offending block's address and size;
* freed payloads are poisoned, which matters more since Alloc stopped
zeroing: without it a freed block keeps plausible stale contents and a
use-after-free silently keeps "working".
The check is deliberately not fatal. A loud, precise report is the point;
aborting mid-run would make the engine harder to debug than the corruption
being reported.
BlockHeader's former alignment-only `pad` field now carries the user-visible
size, maintained whether or not checks are on, so BlockSize() reports what
the caller actually got and realloc copies the right amount.
Verified in both directions before trusting it: a deliberate one-byte
overflow was caught and attributed to the right block ("guard byte 0 of 16
is 0x41, expected 0xbe"), while a clean neighbouring block produced no
report. Then a full run to the race with checks on: zero violations, zero
rejected frees, zero faults. That is partial evidence - the guard is only
checked when a block is freed, so blocks that live to process exit are never
examined - but it is the first evidence of its kind this project has.
Co-Authored-By: Claude <noreply@anthropic.com>
sub_4F3704 (resource lookup by name) is 18.8% of load-time samples and does
a linear strcmp scan over 6232 entries because the game's own hash cache is
disabled (*(self+8) == 0, confirmed by a live probe: 320,000+ calls per load).
Attempt 1 crashed the process: guest strings were read through G2H and
scanned for a NUL with no bounds check, so a bad offset walked off the end
of the mapped region (SIGSEGV, SEGV_ACCERR at a host address).
Attempt 2 was memory-safe (uc_mem_read everywhere, length and count caps,
every cache hit verified against live guest memory, falls through to the
guest on any doubt) but caused a visible frame-rate drop on the prologue
loading screen. The cause was a design flaw the earlier probe log had
already shown and I misread: the cache was keyed on the table ADDRESS, yet
this game reuses one address for different tables (35 and 59 entries
alternating in the log). The descriptor comparison therefore marked the
index stale on nearly every call, and each rebuild re-read all ~6232 entry
strings byte-by-byte - far more work than the scan it replaced.
Kept in the tree, unregistered, with both mistakes documented. The fix for a
third attempt is to key the cache on the DESCRIPTOR CONTENTS rather than the
address, so alternating tables each keep their own index, plus bulk string
reads instead of per-byte.
Co-Authored-By: Claude <noreply@anthropic.com>
Verified live on a Pixel 6a: the game passes its EULA, loads the prologue
and renders real 3D gameplay, with zero heap exhaustion, zero faults and
zero rejected frees over a full session.
Root causes fixed in this state, each backed by a measurement (details and
the list of refuted theories live in ARM64_TRANSLATION_LAYER.md):
* JNI varargs float->double promotion. C promotes float to double in any
varargs call and every Call*Method form is varargs, so reading one 4-byte
slot yielded the double's always-zero low half. EVERY float argument
passed to Java was silently becoming 0; text was just where it showed.
* GuestHeap ~4x memory overhead. Power-of-two size classes carving the full
class, plus segregated free lists that could never share memory between
sizes. Reworked to exact sizing with O(log n) best-fit reuse and splitting
(deliberately not a linear scan - this allocator already had an O(n) perf
cliff in its history). Peak live now 207MB against the real A9's 199MB,
fragmentation ~2.5MB. Also fixed: realloc reading past the old block on
shrink, a 32-bit overflow in calloc, and drifting payload alignment.
* Unbounded FMOD fake-handle leak into the never-freeing permanent arena,
which is why enlarging that arena had not helped.
* Frame presentation, corrected against A9 ground truth: the real frame has
three default-framebuffer colour clears and ONE present at the end; this
engine had been presenting on each of them.
Load-time acceleration (zlib_accel.cpp): host zlib now serves inflate and
crc32, the latter measured by the block profiler as the single hottest guest
routine at 17.7%. Streams are only taken over when this layer saw their own
inflateInit2_, so unknown streams (libpng's, among others) still run the
original emulated path.
name_lookup_accel.cpp is present but its hook is NOT registered - it crashed
on bad assumptions about guest table lifetime and is kept as a starting
point, with both mistakes recorded in its comments.
Co-Authored-By: Claude <noreply@anthropic.com>
GuestHeap previously bump-allocated any request >64KiB and never
reclaimed it on free() - a real, sustained gameplay session eventually
exhausted the arena (traced live to two separate crashes: onCreate
never completing, and a NULL-vector dereference in sub_4BA588 caused
by a failed ~175KB allocation going unchecked by the real EA code).
Widen the existing O(1) size-class free list (already used for <=64KiB
requests) to cover the whole arena instead of adding a second reuse
mechanism next to it - oversized allocations now pool and reclaim the
same way small ones already did. Add matching desktop tests
(guest_heap_test.cpp, run via run_heap_tests.sh) covering the exact
failing size from the crash log and a sustained alloc/free cycle in an
arena too small to survive without reclaim.
Verified live on device: the crash is gone across a full 10-minute
session with sustained GLES rendering throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Hardcode car class enum for lobby car selection (fixed, small set by design)
- Fix GetCurrentCarColor() crashing on an unvalidated color index from an
unzeroed hashmap-miss fallback record on a genuinely fresh save
- Default kEnableTrackSubstitutionHook to false: it substituted every race's
track unconditionally, crashing the game's own scripted prologue race
- Remove GetComponentNameSkipHook from JNI_OnLoad: it unconditionally
replaced a real name used by the same cache-context the per-event medal
progress record resolves through, silently breaking medal/street-completion
tracking for every real race
- Split lan_event_injection.h (3149 lines) into car_selection.h,
crash_workarounds.h, mod_slot_tracking.h, and a shared
util/hook_install.h trampoline helper; delete ~490 lines of confirmed-dead
experimental code; trim comment-heavy sections to essential context
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes the synthetic car_select jump for cold sessions, makes the loadout
exit chain safe for real (non-synthetic) events, and adds a native->Kotlin
GameEvents bridge (onMapLoaded/onRaceStarted/onRaceEnded/onUpgradesAccepted/
onCarSelected) so both the UI layer and a future native RatNet client can
learn what the player picked - car id, accepted upgrades, and paint color
(name + RGBA) are all resolved live from the game's own engine state
rather than a static extracted table, so they stay correct for any car
added later. Includes a Jetpack Compose overlay as a worked example of a
UI-side GameEventListener consumer.
Full investigation history, root causes, and the several dead ends ruled
out along the way are documented in PROGRESS.md (cont. 30-63b).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces the "inject into every pin" visual-confirmation mode with
matching against MapTrack's own event-group-name vector (offsets
0xE0/0xE4), which is available as soon as HandleEvent(evtype=1025)
fires regardless of whether any RaceEvent has resolved yet. Injection
now targets exactly one street (kTargetGroupName) and the card's
displayed name is an explicit parameter instead of a fixed constant.
Live-tested: single-target injection hits exactly the intended widget
with zero effect on the other 11 reachable MapTrack instances. The 12
reachable group names are all region/career-progression placeholders
(region{1,2,4,5}_{foothills,desert,chicago,newyork}_track{1,2,3}) -
none correspond to the currently visible/playable on-screen streets,
which a live AddEvent-level trace confirmed never call AddEvent during
normal play at all (see PROGRESS.md 2026-08-08 cont. 9/10 for the full
investigation, including the now-reverted diagnostic hooks used to
establish this).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Hooks MapTrack::HandleEvent to fabricate a RaceEvent+CashReward+fake
Actor and insert it via the engine's own AddEvent, making a synthetic
"LAN: <lobby>" card appear on an already-loaded street entirely at
runtime (no game_cache/OBB/native_lib changes). Also hooks the
per-frame MapScreen tick to neutralize a QA-only "Soak Test" feature
whose index-based scan of a parallel, unsynced list was the root
cause of a delayed crash on injected entries.
Verified live on Galaxy A9 (2018): zero crashes across all reachable
pins, and a 10-minute soak test with zero crashes after the Soak Test
fix, confirmed against a genuinely responsive post-test map screen.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Lets the track-substitution hook be enabled/disabled by flipping one
constexpr bool in main.cpp instead of commenting/uncommenting code, so
before/after comparisons don't need back-and-forth edits.
megboyzzandClaude