Building an In-Browser Replay System: Recording Player Input for Debugging and Highlight Reels
Have you ever watched a bug happen exactly once, in front of a player, with nothing in the console to show for it? Every web game team eventually hits that wall, and logs alone will not get you across it.
An input replay system is the standard answer to that problem, and it happens to be the cheapest highlight-reel pipeline you will ever build. Both capabilities fall out of the same recording, provided that recording is deterministic enough to trust.
An input replay stores a header, a random seed, and a stream of tick-indexed input events. It never stores rendered frames, which is why a ten-minute session fits in kilobytes instead of megabytes.
Why Input Capture Beats Video Capture For Debugging
The instinct is usually to reach for MediaRecorder and record the canvas, because video is easy to explain to a QA team and easy to attach to a ticket. That works fine as evidence, and it is close to useless as a reproduction.
A video tells you what the player saw at 30 frames per second. It does not tell you what tick the desync started on, what the entity count was, or which input arrived one frame late.
Input capture gives you the actual causes rather than the pixels downstream of them. Feed the same inputs back into the same simulation and the bug happens again, on your machine, with a debugger attached and the ability to step tick by tick.
The cost difference is not subtle either. Here is how the four common capture strategies compare on a typical browser title running a 60 Hz simulation:
| Capture strategy | Data per minute | Survives a build change | Best for |
|---|---|---|---|
| Input-only event stream | 1 KB to 12 KB | No, requires build pinning | Crash repro, desync hunting, clips |
| Input plus periodic keyframes | 30 KB to 300 KB | Partially, keyframes reseed state | Scrubbable replays, long sessions |
| Full state snapshot every tick | 2 MB to 20 MB | Yes, no simulation needed | Short forensic captures only |
| Encoded canvas video | 4 MB to 15 MB | Yes, it is just pixels | Marketing assets, bug screenshots |
Most teams end up running the second row in production and the fourth row in marketing. The input stream is the source of truth, and the keyframes exist purely so a producer can drag a scrub bar without waiting six minutes for the simulation to catch up.
What Actually Goes Into A Replay File
A replay file is a header followed by a delta-encoded event stream, and the header matters more than most teams expect. Everything you fail to record in the header becomes an assumption that quietly breaks six weeks later.
The header carries the context needed to rebuild the starting conditions exactly. A workable header includes but is not limited to:
- Build hash. The exact commit or bundle fingerprint the recording came from. This is the single field that decides whether a replay is evidence or noise.
- Simulation seed. The integer that initializes your seeded PRNG. Every random draw in the session must come from this generator and nowhere else.
- Tick rate and timestep. The fixed integer milliseconds per simulation step, recorded rather than assumed, so a later tuning change does not silently reinterpret old files.
- Initial world state or level ID. Either a full serialized snapshot at tick zero or a deterministic level identifier plus generation parameters.
- Schema version. A monotonically increasing integer on the replay format itself, separate from the build hash, so you can write migrations.
- Input device profile. Whether the session came from keyboard, touch, or a pad, since axis quantization and dead zones differ across all three.
The event stream that follows is deliberately boring. Each entry is a tick index, a device channel, an action ID, and a value, with the tick index stored as a delta from the previous event so long idle stretches cost almost nothing.
Note that you record actions, not raw key codes. Recording JUMP rather than KeyW means a replay survives a keybinding change, and it keeps the same file playable whether the session originated on a keyboard or through the Gamepad API.
Why Determinism Is The Whole Ball Game
An input replay only works if replaying the inputs produces the same world. That sounds obvious, and it is where nearly every first attempt falls apart.
Determinism means identical inputs produce identical outcomes on every machine. Without a fixed timestep, a seeded PRNG, and stable iteration order, a replay drifts within seconds and stops being usable as evidence.
The hazards are well known and they are all fixable, but each one has to be closed deliberately. In practice the drift comes from a short list of sources:
- Variable timestep. Any simulation that multiplies by
deltaTimefromrequestAnimationFramewill diverge between a 60 Hz laptop and a 144 Hz desktop. A fixed timestep game loop with an accumulator is a hard prerequisite, not a nice-to-have. - Unseeded randomness. A single stray
Math.random()anywhere in the update path poisons the whole recording. Ban it at lint level and route every draw through one seeded generator whose state advances only inside the simulation step. - Iteration order. Object key order,
Setinsertion order after a delete, and hash-map traversal all vary in ways that change collision resolution order. Sort entities by a stable numeric ID before every pass that can mutate shared state. - Wall-clock reads.
Date.now()andperformance.now()inside gameplay logic make the outcome depend on when you pressed play. Derive all in-simulation time from the tick counter. - Floating-point drift. Cross-browser transcendental functions such as
Math.sinandMath.poware not bit-identical across engines. Fixed-point integer math or a lookup table removes the problem entirely for the physics that matter.
If your title already runs deterministic lockstep networking, most of this work is done and the replay system is close to free. If it does not, treat the replay project as the forcing function that finally makes the simulation deterministic, because the payoff extends well past debugging.
Keep in mind that determinism also disciplines your input buffering layer. Inputs have to be sampled and committed on tick boundaries, not applied the instant the browser event fires, or the recorded tick index will not match the tick where the effect actually landed.
How Big The Storage Budget Gets
Storage is where replay projects get killed, usually because someone benchmarked the naive version. The naive version writes every entity's position every tick and produces megabytes per minute.
Budget roughly 20 to 200 bytes per second of gameplay for input-only capture. Add a state keyframe every five to ten seconds and the file grows by your serialized world size divided by that interval.
Real numbers help here. A platformer with six actions and a moderately active player generates somewhere between 4 and 20 input events per second, at roughly 5 bytes per delta-encoded event, which is 20 to 100 bytes per second before compression.
That is a 12 KB to 60 KB file for a ten-minute session, and gzip or the browser's CompressionStream typically removes 70 to 85 percent of it. A twin-stick shooter with analog aim sampled every tick sits higher, near 200 to 400 bytes per second, because the aim axis changes constantly and delta encoding has less to exploit.
Quantization is the lever that fixes the analog case. Storing aim as a single byte of 256 angular steps instead of two 32-bit floats cuts that channel by 87 percent, and no player has ever noticed the difference in a replay.
Keyframes are the other budget line. If your serialized world state is 40 KB and you write a keyframe every 600 ticks, a ten-minute recording carries 60 keyframes and 2.4 MB of state, which dwarfs the input stream and is still a fraction of the video equivalent.
Where To Store Replays In The Browser
The storage decision follows directly from those file sizes. Anything past a few hundred kilobytes rules out the simple option.
Store replays in IndexedDB rather than localStorage. localStorage is synchronous, string-only, and capped near 5 MB, while IndexedDB handles binary blobs asynchronously and scales into the hundreds of megabytes.
Write the event stream as an ArrayBuffer and store it directly, without a JSON or base64 round trip that would inflate it by a third. The patterns for quota handling, eviction, and transaction batching are the same ones covered in IndexedDB for browser games, and replay blobs are a textbook case for them.
Do not persist every session. Keep a rolling in-memory ring buffer holding the last 60 to 120 seconds of input, and flush it to IndexedDB only on a crash handler, an error report, or a player pressing the clip button.
This one decision usually drops storage pressure by two orders of magnitude, because the overwhelming majority of sessions contain nothing anyone will ever watch.
Uploads deserve the same restraint. A replay attached to a crash report is worth the bandwidth, and a background upload of every session is a quota bill with no reader on the other end.
If replays sync across devices alongside progression data, the identity and conflict rules in cross-platform save state apply unchanged. A replay is just another versioned blob keyed to a player.
Building The Playback Path
Playback is the same simulation with the input source swapped. Instead of reading from the live event queue, the loop reads from the recorded stream and applies whatever is stamped for the current tick.
That symmetry is the whole design, and it is worth protecting architecturally. The moment playback needs a special code path inside gameplay logic, the replay stops testing the thing you actually ship.
Seeking is where the design earns its keep. Without keyframes, jumping to minute eight of a ten-minute recording means simulating 28,800 ticks, which will lock the main thread for several seconds.
With a keyframe every 300 to 600 ticks, seeking means loading the nearest snapshot and fast-forwarding at most ten seconds of simulation with rendering disabled. That reliably lands under a few hundred milliseconds, which is fast enough that a scrub bar feels responsive.
Fast-forward should skip rendering entirely rather than skipping simulation. Every tick still runs, because dropping ticks changes the outcome, and the frame you never drew cost nothing.
Playback at 4x or 8x speed follows the same rule of running more ticks per frame rather than larger ticks. If the catch-up work threatens frame budget, moving the simulation into a worker alongside OffscreenCanvas rendering keeps the UI thread free for the scrub bar and transport controls.
Cutting Highlight Reels From The Same Recording
The highlight feature is largely a marketing request that a debugging system can satisfy for free. Once the replay is deterministic and scrubbable, a clip is a very small amount of additional metadata.
Highlight reels and debugging use the same recording. A clip is a tick range plus a camera track, so exporting a highlight means replaying ticks 4,200 through 4,800 with a follow camera instead of the player camera.
Marking clips is best done by the simulation rather than the player. Emit a candidate marker whenever a scoring event, a multi-kill, a personal best, or a near-death recovery fires, then keep the surrounding 8 to 15 second window.
Because the camera is not part of the simulation, you can re-frame the same moment freely. A cinematic orbit, a slow-motion segment at a quarter tick rate, or a wider field of view all come from re-rendering identical simulation output, which is the same decoupling that makes browser spectator mode work.
Rendering to a shareable file is the last step, and it is the one place video belongs. Run the replay at fixed speed into MediaRecorder with the canvas as the source, and let the encoder produce a WebM the player can post anywhere.
What Breaks In Production, And How To Catch It
Replay systems fail quietly, which is the worst failure mode a debugging tool can have. A replay that plays back subtly wrong is more expensive than no replay at all, because someone will chase the wrong bug for a day.
The most common replay failure is a build mismatch. Pinning each recording to a build hash and refusing playback on mismatch converts a silent wrong answer into a loud, honest error.
Schema drift is the second offender. When someone adds an input channel or reorders an enum, older files decode into plausible-looking nonsense unless the format version gates them into a migration path.
The defense is a validation suite that runs in CI on every commit. Keep a corpus of 20 to 50 recorded sessions, replay each one headlessly, and compare a checksum of the world state at fixed tick intervals against the stored expectation.
That suite catches accidental non-determinism the day it lands rather than the week you need it. It also doubles as a regression net for gameplay tuning, since an intentional balance change will light up the corpus and force someone to re-record the baselines on purpose.
Be aware that replay data is not an anti-cheat mechanism on its own. A client-authored input stream can be fabricated as easily as any other client payload, so server-side verification along the lines described in anti-exploit validation still has to do that job.
A Shipping Checklist
Most of the work in a replay system is sequencing, not novelty. Here is the order that tends to avoid rework:
- Make the simulation deterministic first. Fixed timestep, seeded PRNG, sorted iteration, no wall-clock reads in gameplay. Nothing downstream works until this holds.
- Record actions, not raw events. Semantic action IDs with tick indices survive keybinding changes and device differences that raw key codes do not.
- Ship the header before the optimizer. Build hash, seed, tick rate, schema version, and device profile cost a few dozen bytes and prevent the failures that waste days.
- Buffer in memory, persist on signal. A 120-second ring buffer flushed on crash or on demand keeps quota usage flat regardless of session length.
- Add keyframes only when scrubbing is a requirement. They multiply file size, so introduce them when a human needs a timeline, not before.
- Gate playback on the build hash. Refuse mismatched files with a clear message rather than playing them and producing a wrong answer.
- Run the corpus in CI. Checksum comparisons at fixed ticks turn determinism from a hope into a test.
Worked in that order, the debugging tool and the highlight feature ship from one codebase. The alternative — building a clip exporter first and retrofitting determinism later — reliably costs more, because the retrofit touches every system the exporter already depends on.
Common Questions About Browser Replay Systems
A few questions come up on nearly every implementation, usually once the first version is running and the edge cases start arriving:
What happens when a replay hits a network response?
Server responses are inputs like any other, so record them into the same tick-indexed stream with their arrival tick. On playback the recorded response is injected at the same tick and no network call is made, which is what lets a replay run fully offline.
How much frame time does recording actually cost?
Appending a delta-encoded event to a preallocated typed array costs well under 0.05 ms per tick, which is noise against a 16.6 ms budget. The measurable cost is the keyframe serialization, so run it on a tick where you already expect a spike and never inside a scroll or animation handler.
Can a replay be exported as a standard video file?
Yes, by piping the canvas stream through MediaRecorder while the replay runs at fixed speed. Record at a locked frame rate rather than real time, since a dropped frame during encoding produces a stutter in the output even though the simulation itself was correct.
Do replay files contain personal data?
An input-only stream carries no names, no chat, and no device identifiers unless you deliberately put them in the header. Keep player identifiers out of the file body and attach them at upload time, so a replay shared with a teammate for debugging leaks nothing on its own.
Is a replay system the same thing as rollback netcode?
They share the determinism requirement and most of the same plumbing, but the direction differs. Rollback re-simulates forward from a recent state to correct a prediction, while a replay re-simulates from the beginning to reproduce an outcome that already happened.
Where To Take This Next
If you are scoping a replay system for a browser title and want a second read on the capture format before you commit to it, the format decision is the one worth getting right early. Everything else in the pipeline can be rewritten cheaply, and a file format with a missing header field follows you for the life of the product.
Start with the determinism audit, keep the first version input-only, and add keyframes the week someone asks for a scrub bar. The debugging value arrives on day one, and the highlight reels arrive later at almost no additional engineering cost.
