Benchmarking

The engine can time itself. A benchmark builds a procedural load in front of a scripted camera, renders it offscreen for a fixed number of frames, and writes the frame-time distribution to a JSON file and a table you can read. No window, no swapchain, no server, no simulation — the same offscreen path --render uses, timed instead of photographed.

This is not a routine check

Nothing runs a benchmark for you. It is not in CI, it is not in a pre-commit hook, and it is not part of the test suites — it needs a real GPU, it saturates it for the length of the run, and its numbers are only comparable against numbers from the same machine in the same state.

Reach for it at one specific moment: you changed something graphical and want to know what it cost. Take a reading before the change, take one after, and diff them. Outside that moment it tells you nothing you did not already know, and it costs you a GPU for a few minutes.

From the console

benchmarkList                        list every scenario and suite
benchmark                            run the default suite (quick) into the engine's benchmarks folder
benchmark <scenario|suite>           run one scenario, or a named suite
benchmark <scenario|suite> <dir>     …and write the result file to <dir>

With no directory the result lands under the engine’s user data root, in benchmarks/.

Disconnected only

benchmark refuses to run unless the client is disconnected. That is not fussiness. A run opens its own device and then saturates the GPU on the thread that also owns the window, so a session behind it would be starved of frames, its prediction would run against timings that describe nothing, and the numbers would be a measurement of the renderer plus whatever the session was doing.

Get to a disconnected client with --menu at launch, or by leaving the session. A dedicated server never registers the command at all — it has no client and no renderer, so there is nothing there to benchmark.

The run blocks the frame loop until it finishes. The command says so before it starts. Sharing the frame loop with the thing being timed would put the client’s own rendering inside every reading.

From the command line

For scripted runs — a build agent, a before/after pair driven from a batch file, a bisect — the same run is a launch mode:

DigitalHeaven.Engine.Host --benchmark <scenario|suite> --benchmarkOutput <dir>
                          [--benchmarkFrames <n>] [--map <barcode|name>]
                          [--width <px>] [--height <px>] [--set <key>=<value>]...

Flag	Argument	Default
--benchmark	A suite (quick, full) or a single scenario slug. Required — it is what selects this mode.	—
--benchmarkOutput	Directory the JSON result file lands in. Required.	—
--benchmarkFrames	Frames measured per scenario (1–100000).	240
--map	The backdrop map the procedural load is built over.	The bundled map
--width	Render width in pixels.	1600
--height	Render height in pixels.	900
--set	<key>=<value>. Overrides one preference for this run only. Repeatable.	—

An unknown scenario name is rejected at parse time, before a Vulkan device is created and a map is opened, and the message lists every suite and scenario there is. So is a mistyped frame count.

The mode is exclusive, and the contradictions are refused rather than half-honored:

• --benchmark with --render — different offscreen modes: one photographs a map, the other times a procedural load.
• --benchmark with --headless — one times the renderer; the other runs a server with no renderer at all.
• --benchmark with --connect — a benchmark times a local offscreen renderer; a connected client joins a server instead.
• --benchmark with --menu — an offscreen run has no session to hold at a menu.
• --benchmark without --benchmarkOutput — a run whose numbers are never written down is a GPU spent on nothing.
• --benchmarkOutput or --benchmarkFrames without --benchmark — they size and place a run that was never asked for.

Scenarios

A scenario is a named load plus a named set of renderer features, measured over the shared camera path. Scenarios come in two shapes.

Ladders hold the features at their defaults and vary only the load, so they answer how does this scale:

Scenario	Props	Lights
geometryLow	128	32
baseline	512	32
geometryHigh	2048	32
lightsLow	512	8
lightsCut	512	64 — exactly the shading cut
lightsHigh	512	256 — four times the cut

The light ladder deliberately straddles the shader’s limit. Past the cut the extra lights are still submitted, ranked and discarded, so lightsHigh minus lightsCut is the cost of light selection on its own.

Pairs hold the load at the baseline and differ in exactly one toggle, so subtracting one half from the other is a fair reading of that feature’s cost:

Pair	Toggle
aoOff / aoOn	client.render.ao
taaOff / taaOn	client.render.taa
bloomOff / bloomOn	client.render.bloom
shadowsOff / shadowsOn	world.sun.shadows
msaaOff / msaa4x	client.render.msaa
lightmapOff / lightmapOn	client.render.lightmapIntensity

Two named suites group them:

• quick — the baseline, the far end of each ladder, and the two toggles most likely to have just changed (aoOn, taaOn). Sized so a run finishes while you are still looking at it. This is what a bare benchmark runs.
• full — every scenario, in catalog order.

The baseline earns its place in every suite

baseline is what every other row in the summary is compared against. A run without it still produces numbers, but every “vs baseline” column is blank — which makes it the run that answers the least. Both shipped suites include it; a single-scenario run deliberately does not fabricate one.

The scene is procedural, not content

The load is built in code, not authored as a map: prop count and light count are numbers, so a tier is exact and a rung of a ladder is reproducible without anyone opening an editor. A content-authored benchmark scene would drift the moment someone tidied it, and every historical reading would silently stop being comparable.

Props are unit cubes placed on a golden-angle spiral through a fixed field; lights ring the field at a fixed radius and height. Both are pure functions of the tier, so geometryHigh is the same 2048 boxes today as it was last month.

A real map is still opened underneath, as a backdrop: it supplies the static geometry the scene is measured against, the material names the props borrow, an anchor point for the camera path, and the world look and lighting the renderer resolves. By default that is the engine’s bundled map; --map picks another. The backdrop’s own bloom and sun-shadow authoring is overridden by the scenario, because an A/B pair has to differ in the toggle rather than in whatever the map happened to author.

The camera path

Every scenario is measured over the same flythrough: one closed orbit around the scene anchor, dollying in and out twice and rising and falling once, always looking at the anchor.

It is a pure function of the frame index and the frame count — no clock, no integration, no state. Frame n of a 240-frame run puts the camera in exactly the same place it did last time, so a difference between two runs is a difference in the renderer. The path is a loop rather than a line, so a run of any length covers the scene evenly and ends where it started, and a warmup pass that runs past the end keeps moving instead of parking on one frame.

The dolly is what makes it worth measuring. A fixed-radius orbit draws roughly the same number of pixels every frame; pushing the eye from the edge of the prop field to its center and back sweeps overdraw across its whole range, which is where a fill-bound regression actually shows up.

Warmup

Each scenario renders discarded frames over the same camera path before the measured ones begin — for 750 ms, and never fewer than 30 frames. Pipelines compile, descriptors allocate, the driver caches its shaders, and the temporal resolve converges. All of that is real work and none of it is what the run is asking about.

The warmup is a duration, not a frame count — and that is load-bearing

A frame-count warmup is self-defeating on a modern GPU. The dominant thing a warmup has to outlast is not shader compilation, it is the boost-clock ramp: an idle device sits at a low clock and takes hundreds of milliseconds to come up. A cheap scenario burns through 30 frames in about 30 ms, so it is measured on a device that is still spinning up, while a scenario later in the same run inherits the heat of the one before it.

The first build of this harness did exactly that, and it produced an aoOn reading 29% faster than a baseline with identical settings — a difference that was purely a function of where in the run the scenario sat. Timing to a wall clock instead makes every scenario start from the same clock state, whatever its cost per frame.

This is also the harness’s own validity check. aoOn and baseline differ only in a preference that is already at its default in both, so they must read the same. When they do not, do not trust the rest of the table — something is drifting under the measurement.

The whole run shares one device and one loaded map. A per-scenario device would put a fresh Vulkan instance, a fresh allocator and a fresh texture upload between every pair of readings, and the difference between two scenarios would then include whatever those cost. Each scenario still gets its own virgin preference store, exactly as --render does, so no saved profile and no earlier scenario can leak into a reading.

Output

Both outputs are rendered from the same finished result, so the file and the console can never disagree about what happened.

The summary

benchmark quick on NVIDIA GeForce RTX 4080 SUPER at 1600x900, 240 measured frames per scenario (750 ms warmup)
scenario        props lights   draws  msaa   p50 ms   p95 ms   p99 ms   max ms  p50 fps
baseline          512     32     516     4     1.01     1.43     1.67     2.15      992
geometryHigh     2048     32    2052     4     2.33     3.27     3.57     4.05      429   +131.1% vs baseline
lightsHigh        512    256     516     4     1.47     2.23     2.64     2.85      681    +45.7% vs baseline
aoOn              512     32     516     4     1.03     1.49     1.86     2.11      966     +2.7% vs baseline
taaOn             512     32     516     1     1.04     1.51     1.80     2.16      959     +3.5% vs baseline

The relative column is what a “did this cost anything” question is actually read off, so it is computed for you rather than left to whoever opens the file. The baseline is not compared with itself.

The JSON

A timestamped file, benchmark-<selection>-<yyyyMMdd-HHmmss>.json, written by hand rather than serialized off the result types — so the shape an external tool depends on is stable against a field being renamed.

{
  "schema": "dh.benchmark/1",
  "startedUtc": "2026-07-27T09:30:00.0000000Z",
  "device": "NVIDIA GeForce RTX 4080 SUPER",
  "map": "core",
  "selection": "quick",
  "width": 1600,
  "height": 900,
  "warmupMs": 750,
  "measuredFrames": 240,
  "maxShadedLights": 64,
  "overrides": { "client.render.aoQuality": "2" },
  "scenarios": [
    {
      "scenario": "baseline",
      "summary": "Every feature at its default over the shared baseline load.",
      "load": { "props": 512, "lights": 32 },
      "features": {
        "client.render.ao": "true",
        "client.render.taa": "false",
        "client.render.bloom": "true",
        "world.sun.shadows": "true",
        "client.render.msaa": "4",
        "client.render.lightmapIntensity": "1"
      },
      "drawItems": 516,
      "submittedLights": 32,
      "msaaSamples": 4,
      "warmupFrames": 650,
      "frames": 240,
      "frameTimeMs": { "mean": 1.042, "min": 0.792, "p50": 1.008, "p95": 1.43, "p99": 1.668, "max": 2.151 },
      "fps": { "mean": 959.562, "p50": 992.26, "p99": 599.377 }
    }
  ]
}

Every field under features is a real preference path you can paste back into the console, so a result file says how to reproduce itself. Every field under a scenario is observed rather than requested: msaaSamples is the count the device and the temporal pass actually granted (temporal antialiasing forces multisampling off while it runs, and the file records the truth), drawItems is what the scene really queued, and warmupFrames is how many frames the 750 ms warmup got through — which is also a rough read on how cheap the scenario was. The 650 above is the whole argument for a duration: a frame-count warmup would have stopped at 30.

Percentiles are nearest-rank — the p-th percentile is the value at rank ceil(p x n), an observed frame rather than an interpolation between two of them. A p99 that is a real frame is a frame someone actually waited through. The same arithmetic backs the frame-time overlay, so a p99 means one thing wherever it is printed.

What is not measured

The engine has no GPU timer queries, so there is no per-pass GPU breakdown to record — and none is invented. What is measured is wall time around a fully fenced frame: command recording, submission and GPU execution together. That is the number that moves when a graphics feature gets more expensive, and it is the number a player experiences.

There is also no triangle counter. drawItems and submittedLights are recorded instead, alongside maxShadedLights, so a reading can at least be attributed to a quantity of work.

Reading a result honestly

• Compare like with like. Two runs are comparable when they came from the same machine, the same resolution, the same frame count and the same driver. The result file records all four so you can check.
• Close everything else first. A browser playing video behind the run is in the reading.
• Trust the median before the mean. One 400 ms hitch moves a mean on its own; p50 will not notice it and p99 will show it to you separately.
• A single scenario is a spot check, not a benchmark. Feature pairs exist because the interesting number is a difference, and a difference needs both halves measured back to back on the same device.
• Do not cut --benchmarkFrames to make a run finish sooner. The default of 240 exists because these frames are short. On a fast card the baseline scenario runs at about 1 ms, so 60 frames is 60 ms of measurement — short enough that one scheduler hiccup moves the median. Measured on an RTX 4080 SUPER, aoOn against baseline — a pair that must read equal — stayed inside a few percent across repeats at 240 frames, and swung between +9% and +23% at 60. Use the low counts to check that a run works, not to read a number off it. If a feature’s cost is smaller than that spread on your machine, raise the frame count or the resolution until it is not.