Engine Assets & Audio

Milestone 4 gave the engine three runtime capabilities: it loads content from compiled DigitalHeaven pallets, samples 2D albedo textures in the mesh pipeline, and plays mixed, spatialized audio.

Runtime Assets

The engine consumes the same compiled .pallet artifact that DigitalHeaven.Unity does — it is a content consumer of the DigitalHeaven toolchain, not a second asset format. DigitalHeaven.Engine.Assets reads a compiled pallet through DigitalHeaven.Core’s LoadedCompiledPallet and turns its blobs into GPU-ready resources:

• PNG blobs → textures. Decoded on the CPU with StbImageSharp (a tiny, pure-managed decoder), then uploaded to the GPU.
• GLB blobs → meshes. Loaded into the engine’s own MeshAsset format via SharpGLTF.
• Materials are resolved through semantic texture slots, mirroring how the rest of the DigitalHeaven material system works.

Mesh vertex format

MeshVertex is a fixed 56-byte record — there is no vertex-format negotiation, and every mesh the engine draws has exactly these five attributes:

Attribute	Type	Offset	Notes
Position	float3	0	Meters, world-handed (Y-up, right-handed, −Z forward)
Normal	float3	12	Unit length
Tangent	float4	24	glTF convention: xyz is the unit tangent along +U, w is the handedness ±1, so bitangent = cross(normal, tangent.xyz) × tangent.w
UV	float2	40	The material UV. TEXCOORD_0 from the GLB
Lightmap UV	float2	48	The baked-lighting atlas coordinate. Never read from the GLB — see below

The tangent is read from the GLB’s TANGENT attribute when the exporter wrote one (Blender does) and derived deterministically from UV deltas when it did not, so normal mapping never depends on the exporter’s settings. The loader reads TEXCOORD_0 and nothing else: a GLB’s own TEXCOORD_1 is ignored, because a lightmap UV has to be a non-overlapping atlas parameterization and no exporter can be trusted to have produced one.

The lightmap UV instead arrives from a companion blob the compiler writes next to the mesh — <mesh>.glb.lightmapUv, magic DHLU — generated by the compile-time atlas packer whenever a map turns on lighting.lightmap. The mesh bytes are untouched, so the same GLB loads identically with or without a bake; the loader simply fills the Uv2 lane from the companion when one is present and leaves it zero when it is not. The companion stores its vertex count, and a stale one (the mesh changed but the bake did not) is a hard load error naming the pallet to rebuild rather than a silently misparameterized map.

Loaded assets are cached keyed by their CRC32, so hot rebuilds only re-upload what actually changed.

On-disk cache

Derived, rebuildable artifacts (such as hashed mipmap chains) are cached on disk under %LocalAppData%\DigitalHeaven\Engine\cache — never the install directory or the user’s workspace, so the folder is always safe to wipe. Mip chains live in a mipmaps/ subfolder, each file (<hash>.dhmip) keyed by a content hash (SHA-256 of the source pixels, dimensions and filter), so an unchanged texture reuses its chain across runs and two identical textures share one entry. The Storage section of the in-game settings screen shows the current cache size, offers a Clear cache button (which removes the cache contents only), and exposes a Mipmap cache toggle (client.cache.mipmaps, on by default).

The toggle gates only whether chains are persisted to (and read from) disk — it never affects correctness. With it off, chains are still generated in memory each load, so textures are always mipped; they just aren’t written to disk. Turning it back on takes effect immediately (the change is observed live).

The core pallet

The engine’s default content is consolidated into a single core pallet — the debug map and its Source-style dev textures — authored under Engine/content/core/ and compiled to Engine/content/core.pallet. It is loaded at runtime: the server installs the map’s colliders and spawns while the client renders the textured GLB. If the compiled pallet is missing, the engine falls back to a procedural gray-box map so the game always boots.

Renderer Textures

Set 1 is a per-material descriptor carrying all five texture channels as combined image samplers, in the fixed order baseColor, normal, metallicRoughness, occlusion, emissive. Every binding is always filled: a channel the material did not bind gets a 1×1 neutral default — opaque white (uploaded sRGB) for the four color/mask slots, and a flat (128, 128, 255) tangent-space normal (uploaded linear) for the normal slot — so a material that declares only a base color shades exactly as it did before PBR landed, with no shader permutations. A bitmask on the material tells the shader which channels are real, so the normal-map path is skipped entirely rather than multiplied by a no-op.

Who retires a descriptor set, and when

The set-1 pool is finite — TextureRegistry.MaxMaterialSets slots, one per live material — and a client that server-hops all session loads and unloads maps over and over. So the pool needs a rule that reclaims every slot, and the rule is: a material set is retired by whoever created it, on that owner’s teardown.

• The registry allocates; it does not own. CreateMaterialSet hands back a set and tracks it (so a mip upgrade can rebuild it and a mipmaps toggle can rewrite it), but the caller is on the hook for ReleaseMaterialSet. For a map’s materials that caller is the render scene, which releases every set it asked for in Dispose — first, before anything those sets bind is destroyed.
• Texture disposal is only a safety net. When a texture dies, every set still bound to it is retired too, because a descriptor may not outlive what it points at. That is a correctness backstop, not the accounting rule — and it cannot be, because a material that authors no texture channels binds only the registry’s neutral 1×1 defaults, which outlive every map. Nothing about such a set ever dies, so nothing would ever reclaim its slot: one leaked slot per all-default material per map load, until the pool ran dry mid-session.
• The two paths are idempotent against each other. A scene releases everything it created without caring which of its textures happened to go first, and a set a texture teardown already took is silently accepted rather than double-freed.
• Retirement is deferred, never immediate. A handle recorded into a command buffer must outlive that buffer’s execution, so a retired set ages out over frames-in-flight + 1 render-thread pumps before vkFreeDescriptorSets gets it. The same rule the replaced image and view of a mip upgrade follow.

The bookkeeping lives in MaterialSetLedger, deliberately split out of the registry with no device and no Vulkan calls in it, so the load → unload → load slot accounting is covered by headless tests rather than by reading the code and hoping.

The push-constant budget is full

Per-draw material state rides push constants, and the opaque layout now uses exactly 128 bytes — a 64-byte model matrix in the vertex stage plus 64 bytes of material parameters in the fragment stage. 128 bytes is the Vulkan guaranteed minimum for maxPushConstantsSize, so this is the portable ceiling rather than a device-specific one, and a unit test asserts the total.

The practical consequence for future work: there is no room left for another per-draw field. Anything new has to move into a uniform buffer indexed by a handle passed through the existing bytes, not appended to the push block.

Mipmapping

Textures are sampled trilinearly with anisotropic filtering — the shared sampler uses LINEAR mipmap mode across the full LOD range, and enables anisotropy (up to min(deviceLimit, 8)×) wherever the physical device advertises the samplerAnisotropy feature — so surfaces no longer shimmer or alias into noise at distance or at grazing angles (the flat map’s grid was the worst offender). Where the device lacks the feature the sampler falls back to trilinear-only. Mip chains are produced by a single shared, dependency-free CPU downsampler, DigitalHeaven.Core.Imaging.MipmapGenerator, which box-filters RGBA8 pixels down to a 1×1 level (floor(log2(max(w,h))) + 1 levels). Writing the algorithm once means the build-time compiler and the runtime engine produce identical chains.

Generation is preferably build-time, with a hybrid, threaded runtime path as the fallback — either way a frame never stalls:

1. Build time (pallet path). When a texture is compiled into a .pallet, the compiler generates its full mip chain and stores the reduced levels (level 1 downward) in a companion MipContainer blob at the texture’s path plus .mipchain. At load the engine reads that blob, prepends the decoded base level, and uploads the whole chain in one shot — no runtime generation, no threaded swap. The base level is never duplicated on disk (it lives in the texture blob) and the companion is distinct from the runtime disk cache’s .dhmip files.
2. Runtime (fallback). Textures without a precomputed companion — the built-in white/checker defaults, or images a game mod downloads at runtime — upload only the base level first (drawable immediately, crisp and un-mipped), while a background worker builds the full chain (reading it from the on-disk cache when present, otherwise generating and persisting it, subject to the mipmap-cache toggle above). A once-per-frame render-thread pump uploads each finished chain and swaps it into the live texture in place: because the draw list holds the same texture handle, the swap needs no resubmission, and the replaced GPU image is retired only after the frames that might still reference it have completed.

Mipmaps on/off (render-time)

The Graphics section of the settings screen exposes a Mipmaps toggle (client.display.mipmaps, on by default) — a separate master switch for whether mip chains are sampled at all, distinct from the Mipmap cache toggle under Storage (client.cache.mipmaps), which only governs on-disk persistence of generated chains. The two are orthogonal: the cache toggle decides whether a chain is reused across runs, this one decides whether it is used for rendering.

The switch is implemented as a pure sampler swap, so it applies live — no relaunch, and no map reload or player respawn. The texture registry owns two samplers: the trilinear/anisotropic one across the full LOD range (mipmaps on) and an otherwise-identical one clamped to the base level (MaxLod = 0, anisotropy off — mipmaps off). A texture’s set-1 descriptor binds whichever the ambient MipmapPolicy selects. Chains are always generated regardless of the switch, so toggling never re-uploads an image: on a change the registry waits for device idle (a rare, user-initiated settings event) and rewrites every live texture’s descriptor set — including the fallback white texture — to point the selected sampler at the texture’s current image view. Turning mipmaps off makes distant surfaces crisper but prone to shimmer/aliasing; leaving them on is the default.

glTF materials remain unsupported by design. DigitalHeaven pallets are the material system; the renderer takes its textures from pallet materials and map slots, not from material data embedded in a GLB.

Audio

DigitalHeaven.Engine.Audio is a hand-rolled audio stack built on two vendored native libraries:

• miniaudio owns the output device and a node-graph mixer running at a fixed 48 kHz.
• Steam Audio (phonon) provides per-source binaural HRTF 3D spatialization, wired in as a DSP node. Spatialization parameters are handed to the audio thread through an atomic swap, so the game thread never blocks the mixer.

The milestone ships the device and mixer, 2D sound playback, and one 3D spatialized source.

The engine is device-less; an output pumps it

The ma_engine is always initialized with noDevice. It holds the node graph, every voice, every gain and the Steam Audio wiring, and it never talks to hardware. Sound comes out because a separate ma_device is attached to it and calls ma_engine_read_pcm_frames from its data callback.

That split is the whole reason a device change is survivable. Replacing an output means tearing down a ma_device and initializing another one; the engine underneath is not touched, so nothing audible is reset — playing voices keep their position, master/SFX/UI gains, per-voice volume and pitch, 3D source positions, the listener transform and the HRTF state all carry straight across. Sounds do not restart, and nothing re-loads.

Three shapes exist, chosen by AudioOutputMode:

Mode	What pumps the mixer
Default	A supervised real device that follows the system default. The engine uses this.
Silent	miniaudio’s null backend — headless, no hardware, real-time.
Manual	Nothing; the caller pumps ReadInterleaved for deterministic offline tests.

Following the default output device

The engine plays on whatever the OS says the default output is, and moves when that moves — headphones plugged in, a USB interface connected or removed, the output switched in the OS mixer. No relaunch, and no silence that needs one. There is no device picker: following the default is the only behavior.

Two signals feed one decision, because neither alone is enough:

• A poll of the default device (client.audio.device.pollInterval, 0.5 s) is what notices the default moving to a different device while ours is still perfectly healthy. No backend raises a notification for that — a reroute notification fires when your device is taken away, not when a different one becomes preferred — so it is polled with ma_context_get_device_info(…, NULL, …), whose returned id blob is the identity that gets compared.
• A device notification callback (ma_device_notification_type_stopped) catches a device that is invalidated or removed, immediately, without waiting out a poll interval.

miniaudio’s own WASAPI automatic stream routing is disabled (wasapi.noAutoStreamRouting). It half-solves the problem — it reroutes on a default change but discards the result, and when the last device disappears it stops the device and never restarts it, which is exactly the “goes silent until you relaunch” failure — and two authorities racing to reopen one device is worse than one authority that always wins. Backends that reroute a live device internally (CoreAudio) are still absorbed: the output reports its identity by reading the device live, so a device that was rerouted underneath us already matches the new default and no redundant switch happens.

Both signals arm one small state machine (AudioOutputSupervisor), which runs on its own thread and never on the frame thread — opening a WASAPI endpoint takes tens of milliseconds and would otherwise hitch the render. Its rules:

• Debounce. A change must settle for client.audio.device.debounce (0.25 s) before it is acted on, and each further change restarts that timer. A burst — an unplug and replug, or a dock that enumerates in stages — therefore costs exactly one reinitialization, never one per notification. Opening and closing only ever happen on that one thread, so overlapping reinitializations are impossible by construction.
• A failure never costs sound that is already playing. If the new default will not open, playback stays on the device it is already on and the attempt is retried on a doubling wait (client.audio.device.retryInterval → client.audio.device.maxRetryInterval). Only when there is nothing playing does it fall back to silence.
• No device at all is a normal state. With no playback device — including at startup — the mixer runs on the null backend: still real-time, so voices finish and loops loop and nothing piles up to blare later, just inaudible. When a device appears the poll picks it up and hardware resumes on its own. The engine never crashes, never blocks and never gives up.
• No click. Every output fades in over client.audio.device.fade (8 ms), and a planned teardown fades out first, so neither end of a switch snaps. An unplanned loss cannot be faded — the hardware is already gone — but nothing stale is emitted either: miniaudio pre-silences the output buffer and the callback never writes past what the engine produced.

Each real transition writes one line on the audio log channel — output: <device>, output follows the new default: <device>, no playback device — running silent, could not open <device> — …. Nothing is written for a poll that found no change, so this is a line per transition, not per frame. There is deliberately no toast: following the default is meant to be invisible, and a user who just unplugged their headphones does not need to be told.

Portability: the mechanism is backend-neutral. The default-device query and the notification callback are core miniaudio, so macOS (CoreAudio) and Linux (ALSA/PulseAudio) follow the default by the same path — with CoreAudio additionally doing some of the work itself, which the live identity read absorbs. Only wasapi.noAutoStreamRouting is Windows-specific, and it is ignored elsewhere. Note that only win-x64 miniaudio is prebuilt in the repo; other platforms need the recipe in native/build-miniaudio.md, which now also builds the new dh_output_* shims.

The reconnection logic is testable headlessly: the device layer sits behind IAudioOutputBackend / IAudioOutput (the same split as ITickWaiter under TickPacer), so debounce, coalescing, backoff, the silent fallback and the resume are all exercised against a fake with no hardware.

Volume controls

One-shots route through one of a small set of volume buses (categories). The mixer’s Master gain sits over the whole mix; under it, each category applies its own multiplier to the fire-and-forget one-shots routed to it. The Audio section of the settings screen exposes one live slider per bus, all stored as linear 0..1 fractions and shown as percentages:

• Master volume (client.audio.master) scales the whole mix via the mixer’s master gain.
• SFX volume (client.audio.sfx) attenuates the SFX bus — gameplay one-shots (footsteps, landings, fall pain, the spawn gasp) and the fall-wind loop — on top of the master volume.
• UI / Menu volume (client.audio.ui) attenuates the UI bus — interface/menu one-shots (the noclip toggle blip today; future menu sounds) — on top of the master volume.

The bus is chosen per call: PlayOneShot2D(path, volume, pitch, AudioCategory category = AudioCategory.Sfx) scales the caller’s volume by that category’s multiplier (SfxVolume for Sfx, UiVolume for Ui) inside the audio system. The category-less overloads default to SFX, so gameplay callers are unchanged. None of the buses affect loaded Sounds or 3D sources.

The fall-wind loop (FallWind) is an SFX sound but plays through a loaded looping Sound rather than PlayOneShot2D, so it applies the SFX multiplier itself: it reads IAudioSystem.SfxVolume and scales the loop’s live gain by it every frame, so moving the SFX slider mid-fall quiets the wind at once.

The noclip blip is the one UI sound today. It is routed through the UI bus at a fixed 0.5 base gain (ClientHost.NoclipVolume) so the debug toggle sits under gameplay — the base drop stacks with the sliders: effective = NoclipVolume × UiVolume × Master.

All buses apply live — the host pushes the client.audio.* preferences onto the audio system every frame, so a slider or a loaded profile takes effect with no restart.

Output device preferences

The device-following timings are pushed the same way and are retunable from the console with no relaunch. They are not on the settings screen; the defaults are meant to be right.

Preference	Default	What it does
client.audio.device.pollInterval	0.5	Seconds between checks of the system default output device. Bounds how long a switch takes to begin.
client.audio.device.debounce	0.25	Seconds a detected change must settle before it is acted on; restarted by each further change.
client.audio.device.retryInterval	1	First wait before a failed open is retried; doubles per consecutive failure.
client.audio.device.maxRetryInterval	8	Ceiling the doubling retry wait is clamped to.
client.audio.device.fade	0.008	Gain ramp at each end of a switch, so a device that starts mid-waveform does not click.

Gameplay sound files (footsteps, impacts) ship as loose files beside the core pallet at content/core/sounds/ — they are not yet bundled into the .pallet blob. SoundLibrary resolves them from AppContext.BaseDirectory/content/core/sounds at runtime; on a no-content build they resolve to nothing and audio degrades to silence.

Movement audio timing

Local-player footsteps, landings and jumps (MovementAudio) and the two speed-driven loops (FallWind, SurfaceScrape) are driven from the predicted motion, and stepped once per simulation tick inside the client’s fixed-tick loop — never once per rendered frame. Movement edges can live and die inside a single 60 Hz tick, so sampling them at frame rate would alias them away non-deterministically.

The landing thud keys off an explicit CharacterFlags.Landed edge the mover raises on the tick of fresh ground contact, not the persistent Grounded flag. This matters for a platform bounce: a landing immediately consumed by a buffered jump (Space tapped within the jump buffer of touchdown) never latches Grounded — the mover clears it in the same tick it applies the jump impulse — but it still thuds. Landed is transient and never networked (SnapshotCodec.ToReplicaFlags maps only the persistent stance flags); the local predictor recomputes it during reconciliation replay.

A landing hard enough to hurt layers a fall-pain grunt on top of the impact thud. The descent speed at/above which a fall hurts is the live client.audio.fallPainSpeed preference (default 11.0 m/s ≈ a 4 m drop; range 4–25) — a genuine fall, never an ordinary hop. The same threshold drives the fall-impact view punch’s hard bonus (see Camera & View Effects), so the grunt and the extra snap fire together. The host pushes it into the movement audio each frame, so a console edit retunes the “this hurt” line without a restart.

Noclip flight has its own wind curve. While the local pawn is flying, FallWind swaps the target gain to a second, direction-agnostic curve driven by the total velocity magnitude rather than descent speed — so climbing, strafing and diving at the same speed all rush the same amount — and tops out at a lower ceiling so free flight never roars like a real long fall. Both curves feed the same smoothed envelope, so toggling noclip mid-flight eases between them instead of popping. Three live preferences tune it:

Preference	Default	Meaning
client.audio.noclipWind.startSpeed	10 m/s	Total speed below which flight is silent (just under the default world.noclipSpeed cruise of 12, so drifting is near-silent).
client.audio.noclipWind.fullSpeed	30 m/s	Total speed at which the flight wind reaches its ceiling (full-throttle sprint flight: 12 × 2.5).
client.audio.noclipWind.maxGain	0.35	Ceiling gain, well below the falling wind’s own 0.62.

The falling curve has the matching three, so the same knobs exist on both sides:

Preference	Default	Meaning
client.audio.fallWind.startSpeed	12 m/s	Airborne speed below which falling is silent. This is the dial for how eager the wind is: because the signal is a magnitude, a sprint-jump leaves the ground at sqrt(10.16² + 5.08²) = 11.36 m/s, and 12 is what keeps takeoff quiet. Deliberately equal to client.audio.surfaceScrape.startSpeed, but not to client.camera.speedFov.startSpeed — the view opens earlier (at 9) by design, because a sprint is meant to be seen and not heard.
client.audio.fallWind.fullSpeed	24 m/s	Airborne speed at which the fall wind reaches its ceiling.
client.audio.fallWind.maxGain	0.62	Ceiling gain for a genuine fall.

Sliding across a surface scrapes. The grounded twin of the fall wind, SurfaceScrape, is a second looping voice driven off the same SpeedRush signal — which on a grounded tick is the horizontal ground speed. It is silent through ordinary locomotion (the default walk is 5.08 m/s and a sprint 10.16 m/s) and opens up only once something other than legs is moving the player: a slick slope, a preserved bunny hop, a launcher. Airborne and noclip ticks target silence, so leaving the ground is a clean handover to the fall wind rather than two loops layered. Both the gain and the pitch ride the one ramp, so a faster slide is louder and brighter — that is what carries a single sample across the whole speed range.

Preference	Default	Meaning
client.audio.surfaceScrape.startSpeed	12 m/s	Ground speed below which sliding is silent. The dial for what counts as “sliding” rather than “running” — above the 10.16 m/s sprint by design, and deliberately equal to client.audio.fallWind.startSpeed so the grounded rush and the airborne one open at one number.
client.audio.surfaceScrape.fullSpeed	18 m/s	Ground speed at which the scrape reaches its ceiling gain and top pitch.
client.audio.surfaceScrape.maxGain	0.45	Ceiling gain — under the fall wind’s 0.62, because the scrape is a texture beneath the footsteps and impacts.
client.audio.surfaceScrape.minPitch	0.85	Pitch multiplier at the deadzone edge: a low, heavy drag.
client.audio.surfaceScrape.maxPitch	1.25	Pitch multiplier at the full-gain speed: brighter and more abrasive.

One sample, every surface — for now. The mover already resolves the grip of the dh.material underfoot (surfaceprop) each tick, so a per-surface scrape is the natural eventual hook, but it is not wired: the repo ships no per-surface scrape assets to select between, and the mover keeps surfaceFriction as a tick-local, so surfacing it to the client would mean widening the replicated character-motion struct for a cosmetic sound. When scrape assets per surface exist, the selection belongs beside the footstep-variant selection in MovementAudio and the friction plumbing can be paid for once, for both. The scrape sample itself is synthesized, not imported — Engine/scripts/make-surface-scrape.ps1 generates the seamless loop deterministically (see Engine/content/core/sounds/PROVENANCE.md), so unlike the HL2 placeholders it carries no third-party licensing.

The client reads the noclip state locally from the predictor (ClientPredictor.PredictedNoclip) — the effect is purely cosmetic and adds no wire state.

The wind does not compute its own speed. Both curves are fed the shared SpeedRush signal — the single scalar the host reduces each tick’s predicted motion to (the full velocity magnitude whenever the pawn is off the ground, whether falling or flying; the horizontal ground speed while grounded) — and both are that signal through the shared SpeedRush.Ramp deadzone/clamp, scaled by their own ceiling. The speed field of view is handed the very same float on the very same tick, so the rush you hear and the rush you see cannot drift apart. See Camera & View Effects.

The client.debugSoundCues convar toggles the SoundCueOverlay, a rolling top-right panel listing recent audio cues — kind, computed volume and pitch, age, and whether each played or was suppressed (a touchdown too soft to clear the audible floor is logged as a suppressed landing). Its landing readout shows each measured descent against the live roll / pain / max thresholds, making it the tuning companion for client.camera.* and client.audio.fallPainSpeed. It is a developer tool for diagnosing audio-timing issues, off by default.

Native libraries

The native binaries are vendored and stored via git-LFS (contributors need git-LFS installed to check them out). A per-RID resolver (NativeAudioResolver, installed with NativeLibrary.SetDllImportResolver) maps the platform-neutral miniaudio / phonon P/Invoke names to the right file at runtime, so no DllImport hard-codes a filename. miniaudio is currently built for win-x64 only; the Linux/macOS build recipe lives in Engine/DigitalHeaven.Engine.Audio/native/build-miniaudio.md.

Licensing: miniaudio is public-domain / MIT-0, and Steam Audio is Apache-2.0. See the third-party notices under Engine/DigitalHeaven.Engine.Audio/.

Deferred

Ambisonics, occlusion/reverb, GPU acceleration (TrueAudio Next), streaming, and a formal dh.sound asset type are planned but not yet implemented.