Visitar URL original
Add SOG v3 loading and over-unity opacity rendering by slimbuck · Pull Request #9873 · playcanvas/engine · GitHub
Skip to content

Add SOG v3 loading and over-unity opacity rendering - #9873

Open
slimbuck wants to merge 5 commits into
playcanvas:mainfrom
slimbuck:sog-v3
Open

slimbuck wants to merge 5 commits into
playcanvas:mainfrom
slimbuck:sog-v3

Conversation

@slimbuck

Copy link
Copy Markdown
Member

Adds support for SOG v3, proposed in playcanvas/splat-transform#362, and draws gaussians whose opacity is above 1. SOG v2 and v1 loading is unchanged.

SOG v3 loading

  • Rotations are stored as a 24-bit exponential map in the alpha channels of means_l, means_u and scales. quats.webp is not fetched for v3.
  • Opacity is read through meta.opacity.codebook, so values above 1 are allowed.
  • Spherical harmonics support 4 bands (a 1536-pixel-wide palette).
  • SH palette labels are 24-bit: shN_labels.b holds bits 16–23, so palettes can exceed 65536 entries. This applies to the GLSL and WGSL SH chunks and the CPU iterator.
  • The optional blind.webp is exposed as a resource stream, readable with getBlind() in work-buffer modifiers. The CPU iterator and decompress() output opacity_excess and the blind column.
  • The extras of meta.json / lod-meta.json are exposed as resource.extras, and kept by decompress().
  • The parsers report an error for unknown SOG versions.

Over-unity opacity

Decimation merges several gaussians into one, and the merged gaussian can carry more mass than a standard gaussian with opacity 1 can hold. SOG v3 and PLY (opacity_excess column) can store that true opacity. The unified renderer draws these gaussians with a flat-topped plateau profile: an opaque core joined smoothly to a gaussian edge, with a larger footprint. The profile's mass matches the stored opacity, with a 0.4 plateau shrink.

  • An R8 work-buffer stream (splatPlateau, log2(D')/6) is added only once a resource with over-unity opacity is in the world. The copy pass solves the plateau shape once per splat.
  • Quad renderer (GLSL and WGSL), GPU-sort projector, shadow and pick:
    • All cull by the grown footprint.
    • All trim quads where the plateau profile falls to alphaClip, so coverage matches the drawn shape.
    • The projector's contribution test also counts the plateau's mass.
  • Pick and shadow materials enable GSPLAT_OVERUNITY, and refresh when streams are added to the format in place.
  • PLY: GSplatResource adds opacity_excess to the sigmoid opacity and reports hasOverUnity. The uncompressed format clamps opacity to 1 unless GSPLAT_OVERUNITY is defined, so the non-unified renderer is unchanged.
  • app.scene.gsplat.compensation = false clamps opacity to 1, for debugging and comparison.

Fix

GSplatWorkBuffer.syncWithFormat() didn't rebuild its render targets when a stream was added after getTexture() had already synced the streams. This affects any stream added at runtime on the GPU-sort path.

Testing

  • New test/scene/gsplat-sog-v3.test.mjs and test/scene/gsplat-unified/gsplat-overunity.test.mjs.
  • Checked on a synthetic two-splat scene that coverage now matches the drawn footprint:
    • Pick area went from 1854 to 5092 px on the GPU-sort path, and from 3362 to 5082 px on the quad path.
    • Shadow area went from 1094 to 3077 px.
    • The forward render didn't change.
  • A 128K-entry SH palette scene renders identically on WebGPU and WebGL2.

slimbuck and others added 4 commits October 28, 2026 09:57
SOG v3 loading (v2 unchanged):
- 24-bit exponential-map rotations read from the alpha channels of means_l, means_u and scales; quats.webp is no longer fetched for v3.
- Opacity read through meta.opacity.codebook (via the lookup texture's alpha), allowing values above 1.
- 4 SH bands (1536-pixel palette).
- Optional blind.webp exposed as a resource stream, with getBlind() in work-buffer modifiers; the CPU iterator and decompress() emit opacity_excess and the blind column.
- meta.json / lod-meta.json extras exposed as resource.extras.
- Parsers report unknown SOG versions as an error.

Opacity above 1 (decimated gaussians that carry more mass than a standard gaussian) is drawn with a flat-topped plateau profile whose mass matches the stored opacity, with a 0.4 plateau shrink. An R8 work-buffer stream (splatPlateau) is added only once a resource with over-unity opacity is in the world; the copy pass solves the plateau shape once per splat. Both the quad renderer (GLSL and WGSL) and the GPU-sort projector path grow the footprint and draw the profile. app.scene.gsplat.compensation = false forces opacity <= 1 for debugging.

Also fixes GSplatWorkBuffer.syncWithFormat() not rebuilding its render targets when a stream is added after creation and getTexture() has already synced the streams (affects any stream added at runtime on the GPU-sort path).

Not covered: shadow and pick passes draw plateau splats at the standard extent; the PLY path ignores opacity_excess.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PLY files written by decimation can carry an opacity_excess column holding the opacity above 1. GSplatResource now adds it to the sigmoid opacity in the half-float color texture and reports hasOverUnity, so the unified renderer enables the plateau stream as it does for SOG v3. The uncompressed format clamps opacity to 1 unless GSPLAT_OVERUNITY is defined, so compensation off and the non-unified renderer are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Attach the plateau stream before the frame's bake when a resource that needs it arrives, so the first bake writes it.
- Cull by the plateau's grown footprint in the quad shaders, the GPU-sort projector and the shadow cull; the projector's contribution test also counts the plateau's mass.
- Trim quads where the plateau profile falls to alphaClip, rather than scaling a plain gaussian's trim, so shadow, pick and prepass coverage matches the drawn shape.
- Enable GSPLAT_OVERUNITY on the GPU-sort pick material and the shadow materials, which also refresh when streams are added to the format in place.
- Keep meta.json extras when SOG data is decompressed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SOG v3 palettes may exceed 65536 entries: the label's third byte (shN_labels.b) holds bits 16-23. The GLSL and WGSL SH chunks and the CPU iterator decode it for v3 files; v2 and v1 still read 16-bit labels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@slimbuck
slimbuck requested a review from mvaligursky October 29, 2026 14:45
@slimbuck slimbuck self-assigned this Oct 29, 2026
@slimbuck slimbuck added the area: graphics Graphics related issue label Oct 29, 2026
@mvaligursky

Copy link
Copy Markdown
Contributor

A concern about where the plateau is stored. Decimated data will be the norm for large assets (most scenes over ~1M splats), and almost all of them will have some splats with opacity above 1. In our tests the merged opacity reaches about 20. So the plateau path will be on nearly all the time, and these become permanent costs:

  • Extra splatPlateau work buffer stream: one more render target in the copy pass, and one more texture read in the quad VS, the projector, the shadow cull and pick.
  • 9th projection cache word: entries grow from 32 B to 36 B, so most of them cross a 32-byte sector boundary. That nearly doubles the sectors the hybrid VS reads.
  • Runtime detection: _checkOverUnity, the format change, the GSPLAT_OVERUNITY toggles, and the recompile plus work buffer recreation when the first decimated LOD arrives. Once most data is decimated, this mostly just adds a hitch.

The plateau fits in the existing storage instead, with no detection at all.

1. Work buffer: store the plateau in the alpha slot

When the plateau is above 1, the copy pass clamps the stored opacity to exactly 1 (color.w = min(color.w, 1.0) after encodePlateau). So for those splats the alpha slot holds no information, and D' can go there. The copy pass writes color.w = A <= 1 ? A : D', and each format encodes that value. The writeSplat signature doesn't change.

Large format (alpha is a half): the value is written as is, with no loss.

float getOpacity() {
    // ... loads and caches cachedColor as now
    return min(cachedColor.a, 1.0);
}

// call after getOpacity(), reads the same cached data
float getPlateau() {
    return max(cachedColor.a, 1.0);
}

Compact format (alpha is the top byte of dataTransformA.w): split the byte.

  • Codes 0–191 are opacity.
  • Codes 192–255 are opacity 1, with log2(D') in (0, 2]. That covers D' ≤ 4, so true opacity up to ~31.5; anything above is clamped.
// writeSplat
float a = color.a;
uint alphaBits = a > 1.0 ?
    191u + uint(clamp(log2(a) * 32.0 + 0.5, 1.0, 64.0)) :
    uint(clamp(a, 0.0, 1.0) * 191.0 + 0.5);

// read
float getOpacity() {
    uint code = cachedTransformA.a >> 24u;
    return code > 191u ? 1.0 : float(code) / 191.0;
}

float getPlateau() {
    uint code = cachedTransformA.a >> 24u;
    return code > 191u ? exp2(float(code - 191u) / 32.0) : 1.0;
}

Precision compared with the separate R8 (maximum error, true opacity 1–30):

Separate R8 Split byte
Plateau mass ±2.2% ±2.9%
Opacity ±0.0020 ±0.0026

Both are well under the error compact already has from its 8-bit log scales, up to ±8.6% in area. If low-opacity precision matters, a sqrt mapping over the 192 opacity codes would make faint splats more precise than they are today.

Non-decimated data never produces a plateau above 1, so the decode can always be compiled in. The plateau > 1.0 branch takes the same path for every vertex and fragment of a splat. That removes:

  • the splatPlateau stream
  • _checkOverUnity, _enableOverUnity and _compensationVersion
  • the GSPLAT_OVERUNITY toggles
  • the shadow material reconfiguration

compensation = false becomes a clamp in the copy pass. The clamp in the non-unified uncompressed.js stays.

One detail: in the large format, alpha lives in dataColor, which the color-only (SH update) pass rewrites, so that pass needs D' too. Either keep alpha with a color write mask in the color-only pass, or replace the 24-step bisection with a cheaper solve. Compact isn't affected, because its alpha isn't in dataColor.

2. Projection cache: stay at 8 words

The plateau can't share the cached alpha slot. That alpha includes the AA compensation and modifySplatColor, so plateau splats can be below 1 there. But v2 is redundant. The projector writes:

v1 = l1 * dir;
v2 = l2 * vec2f(dir.y, -dir.x);

So v2 is always v1 rotated by 90° and scaled by l2 / l1. Store that ratio and the plateau in v2's word:

// projector
projCache[base + 5u] = pack2x16float(vec2f(l2 / max(l1, 1e-6), plateau));

// hybrid VS
let ratioPlateau = unpack2x16float(projCache[base + 5u]);
let v2 = ratioPlateau.x * vec2f(v1.y, -v1.x);
let plateau = ratioPlateau.y;

The cost is one multiply, and v2 stays exactly perpendicular to v1. The same change applies to XR (word 6) and pick (word 5). Color, alpha and the user varyings stay as they are, and the stride stays at 32 B.

syncWithFormat() fix

I've extracted the syncWithFormat() fix into #9876 so it can land on its own, with a test that adds a stream through addExtraStreams(). It can be dropped from this PR once that merges.

mvaligursky added a commit that referenced this pull request Oct 29, 2026
…at runtime (#9876)

GSplatWorkBuffer.syncWithFormat() compared the streams' format version before and after its own
sync. GSplatStreams.getTexture() also syncs, so when a texture was fetched by name after a stream
was added, the streams were already up to date and the render targets were never rebuilt. The
new stream then had a texture but no attachment, and the copy pass never wrote it.

Track the version the render targets were built for and rebuild whenever it is behind.

Extracted from #9873.

Co-authored-by: Martin Valigursky <mvaligursky@snapchat.com>
Co-authored-by: Donovan Hutchence <slimbuck7@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: graphics Graphics related issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants