fennec/planning/3D_GRAPHICS.md

3D Graphics (gfx3d)

Table of Contents

• Home
• 3D Graphics (gfx3d)
  • Table of Contents
  • Introduction
  • Structures
  • Stages

Introduction

  This system handles the rendering of 3D meshes, materials, lights, and post-processing
effects. DirectX will never have official support. If you would like to make a fork, have at it, but know that I will hold a deep disdain for you.

The graphics pipeline will have a buffer with a list of objects and their rendering data.
This will be referred to as the Object Buffer. There will be two, for both the Deferred and Forward Passes.

The buffers will be optimized by scene prediction.
This involves tracking the meshes and textures directly and indirectly used by a scene.
A callback function in the graphics system for scene loading can do this.

Materials and Lighting models will be run via a shader metaprogram to make the pipeline independent of this aspect. This allows the GPU to draw every single deferred rendered mesh in a single draw call for each stage of the renderer.

Specifications for debugging views via early breaks are included in the stages.

Links:

• https://en.wikipedia.org/wiki/Octree
• https://www.adriancourreges.com/blog/2015/11/02/gta-v-graphics-study/
• https://learnopengl.com/PBR/Lighting
• https://learnopengl.com/PBR/IBL/Diffuse-irradiance
• https://en.wikipedia.org/wiki/Schlick%27s_approximation
• https://pixelandpoly.com/ior.html
• https://developer.download.nvidia.com/SDK/10/opengl/screenshots/samples/dual_depth_peeling.html

Structures

Object Structure. The mesh is implicit data.

struct Object
{
    vec3 location, scale; // A matrix would be 64 bytes, this is instead 28 bytes
    quat rotation;
    uint32 material, lighting;
}

Objects are identified with a manually provided type and ID. Object types include:

• Static (transform does not change after creation)
• Dynamic (transform changes after creation)
• Forward Static (object is forward rendered and its transform does not change after creation)

A user should never have to specify these and should be automatically generated by the respective components.

Textures for 3D rendering are stored in various buffers with sizes of powers of 2. Ratios of 1:1
and 2:1 are allowed. The 2:1 ratio is specifically for spherical and cylindrical projection. UVs
may be transformed to use a 2:1 as if it were 1:2. Cubemaps and 3D textures may only be 1:1.

• 8-Bit R Texture 4096, 2048, 1024, 512 (8)
• 8-Bit RG Texture 4096, 2048, 1024, 512 (8)
• 8-Bit RGB Texture 4096, 2048, 1024, 512 (8)
• 8-Bit RGBA Texture 4096, 2048, 1024, 512 (8)
• 8-Bit R Cubemap 1024, 512, 256, 128 (4)
• 8-Bit RGB Cubemap 1024, 512, 256, 128 (4)

• 16-Bit HDR R Texture 4096, 2048, 1024, 512 (8)
• 16-Bit HDR RGB Texture 4096, 2048, 1024, 512 (8)
• 16-Bit HDR RGBA Texture 4096, 2048, 1024, 512 (8)
• 16-Bit HDR R Cubemap 1024, 512, 256, 128 (4)
• 16-Bit HDR RGB Cubemap 1024, 512, 256, 128 (4)

• 16-Bit Shadow Texture 4096, 2048, 1024, 512 (8)
• 16-Bit Shadow Cubemap 2048, 1024, 512, 256 (4)

• 8-Bit 3D R Texture 256, 128, 64, 32 (4)
• 8-Bit 3D RGB Texture 256, 128, 64, 32 (4)
• 16-Bit 3D HDR R Texture 256, 128, 64, 32 (4)
• 16-Bit 3D HDR RGB Texture 256, 128, 64, 32 (4)

Documentation should provide guidelines on categories of Art Assets and the resolution of textures to use.

Textures are identified by a 32-bit integer.

• 8 bits → the texture buffer
• 24 bits → the layer in the buffer

Artists should be informed on the texture structure of the engine and its limitations. However, these principles should be followed in other game engines as these are guided by what is most efficient for typical GPU hardware.

Materials are, for the most part, user-defined. Documentation should make the user aware of this. Material buffers will be a sequence of the Material Struct instances. They will at the very least contain the id of their shader.

Types of materials:

• Surface (Vertex & Fragment)
  • Opaque / Masked
  • Translucent (Forward)
• Volume (Path Traced Volumetrics / Forward)
• Light (Lighting Model)
• Post-Process

Stages

This is the set of stages for the graphics pipeline that runs every frame: Unless otherwise specified, each stage will be run on the GPU.

• BVH
  • Octree (8 Bpn, 64 bpn) [6-Layers ≈ 2.1MB]
    • Leaf Size and Tree Depth should be calculated by the scene, constraints are as follows:
      • Min Object Size
      • Max Object Size
      • Scene Center
      • Scene Edge
    • Buffer has implicit locations due to the tree having 8 children.
    • Insertions and Updates are done on the CPU
    • Nodes
      • Start Index int32
      • Object Count int32
    • Objects
      • Buffer of Object IDs grouped by Octree Node
  • Leaf Culling
    • Starting at each Octree Leaf, traverse upwards.
    • Insert Visible Leaf IDs
      • Track using atomic buffer
  • Generate the Command Buffer for Culled Mesh LODs from the Visible Leaf Buffer
    • Track counts using atomic buffers
    • To avoid double counting due to the construction of the Octree output, we have some options
      • Ignore Leaf Instances based on occurrences of the mesh in the surrounding 8 Quadtree Leaves. This would require a bias towards a specific corner of the filter.
      • Perform a preprocessing step on the CPU to erase duplicate elements and fix the buffer continuity.
      • Let the duplicates be rendered.
    • Generate the Culled Object Buffer with the respective object IDs
      • Adjust Buffer Size using the counts
      • Insert by reusing the count buffer, clipped to only contain used meshes

Debug View: Object ID, Mesh ID, LOD

• Visibility
  • Buffer (15 Bpp, 120 bpp) [1920x1080] ≈ 39.4MB
    • Depth Buffer → D24
    • Visibility Info → RGB32I
      • R = Object ID
      • G = Mesh ID
      • B = Material ID
  • Regenerate the Command Buffer for Visible Mesh LODs
  • Regenerate the Culled Object Buffer

Debug View: Visibility Buffer

• G-Buffer Pass (17 Bpp, 136 bpp) [1920x1080] ≈ 35.3MB
  • Depth - Stencil → D24_S8
    • S → used to represent the lighting model.
  • Diffuse → RGBA8
    • A → Ambient Occlusion
  • Emission → RGB8
  • Normal → RGB8
  • Specular → RGB8
    • R → Roughness
    • G → Specularity (sometimes called the Metallicness)
    • B → Index of Refraction (IOR)

Debug View: Depth, Stencil, Diffuse, Emission, Normal, Specularity

• Generate Dynamic Shadows
• Generate Dynamic Reflections (Optional)
• SSAO (Optional)
• Deferred Lighting Pass (10 Bpp, 80 bpp) [1920x1080] ≈ 2 x 16.3MB + 8.3MB ≈ 24.6MB
  • Depth Buffer → D24
  • Lighting Buffer → RGB16 w/ Mipmapping
  • Stencil Buffer $rarr; S8
  • Apply Lighting Model

Debug View: Shadows, Reflections, SSAO, Deferred Lighting

We can combine all of these into one framebuffer:

• Depth - Stencil → D24_S8
• Visibility Info → RGB32I
• Diffuse → RGBA8
• Emission → RGB8
• Normal → RGB8
• Specular → RGB8
• Lighting Buffer → RGB16 w/ Mipmapping
• One more slot left open for another

• Forward Pass
  • BVH, Same as Above
  • LOD Selection, Same as Above
  • Translucent Materials
    • Dual Depth Peeling

Debug View: Forward Mask

• Post Processing
  • Depth of Field (Optional)
    • When enabled, the Visiblity Buffer, G-Buffer, and Deferred Lighting Pass will be double layered.
    • At this point the Lighting Buffers will be Flattened
  • Bloom (Optional) → Mipmap Blurring (6Bpp, 48bpp) [1920x1080] ≈ 16.3MB
  • Tonemapping (Optional)
  • HDR Correction