Shading & Texturing in Unreal Engine

Overview

As a starting texturing and shading strategy, it's important we have a clear understanding of the available approaches. At one end, you have unique, non-tiling macro textures, baked 0-to-1 maps. At the other end, you have layered materials driven by tiling micro textures that are blended at runtime, offering reuse across many assets which are often modular in nature. Both have legitimate roles in production, and in my opinion the most effective strategy is a hybrid of the two.

A well-architected base shader supports all of the following simultaneously:

  • Unique macro textures for baked, asset-specific detail

  • A layered micro texture system for tiling, reusable surface materials

  • A tinting workflow for introducing variation without additional texture lookups

Constraints

The guiding principle is straightforward. The cheapest form of variation is one that requires no additional memory. A well designed tinting workflow gets enormous mileage out of simple math in the material graph, and should be the first go-to tool before using extra texture sets.

Before any of the above decisions are made, two constraints must be established at the project level:

  • Texel density standards. Establishing a consistent texel density from the outset is critical. Without it, artists will make incompatible decisions across the project that are costly to fix.

  • Layer budget as a design constraint. The number of layers an artist can blend in a layered material should be treated as a deliberate production decision, not an afterthought. Because blend masks are stored in up to four channels (RGBA), four layers is a sensible upper bound.

Tinting Workflow

Tinting modifies material attributes, primarily base color, using simple blend operations, without any additional texture samples. Two modes are worth knowing:

  • Multiplicative tinting. The cheapest option. The tint color is multiplied against the base color texture. This restricts results to values at or below the tint color. You cannot tint toward white.

  • Overlay Tinting. Slightly more expensive, but allows the full range from black through the tint color to white. This is generally preferable when you need full artistic range.

For overlay tinting to work correctly, the base color texture must be authored with a luminance value biased around 50% gray in any region intended to receive a tint. This means the texture should sit near the midpoint of the sRGB range in those areas. Color information can absolutely be present, as this is about luminance, not hue. The effect is similar to using PhotoShop. At 50% gray, the overlay output exactly equals the tint color. Lighter areas blend the tint toward white, and darker areas blend it toward black. This gives you full directional control over the tinting result from a single input color.

Setting up your DCC tooling to preview this behavior correctly is well worth the investment. For example, configuring Substance Painter with a custom tinting preview so artists can audition tint colors while texturing. Happy to expand on the specifics of this setup separately.

Virtual Texturing & Compression

Two rendering technologies have meaningfully influenced texture pipeline decisions: virtual texturing and block compression formats. Understanding both is necessary before making informed layout decisions.

Virtual Texturing

The traditional approach to texture memory relies on fully pre-generated mip chains. A 2048×2048 texture must be stored and loaded into memory at every mip level: 1024, 512, 256, and so on, effectively doubling the memory footprint. If a texture is referenced by a material, the full mip chain must be loaded into memory. At scale, across a large tiling texture library, this creates significant memory pressure and cache churn.

Streaming Virtual Textures (SVTs) change this fundamentally. Rather than loading full mip chains, the SVT system uses an indirect page table to determine which mip tiles are actually needed for what is currently visible on screen. Only those tiles are loaded. The tradeoff is real: the page table lookup adds GPU cost, and UV inputs that diverge significantly across a draw call limit opportunities to reuse lookup results.

In collaboration with a rendering team, after extensive low level memory and performance analysis on PS5, we found that SVTs outperformed conventionally mipmapped textures for layered materials, despite the higher instruction count. We believe this also generalizes to modern PC GPU architectures, and the reason is cache behavior. Conventional mipmapping caused significant thrashing in the GPU's L1 and L2 caches as texture data was loaded and evicted to service shader calculations. SVTs, by loading only what the screen requires, allowed texture memory to be managed far more efficiently at the hardware level.

This has a direct implication for material architecture. Using virtual textures for both macro and micro (tiling) textures is recommended. For macro textures sharing a UV0 input, page table lookups can be reused. In Unreal Engine terms, textures sharing the same UV input are placed on the same Virtual Texture Stack. For micro textures with divergent UV inputs, the lookup cost is real, but the memory efficiency gains seem to more than compensate in practice.

This finding does not contradict the general principle that **fewer texture lookups are preferable**. That remains true, and it directly informs the layer budget constraint above. However, when layered tiling textures are in use, virtual texturing is the right storage strategy, even though it adds overhead compared to a single texture lookup.

Block Compression Formats

Choosing the right compression format for each texture is one of the most impactful decisions in texture pipeline design. Note: "bpp" = bits per pixel. BC4's 16-bit depth refers to the precision of its single data channel, not bits per pixel.

BC1 (DXT1) - 4 bpp

  • Channel bit depth: R=5, G=6, B=5

  • Example uses: Base Color, ORM

  • Can be sRGB or Linear

BC4 - 4 bpp

  • Single channel, 16-bit depth

  • Example uses: Alpha, Height, Displacement

  • Always Linear

BC3 (DXT5) - 8 bpp

  • Channel bit depth: R=5, G=6, B=5, A=16 (BC1 + BC4 block)

  • Example uses: Masks, Base Color+Alpha, ORM+Height

  • A channel has full 16-bit precision

BC5 - 8 bpp

  • Channel bit depth: R=16, G=16 (two BC4 blocks)

  • Example uses: Normal Maps

  • B component is reconstructed automatically by the texture sampler

BC7 - 8 bpp

  • Channel bit depth: R=8, G=8, B=8, A=8

  • Example uses: Masks (with caveats, see below)

  • Can be sRGB or Linear

Key observations on compression:

  • BC5 is the correct format for normal maps. It stores X and Y normal components in two independent 16-bit BC4 blocks. I don't advise packing normal XY into BC1 or BC3 RGB channels. At 5 and 6 bits of depth respectively, you will see banding artifacts on surfaces with non-trivial normal angles.

  • BC3's alpha block provides true 16-bit depth. This matters for height/displacement data, which requires high precision to avoid visible banding. Height should always go in a BC3 alpha channel or a dedicated BC4 texture.

  • Avoid BC7 for blend masks. Its channels share index bits and are not truly independent. This is fine for simple linear blends but can produce subtle artifacts in more complex blending logic. BC7 is also always 8 bpp regardless of how many channels you use. There is no memory savings in using it for a single channel mask.

  • AO in the red channel of BC1 (5 bits) is acceptable. Roughness benefits more from the green channel's extra bit (6 bits), which aligns with how BC1/DXT compression historically prioritizes the green channel to reflect human visual sensitivity.

Material Layer System & Strata

Epic's Material Layer Assets and Material Layer Blend Assets are no longer recommended for production use. Based on direct communication with Epic engineers, this infrastructure is not being actively developed going forward. More critically, Material Layer Assets have known incompatibilities with virtual textures that cause texture lookup bloat, a significant problem given the SVT strategy described above.

Strata (previously called Substrate) is the intended path forward for material architecture in Unreal Engine. It is not the same thing as the Material Layer Asset system. Strata is actively developed, not abandoned. If you are evaluating long-term material architecture, Strata is worth tracking closely.

Recommended Approach: Material Functions

Implement your layered system directly in the base material using Material Functions. This gives you full control over the layer stack, better VT compatibility, and no dependency on infrastructure that isn't being maintained.

For passing intermediate data through the layer stack before writing final outputs, two options:

  • Attribute hijacking (no C++ required). On Default Lit, some material attribute slots, such as Subsurface Color and Anisotropy, are unused for standard PBR surfaces and can be temporarily repurposed to carry custom data through layer blending logic. This is a pragmatic intermediate approach that requires no plugin or C++ work, though it constrains your shading model.

  • Custom Material Attribute plugin (requires C++ implementation). Defines typed attributes explicitly, removing the dependency on hijacked slots. The cleaner long-term solution if your project has the technical resources.

Note: This attribute hijacking advice is based on pre-Strata workflows. Strata may change how intermediate attribute passing works. If you are working in or migrating to Strata, first verify that this is a sensible approach.

Macro Texture Layout

Macro textures are unique, 0-to-1 UV baked textures applied to specific assets. They provide asset level surface detail and serve as the base layer in a hybrid system.

Base Color

  • Channels: RGB (+ optional A for Opacity Mask)

  • Compression: BC1 / BC3 (with alpha)

Note: Only include an Opacity Mask when genuinely necessary. With Nanite, masked material evaluation is significantly more expensive than opaque. In most cases it is preferable to invest in additional geometry authored as contiguous, cleanly clustering meshes without floating pieces or large numbers of discrete UV islands.

Normal

  • Channels: RG = Normal XY (Z reconstructed by sampler)

  • Compression: BC5

Note: With sufficient geometric density and Nanite-friendly mesh authoring, it may be possible to forgo a macro normal map entirely. "Nanite-friendly" means contiguous geometry that clusters and streams efficiently and avoids floating pieces. This is worth exploring but should be validated against your specific assets before relying on it in production.

ORM (Occlusion / Roughness / Metallic)

  • Channels: R = AO, G = Roughness, B = Metallic, A = Displacement (optional)

  • Compression: Masks (no SRGB), BC1 (no displacement) / BC3 (with displacement), always Linear

Notes:

  • While Lumen provides GI and indirect shadowing, material AO can still contribute meaningful micro-occlusion detail. Whether it's worth the channel depends on your visual requirements.

  • Nanite runtime displacement should be used with caution on static mesh assets as of UE 5.6. Nanite landscape tessellation now has production ready distance-based rasterization optimizations, but it is not confirmed whether equivalent optimizations apply to static meshes.

Blend Mask

  • Channels: RGB (+ optional A)

  • Compression: Masks (no SRGB), BC1 / BC3 (with alpha), always Linear

Note: Only use the alpha channel if you need BC4 precision for a specific mask. Do not use BC7. For unique baked assets, blend mask channels drive tinting or blend tiling micro layers (dirt, wear) over the base bake. For fully layered assets, the blend mask may be the only macro texture present.

Micro Texture Layout (Tiling)

Micro textures are tiling, reusable surface definitions blended across geometry using the blend mask above. Two layout options depending on whether height is always needed or only on specific layers.

Option 1: Height Optional Per Layer

Recommended when most layers use Base Color, Normal, and ORM, and height is only needed for specific layers.

Texture A - Base Color + Normal.R

  • RGB = Base Color, A = Normal X component

  • Compression: BC3 (sRGB on RGB, Linear on A)

Texture B - ORM + Normal.G

  • R = AO, G = Roughness, B = Metallic, A = Normal Y component

  • Compression: BC3 (Linear, no sRGB)

Texture C — Height/Displacement (optional)

  • Single channel

  • Compression: BC4

  • Bias: Start with 0.5. Adjust based on use case as height blending and Nanite displacement may have different optimal biases.

The Normal X and Y components are stored in the two BC3 alpha blocks, each a full 16-bit BC4 block. The Normal Z component must be reconstructed in the material graph from X and Y (standard sqrt(1 - x² - y²) derivation). This gives you effectively BC5-equivalent normal precision distributed across two texture samples.

Note: 16-bit precision for height data is important. Storing height in a 5- or 6-bit RGB channel will produce visible banding. Always use a BC4 texture or BC3 alpha block.

Option 2: Height Always Present

Recommended when height-based blending or Nanite displacement is used on all or most layers.

Texture A - Base Color

  • RGB = Base Color

  • Compression: BC1 (sRGB)

Texture B - Normal

  • RG = Normal XY

  • Compression: BC5

Texture C - ORM + Height

  • R = AO, G = Roughness, B = Metallic, A = Height/Displacement

  • Compression: BC3 (Linear, no sRGB)

  • Bias: 0.5 as a starting point, adjusted per application.

The normal map gets a dedicated BC5 texture, preserving independent 16-bit precision for both X and Y components. Height is packed into the BC3 alpha block of the ORM texture, also at 16-bit precision. This layout is cleaner at the cost of one additional texture sample compared to Option 1.

Summary

  • Start with a hybrid macro/micro system. Use unique bakes where appropriate, tiling layers for modular reuse, and let tinting carry variation as far as possible before adding texture samples.

  • Establish texel density and layer budgets as project-level design decisions, not per-asset improvisation.

  • Use Streaming Virtual Textures for both macro and micro textures. Despite increased instruction cost for tiling textures, SVTs improve real-world performance by reducing GPU cache pressure. This is validated on PS5 and believed to generalize to modern PC hardware.

  • Choose compression formats deliberately. BC5 for normals. BC4 or BC3-alpha for height. BC1 for ORM without height. Avoid BC7 for blend masks. Avoid packing normal XY into BC1/BC3 RGB channels.

  • Avoid Material Layer Asset infrastructure. Implement layering directly in the base material using Material Functions. Track Strata for long-term architecture decisions.

  • For micro texture layout: use Option 1 when height is only needed on specific layers and Option 2 when height is universally required. In both cases, preserve 16-bit precision for normal components and height data.