Brotli-G: An open-source compression/decompression standard for digital assets that is compatible with GPU hardware   

The technical details of the Brotli algorithm are well documented (see resources section below). At a high level, it is a combination of the LZ77 compression algorithm and Huffman coding (similar to GZIP and Deflate).

It is well documented that Brotli performs competitively vs other standards (such as GZIP and Deflate) in all metrics: compression ratio, compression speed, and decompression speed. Comparisons with various compressions standards can be seen at Squash Compression Benchmark (quixdb.github.io)

Because of the obvious performance and compression ratio advantages over previous compression formats, Brotli has been widely adopted in web applications and content spaces. Brotli can be used to compress web application assets such as fonts, javascript, images, and much more.

• Brotli decompression is supported by over 95% of web browsers today.
• Many web application frameworks support Brotli encoding (such as NGINX, and ASP.NET).
• Some CDNs use Brotli encoding (such as Azure, and ImageKit)

Brotli has already been successful in these areas. When asset file sizes are smaller, they can be delivered over networks faster, and when compression/decompression is fast, web pages load faster and are more responsive.

As a general data compression format, Brotli may also be applied to other software ecosystems such as gaming and cloud gaming. Video games are asset-dense software applications, and many already employ some type of compression to reduce file size. The gaming experience may be improved by improving the compression and decompression performance of compressed assets.

Brotli-G: AMD modifications to Brotli

The AMD modifications to the Brotli bitstream format are focused on allowing efficient, data-parallel decompression on GPUs and multithreaded CPUs, while still maintaining comparatively high, state-of-the-art compression ratios for general-purpose data. Some of the modifications include:

• Limited the size of chunks to compress (pages) providing the ability to use multiple Compute Units (CUs) on the GPU.
• Implemented parallel Huffman substreams, providing the ability to use SIMD processing.
• Simplified the bitstream format to maximize performance on GPU.

The Brotli-G decompression algorithm has been implemented as a ‘shader’ program (written in HLSL) that runs on virtually any DirectX®-compatible graphics card. Implementing decompression in shaders takes advantage of the enormous parallel processing power available in modern GPU hardware, to enable incredibly fast decompression speeds.

Existing optimized Brotli decompression functions (CPU implementations) should be able to decompress the Brotli-G bitstream, while more optimal data-parallel implementations on hosts or accelerators can further improve performance.

One thing for developers to note is that assets that have already been compressed with Brotli cannot be decompressed with Brotli-G decompressor implementations. To take advantage of Brotli-G, assets must be recompressed using a Brotli-G compatible compressor (also supplied in the SDK).

What AMD is releasing

We are releasing a Brotli-G SDK that contains source code for CPU compression and decompression on both the CPU and the GPU. The SDK also contains source code for a CLI application that can be used for testing and measuring the performance of the Brotli-G codec. The SDK can be found in its GitHub repository.

We will also be integrating Brotli-G codecs into our Compressonator content authoring toolchain component as a supported compression format in its SDK library (coming soon).

AMD encourages collaborative, open-source development to further improve compression ratio and multi-threaded compression/decompression performance on CPU and GPU accelerators, in addition to improving integration with the general Brotli format.