Brotli-G: An open-source compression/decompression standard for digital assets that is compatible with GPU hardware   

Introduction and background

Brotli-G is based on the well-established Brotli lossless compression algorithm standard maintained by the IETF (also known as RFC7932). Brotli was first released in 2013 and the open-source specification was developed and generalized between 2013 and 2016. It was developed by Google Research as an open successor to the GZIP and Deflate algorithms, with similar performance (to Deflate) but a better compression ratio.

The original use case was to make websites load faster by compressing content used by web servers and content delivery networks. Brotli is useful as a general compression scheme for many types of data/content/assets. The ‘intended audience’ as defined in the specification is characterized as follows:

This specification is intended for use by software implementers to compress data into and/or decompress data from the Brotli format.

Existing ecosystem for Brotli

Brotli browser acceptance
From https://caniuse.com/?search=brotli

The technical details of the Brotli algorithm are well documented (see resources section below). At a high level, it is a combination of the LZ77 compression algorithm and Huffman coding (similar to GZIP and Deflate).

It is well documented that Brotli performs competitively vs other standards (such as GZIP and Deflate) in all metrics: compression ratio, compression speed, and decompression speed. Comparisons with various compressions standards can be seen at Squash Compression Benchmark (quixdb.github.io)

Because of the obvious performance and compression ratio advantages over previous compression formats, Brotli has been widely adopted in web applications and content spaces. Brotli can be used to compress web application assets such as fonts, javascript, images, and much more.

Brotli has already been successful in these areas. When asset file sizes are smaller, they can be delivered over networks faster, and when compression/decompression is fast, web pages load faster and are more responsive.

As a general data compression format, Brotli may also be applied to other software ecosystems such as gaming and cloud gaming. Video games are asset-dense software applications, and many already employ some type of compression to reduce file size. The gaming experience may be improved by improving the compression and decompression performance of compressed assets.

Brotli-G: AMD modifications to Brotli

The AMD modifications to the Brotli bitstream format are focused on allowing efficient, data-parallel decompression on GPUs and multithreaded CPUs, while still maintaining comparatively high, state-of-the-art compression ratios for general-purpose data. Some of the modifications include:

  • Limited the size of chunks to compress (pages) providing the ability to use multiple Compute Units (CUs) on the GPU.
  • Implemented parallel Huffman substreams, providing the ability to use SIMD processing.
  • Simplified the bitstream format to maximize performance on GPU.

The Brotli-G decompression algorithm has been implemented as a ‘shader’ program (written in HLSL) that runs on virtually any DirectX®-compatible graphics card. Implementing decompression in shaders takes advantage of the enormous parallel processing power available in modern GPU hardware, to enable incredibly fast decompression speeds.

Existing optimized Brotli decompression functions (CPU implementations) should be able to decompress the Brotli-G bitstream, while more optimal data-parallel implementations on hosts or accelerators can further improve performance.

One thing for developers to note is that assets that have already been compressed with Brotli cannot be decompressed with Brotli-G decompressor implementations. To take advantage of Brotli-G, assets must be recompressed using a Brotli-G compatible compressor (also supplied in the SDK).

What AMD is releasing

We are releasing a Brotli-G SDK that contains source code for CPU compression and decompression on both the CPU and the GPU. The SDK also contains source code for a CLI application that can be used for testing and measuring the performance of the Brotli-G codec. The SDK can be found in its GitHub repository.

We will also be integrating Brotli-G codecs into our Compressonator content authoring toolchain component as a supported compression format in its SDK library (coming soon).

AMD encourages collaborative, open-source development to further improve compression ratio and multi-threaded compression/decompression performance on CPU and GPU accelerators, in addition to improving integration with the general Brotli format. 

Licensing

Brotli-G is released under the same licensing terms as Brotli (MIT-based license known as BCP 78) to support broad adoption in many different software ecosystems.

Find out more and download

Brotli-G SDK

Brotli-G SDK

Brotli-G is an open-source compression/decompression standard for digital assets (based on Brotli) that is compatible with GPU hardware.

GPUOpen

GPUOpen

The home for games and graphics developers. Discover how our open source tools, SDKs, and effects can help you find your best graphics performance. Learn tips and tricks with our extensive tutorials and samples.

Enjoy this blog post? If you found it useful, why not share it with other game developers?

You may also like...

Getting started: our software

New or fairly new to AMD’s tools, libraries, and effects? This is the best place to get started on GPUOpen!

Getting started: development and performance

Looking for tips on getting started with developing and/or optimizing your game, whether on AMD hardware or generally? We’ve got you covered!

If slide decks are what you’re after, you’ll find 100+ of our finest presentations here. Plus there’s a handy list of our product manuals!

Developer guides

Browse our developer guides, and find valuable advice on developing with AMD hardware, ray tracing, Vulkan, DirectX, UE4, and lots more.

Words not enough? How about pictures? How about moving pictures? We have some amazing videos to share with you!

The home of great performance and optimization advice for AMD RDNA™ 2 GPUs, AMD Ryzen™ CPUs, and so much more.

Product Blogs

Our handy product blogs will help you make good use of our tools, SDKs, and effects, as well as sharing the latest features with new releases.

Publications

Discover our published publications.