Brotli-G: An open-source compression/decompression standard for digital assets that is compatible with GPU hardware   

Introduction and background

Brotli-G is based on the well-established Brotli lossless compression algorithm standard maintained by the IETF (also known as RFC7932). Brotli was first released in 2013 and the open-source specification was developed and generalized between 2013 and 2016. It was developed by Google Research as an open successor to the GZIP and Deflate algorithms, with similar performance (to Deflate) but a better compression ratio.

The original use case was to make websites load faster by compressing content used by web servers and content delivery networks. Brotli is useful as a general compression scheme for many types of data/content/assets. The ‘intended audience’ as defined in the specification is characterized as follows:

This specification is intended for use by software implementers to compress data into and/or decompress data from the Brotli format.

Existing ecosystem for Brotli

Brotli browser acceptance
From https://caniuse.com/?search=brotli

The technical details of the Brotli algorithm are well documented (see resources section below). At a high level, it is a combination of the LZ77 compression algorithm and Huffman coding (similar to GZIP and Deflate).

It is well documented that Brotli performs competitively vs other standards (such as GZIP and Deflate) in all metrics: compression ratio, compression speed, and decompression speed. Comparisons with various compressions standards can be seen at Squash Compression Benchmark (quixdb.github.io)

Because of the obvious performance and compression ratio advantages over previous compression formats, Brotli has been widely adopted in web applications and content spaces. Brotli can be used to compress web application assets such as fonts, javascript, images, and much more.

Brotli has already been successful in these areas. When asset file sizes are smaller, they can be delivered over networks faster, and when compression/decompression is fast, web pages load faster and are more responsive.

As a general data compression format, Brotli may also be applied to other software ecosystems such as gaming and cloud gaming. Video games are asset-dense software applications, and many already employ some type of compression to reduce file size. The gaming experience may be improved by improving the compression and decompression performance of compressed assets.

Brotli-G: AMD modifications to Brotli

The AMD modifications to the Brotli bitstream format are focused on allowing efficient, data-parallel decompression on GPUs and multithreaded CPUs, while still maintaining comparatively high, state-of-the-art compression ratios for general-purpose data. Some of the modifications include:

  • Limited the size of chunks to compress (pages) providing the ability to use multiple Compute Units (CUs) on the GPU.
  • Implemented parallel Huffman substreams, providing the ability to use SIMD processing.
  • Simplified the bitstream format to maximize performance on GPU.

The Brotli-G decompression algorithm has been implemented as a ‘shader’ program (written in HLSL) that runs on virtually any DirectX®-compatible graphics card. Implementing decompression in shaders takes advantage of the enormous parallel processing power available in modern GPU hardware, to enable incredibly fast decompression speeds.

Existing optimized Brotli decompression functions (CPU implementations) should be able to decompress the Brotli-G bitstream, while more optimal data-parallel implementations on hosts or accelerators can further improve performance.

One thing for developers to note is that assets that have already been compressed with Brotli cannot be decompressed with Brotli-G decompressor implementations. To take advantage of Brotli-G, assets must be recompressed using a Brotli-G compatible compressor (also supplied in the SDK).

What AMD is releasing

We are releasing a Brotli-G SDK that contains source code for CPU compression and decompression on both the CPU and the GPU. The SDK also contains source code for a CLI application that can be used for testing and measuring the performance of the Brotli-G codec. The SDK can be found in its GitHub repository.

We will also be integrating Brotli-G codecs into our Compressonator content authoring toolchain component as a supported compression format in its SDK library (coming soon).

AMD encourages collaborative, open-source development to further improve compression ratio and multi-threaded compression/decompression performance on CPU and GPU accelerators, in addition to improving integration with the general Brotli format. 

Licensing

Brotli-G is released under the same licensing terms as Brotli (MIT-based license known as BCP 78) to support broad adoption in many different software ecosystems.

Find out more and download

Brotli-G SDK

Brotli-G SDK

Brotli-G is an open-source compression/decompression standard for digital assets (based on Brotli) that is compatible with GPU hardware.

Latest news

AMD Matrix Cores (amd-lab-notes)

This first post in the ‘AMD lab notes’ series takes a look at AMD’s Matrix Core technology and how best to use it to speed up your matrix operations.

Looking for a good place to get started with exploring GPUOpen?

Getting started: our software

New or fairly new to AMD’s tools, libraries, and effects? This is the best place to get started on GPUOpen!

Getting started: development and performance

Looking for tips on getting started with developing and/or optimizing your game, whether on AMD hardware or generally? We’ve got you covered!

You may also like...

Explore our huge collection of detailed tutorials, sample code, presentations, and documentation to find your answers to your graphics development questions.

Create wonder. No black boxes.

The home of great performance and optimization advice for AMD RDNA™ 2 GPUs, AMD Ryzen™ CPUs, and so much more.

Browse all our useful samples. Perfect for when you’re needing to get started, want to integrate one of our libraries, and much more.

Discover what our SDK technologies can offer you. Query hardware or software, manage memory, create rendering applications or machine learning, and much more!

Analyze, Optimize, Profile, Benchmark. We provide you with the developer tools you need to make sure your game is the best it can be!