Introduction and background
Brotli-G is based on the well-established Brotli lossless compression algorithm standard maintained by the IETF (also known as RFC7932). Brotli was first released in 2013 and the open-source specification was developed and generalized between 2013 and 2016. It was developed by Google Research as an open successor to the GZIP and Deflate algorithms, with similar performance (to Deflate) but a better compression ratio.
The original use case was to make websites load faster by compressing content used by web servers and content delivery networks. Brotli is useful as a general compression scheme for many types of data/content/assets. The ‘intended audience’ as defined in the specification is characterized as follows:
“This specification is intended for use by software implementers to compress data into and/or decompress data from the Brotli format.”
Existing ecosystem for Brotli
The technical details of the Brotli algorithm are well documented (see resources section below). At a high level, it is a combination of the LZ77 compression algorithm and Huffman coding (similar to GZIP and Deflate).
It is well documented that Brotli performs competitively vs other standards (such as GZIP and Deflate) in all metrics: compression ratio, compression speed, and decompression speed. Comparisons with various compressions standards can be seen at Squash Compression Benchmark (quixdb.github.io)
- Brotli decompression is supported by over 95% of web browsers today.
- Many web application frameworks support Brotli encoding (such as NGINX, and ASP.NET).
- Some CDNs use Brotli encoding (such as Azure, and ImageKit)
Brotli has already been successful in these areas. When asset file sizes are smaller, they can be delivered over networks faster, and when compression/decompression is fast, web pages load faster and are more responsive.
As a general data compression format, Brotli may also be applied to other software ecosystems such as gaming and cloud gaming. Video games are asset-dense software applications, and many already employ some type of compression to reduce file size. The gaming experience may be improved by improving the compression and decompression performance of compressed assets.
Brotli-G: AMD modifications to Brotli
The AMD modifications to the Brotli bitstream format are focused on allowing efficient, data-parallel decompression on GPUs and multithreaded CPUs, while still maintaining comparatively high, state-of-the-art compression ratios for general-purpose data. Some of the modifications include:
- Limited the size of chunks to compress (pages) providing the ability to use multiple Compute Units (CUs) on the GPU.
- Implemented parallel Huffman substreams, providing the ability to use SIMD processing.
- Simplified the bitstream format to maximize performance on GPU.
The Brotli-G decompression algorithm has been implemented as a ‘shader’ program (written in HLSL) that runs on virtually any DirectX®-compatible graphics card. Implementing decompression in shaders takes advantage of the enormous parallel processing power available in modern GPU hardware, to enable incredibly fast decompression speeds.
Existing optimized Brotli decompression functions (CPU implementations) should be able to decompress the Brotli-G bitstream, while more optimal data-parallel implementations on hosts or accelerators can further improve performance.
One thing for developers to note is that assets that have already been compressed with Brotli cannot be decompressed with Brotli-G decompressor implementations. To take advantage of Brotli-G, assets must be recompressed using a Brotli-G compatible compressor (also supplied in the SDK).
What AMD is releasing
We are releasing a Brotli-G SDK that contains source code for CPU compression and decompression on both the CPU and the GPU. The SDK also contains source code for a CLI application that can be used for testing and measuring the performance of the Brotli-G codec. The SDK can be found in its GitHub repository.
We will also be integrating Brotli-G codecs into our Compressonator content authoring toolchain component as a supported compression format in its SDK library (coming soon).
AMD encourages collaborative, open-source development to further improve compression ratio and multi-threaded compression/decompression performance on CPU and GPU accelerators, in addition to improving integration with the general Brotli format.
Brotli-G is released under the same licensing terms as Brotli (MIT-based license known as BCP 78) to support broad adoption in many different software ecosystems.
Find out more and download
Brotli-G is an open-source compression/decompression standard for digital assets (based on Brotli) that is compatible with GPU hardware.