bzip3
430 points
• 7 days ago
• Article
Link
BZip3 是一款高性能压缩工具，作为 BZip2 的精神继任者。它使用 order-0 的 context-mixing 熵编码器和基于 suffix arrays 的快速 Burrows-Wheeler 变换，因而在压缩比和处理速度上都明显优于前代。它还结合了 RLE 、 Lempel–Ziv 与预测（Prediction）等处理步骤，采用类似 LZ77 的字符串匹配和类似 PPM 的上下文建模。与 BZip2 一样，BZip3 对文本和源代码的压缩进行了特别优化。
在以 Perl5 源代码为语料的对比基准测试中，BZip3 在文件体积缩减和速度之间表现出良好的平衡。配置适当的块大小和线程数后，它持续生成比 BZip2 和 Zstandard 更小的文件。此外，当与诸如 lrzip 之类的长距离去重工具配合使用时，BZip3 的压缩效果非常具有竞争力，优于单独使用 LZMA 或 BZip2 的情况。
尽管 BZip3 为了可靠性已在包括各类 ARM 、 MIPS 和 x86 配置在内的广泛架构上进行了大量测试，但开发者仍附带重要免责声明：由于底层算法复杂且存在罕见的边缘情况错误，除非能够接受理论上存在但概率极低的数据丢失风险，否则不应将其用于关键任务的数据。
该软件对编译器选择高度敏感：在 Linux 上使用 clang13 构建的版本在每线程的压缩和解压吞吐量上表现尤为出色。安装简便，支持 autotools 、 CMake 等常见构建流程，并可通过 Homebrew 等包管理器在多种系统上获取。项目采用 LGPLv3 许可证，并对第三方库和作者在 Burrows-Wheeler 变换与熵编码等组件上的贡献予以致谢。
BZip3 is a high-performance compression tool designed as a spiritual successor to BZip2. By utilizing an order-0 context mixing entropy coder and a fast Burrows-Wheeler transform that leverages suffix arrays, it achieves significantly better compression ratios and faster processing speeds than its predecessor. It also incorporates RLE with a Lempel Ziv and Prediction pass, relying on LZ77-style string matching and PPM-style context modeling. Like BZip2, BZip3 is particularly optimized for compressing text and source code.
In comparative benchmarks using a corpus of Perl5 source code, BZip3 demonstrates a strong balance between file size reduction and speed. When configured with specific block sizes and thread counts, it consistently produces smaller files than BZip2 and Zstandard. Furthermore, when combined with long-range deduplication tools like lrzip, BZip3 achieves highly competitive compression results that outperform standalone LZMA or BZip2 implementations.
While BZip3 is designed for reliability and has been extensively tested across a wide array of architectures, including various ARM, MIPS, and x86 configurations, the developer includes a notable disclaimer. Due to the complexity of the underlying algorithms and the potential for rare edge-case bugs, users are cautioned that they should not use the program for mission-critical data unless they are prepared for the slight, albeit theoretically possible, risk of data loss.
Performance of the software is notably sensitive to the choice of compiler, with Linux builds using clang13 showing impressive throughput in both compression and decompression per thread. Installation is straightforward, supporting standard build processes like autotools and CMake, and the project is available for various systems via package managers like Homebrew. The project is licensed under the LGPLv3, with contributions from various third-party libraries and authors acknowledged for components such as the Burrows-Wheeler transform and entropy coding logic.
124 comments • Comments Link
• bzip3 是一款基于 Burrows-Wheeler Transform (BWT) 的压缩器，血缘上类似于 bzip2 但实现和用途截然不同，最近被纳入了长期存在的行业压缩基准测试中。
• zstd 已成为通用压缩的现代行业标准，得益于其极快的解压速度、对文件系统与数据库技术的广泛支持，以及在处理结构化数据（如 JSON）时通过共享字典实现高效压缩的能力。
• 对压缩算法进行公正基准测试非常困难：不同实现的默认设置往往在速度、内存占用和压缩率之间做出不同权衡。要做到公平比较，必须标准化窗口大小和内存上限等参数，因为基于 BWT 的压缩器在不同输入和块配置下的表现会有显著差异。
• 批评者指出，一些关于 bzip3 的性能宣称似乎来自挑选过的数据集或不对等的配置，其中 bzip3 被允许使用远大于 zstd 基准的内存 / 块窗口。当对 zstd 启用相应的长距离匹配选项时，它在压缩比和速度上通常能与 bzip3 匹敌或更优。
• bzip3 文档中关于潜在数据丢失的警告令潜在用户极为犹豫——无论这种声明在法律上是否等同于常见的"按原样"开源许可，这种警示都会影响采纳意愿。
• bzip3 的表现高度依赖于输入数据，使其相比更稳定、广泛集成的 zstd 显得较为不可预测。它的适用场景主要限于那些专业化且一次性的归档任务，用户有时间去尝试多种算法和参数组合以寻找最佳结果。
• 现代高级压缩流程越来越多地使用定制的预训练字典，以在诸如 JSONL 等小型结构化文件上实现高压缩比，同时仍能支持随机访问解压。
• 人们持续对缺乏自动化参数优化表示不满。因为许多算法只有经过针对性微调才能发挥最佳性能，观察者建议使用编码代理或自动化研究方法，按固定目标（例如时间与大小）进行匹配，从而提供更诚实的比较。
• 有人认为该工具的命名容易误导——它并非由 bzip2 的原作者开发，这加剧了外界对其与遗留软件关系的混淆。
总体共识是：bzip3 作为一个有趣的 BWT 项目值得关注，但很难在通用性、成熟度和易用性方面与 zstd 竞争。大多数参与者认为，因 zstd 在性能配置上更为均衡，它是通用工程任务的首选默认工具。比较压缩工具时，简单的基准测试往往不够诚实，因为它们常常忽视窗口大小、内存使用和输入特性等关键因素。由此，人们强烈主张软件库应优先保障可靠性、清晰的文档和标准化的许可，以赢得用于严肃归档或生产工作负载的信任。 • Bzip3 is a Burrows-Wheeler Transform (BWT) based compressor, similar in lineage to bzip2 but functionally distinct, which has recently been added to long-standing industry compression benchmarks.
• Zstd has become the modern industry standard for general-purpose compression due to its high decompression speed, broad support across filesystem and database technologies, and effective use of shared dictionaries for structured data like JSON.
• Benchmarking compression algorithms is notoriously difficult because default settings often favor different trade-offs in speed, memory usage, and compression ratio. Comparing algorithms fairly requires normalizing parameters like window size and memory limits, as BWT-based compressors can show significant performance variances depending on input data and block configuration.
• Critics point out that some bzip3 performance claims appear based on cherry-picked data or disparate configurations, where bzip3 is allowed much larger memory/block windows than the zstd baseline. When zstd is configured with matching long-distance matching flags, it often outperforms or rivals bzip3 in both ratio and speed.
• The warning disclaimer in bzip3's documentation regarding potential data loss causes significant hesitation among potential users, regardless of whether it is legally equivalent to standard "as-is" open-source software licenses.
• Performance for bzip3 is highly data-dependent, making it an unpredictable choice compared to the more consistent and widely integrated zstd. Its utility is largely relegated to specialized, one-off archival tasks where users have the time to trial multiple algorithms and settings.
• Advanced compression workflows now increasingly utilize custom, pre-trained dictionaries to achieve high ratios on small, structured files (like JSONL) while maintaining the ability to perform random access decompression.
• There is ongoing frustration regarding the lack of automated tool-based parameter optimization. Since algorithms often require specific fine-tuning to reach their potential, observers suggest that using coding agents or automated research to match a fixed goal—such as target time or size—would provide more honest comparisons.
• The naming of the tool is viewed by some as misleading, as it is not developed by the original authors of bzip2, creating confusion about its relationship to the legacy software.
The discussion reflects a broader consensus that while bzip3 is an interesting BWT-based project, it struggles to compete with zstd's ubiquity, maturity, and ease of use. Most participants find that zstd is the "go-to" default for general engineering tasks due to its balanced performance profile. When comparing compression tools, the consensus is that simple benchmarks are frequently disingenuous, as they often ignore the critical impact of window sizes, memory usage, and input-specific characteristics. Consequently, there is a strong sentiment that software libraries should prioritize reliability, clear documentation, and standard licensing to gain trust for serious archival or production workloads.