Do I understand correctly that this has all the building blocks which make up various lossless compression schemes (dictionaries, run length encoding, delta encoding, etc) and kit bashes them together for your specific dataset?
I was under the impression that modern file formats and compression schemes already kind of do that as part of the normal compression process, so I was a little surprised by how much they beat some of the competition in their benchmarks.
It can only do the impressive numbers after you give it a schema for the data, and the data has to be structured in a way that follows a rigid schema, so it’s not magic.
OpenZL compresses data into a self-describing wire
format, any configuration of which can be decompressed by a universal decoder. OpenZL’s design
enables rapid development of tailored compressors with minimal code; its universal decoder eliminates
deployment lag; and its investment in a well-vetted standard component library minimizes security
risks.
The freedom afforded by OpenZL’s graph model is a
double-edged sword: the capability it offers comes at
the cost of a combinatorial explosion of choices that
must be made—namely which components to use
and how to compose and configure them. There is
no one-size-fits-all compressor
Webassembly for compression or something, but it’s made by the zstd people so maybe it will turn out to be magic eventually.
On one hand the x2.06 ratio for the SAO data is extremely good, zpaq and paq8px are 50x and 10000x slower at encoding and decoding and still can’t even reach that ratio (I didn’t try even more extreme compression, paq8px -8 already took 20 minutes).
On the other hand,
enwik: A Case Study Where OpenZL
Performs Poorly
We present this case study to show that OpenZL is
not a magic bullet for all use cases. The shortcomings
are twofold. First, the standard codec library can
achieve good results for many datasets, but is not
currently optimized for human text. Second, English
text is a highly compressible format for which domain-
specific transformations are especially useful. xwrt
combines multiple such transformations to achieve
its results. With this in mind, we are optimistic
that porting text-specific codecs and text parsing will
improve the performance of OpenZL compressors in
this domain.
Do I understand correctly that this has all the building blocks which make up various lossless compression schemes (dictionaries, run length encoding, delta encoding, etc) and kit bashes them together for your specific dataset?
I was under the impression that modern file formats and compression schemes already kind of do that as part of the normal compression process, so I was a little surprised by how much they beat some of the competition in their benchmarks.
It can only do the impressive numbers after you give it a schema for the data, and the data has to be structured in a way that follows a rigid schema, so it’s not magic.
Webassembly for compression or something, but it’s made by the zstd people so maybe it will turn out to be magic eventually.
On one hand the x2.06 ratio for the SAO data is extremely good, zpaq and paq8px are 50x and 10000x slower at encoding and decoding and still can’t even reach that ratio (I didn’t try even more extreme compression, paq8px -8 already took 20 minutes).
On the other hand,