Cafu Engine

Here are a few methods for compressing sparse byte arrays:

• PackBits, http://en.wikipedia.org/wiki/PackBits
• zlib's deflation algorithm
• Huffman encoding, http://en.wikipedia.org/wiki/Huffman_coding
  According to Robert Beckebans (Tr3B), this approach is also used in Q3A.
  Huffman encoding seems to be a part of zlib's deflatation algorithm.
• Build a binary tree that enumerates the non-zero bytes, then lists those bytes.

The problem of compressing sparse arrays is encountered when entity state is to be sent over the network. The entity state is normally stored in a bit-field (array of bits, a "BitMessage"), but instead of sending this bit-field directly, the server first builds the delta to the previous state using a simple XOR combine, which yields a (very) sparse array. Only the compressed contents of this sparse array is then sent over the network.


... Should collect real test data of a real Ca3DE session, then excercise the test data through all compression methods and all their reasonable combinations; creating stats will reveal the "best" method for our purposes...