Read further about the difference between Compaction and compression.
How it Works
So, how does Compaction actually work?
Compaction uses machine learning to identify patterns in a big sample set of data files, typically thousands of small machine-generated messages. This process is called "training". When a pattern is seen to be used frequently in the sample, it is added to a “Codebook”, which can be thought of as a list of objects matched to another list of objects, such as by having the letter “A” represented by the letter “X”. A Compaction Codebook matches the data patterns to a kind of index called a “Codeword”. The patterns might be, say, 128 bits long, while the Codewords to which the patterns are matched in the Codebook are much shorter, usually between three to 10 bits long. This Codebook building process, identifying patterns is a big set of sample data, is the heavy lift in terms of computation that Compaction performs, and it is all done before any live data moves.
Once Compaction is finished building the Codebook, it is placed in a source (such as a sensor) and a destination (such as the cloud). Going forward, the sources and destinations communicate solely in Codewords, and none of the original data files ever leave the source.
Faster and smaller messages
Compaction is most effective when compacting small, repetitive data, such as small messages – even as small as 8 bytes – and it is most effective when it is applied to repetitive, low-entropy IoT or machine data. By contrast, compression works only with large groups of messages or files that have a size of at least one KB.
Using Compaction, messages are compacted and transmitted in real-time, and codes are sent immediately. There is no need to accumulate and send messages in batches. With compression, however, real-time transmission is not possible. In order for the algorithm to find sufficient patterns, sufficient data must be collected; compression typically involves multiple scans, eliminating the possibility of real-time transmission, and the data must be accumulated and sent in batches.
Lightning Speed
When compared with compression, compaction produces almost no latency, as it only requires a table lookup, which takes a few microseconds and can send the first bits 400 times faster than compression. The compression process injects latency, and the file is scanned multiple times, looking for patterns. That means the pattern search is conducted file by file, which takes much more time and energy than it should.
The number of transportation related apps that transmit data is growing exponentially, and users want to be able to see the data transmitted by the app in real-time in order to take full advantage of its functionality. AtomBeam's average IoT data reduction using compaction is 75%. The problem with compression is that it often greatly enlarges the size of small messages. It will reduce machine files by less than 20%, the average reduction in IoT messages is 4%, and it often increases the size of IoT messages.
Atombeam and Compaction Wins
At the end of the day, AtomBeam is lossless. AtomBeam’s Compaction doesn’t lose any data – when compression is used, some compression is lossy, which makes it impractical for a large number of applications. Using our unique software, you can reduce bandwidth costs by a factor of four and save what would normally cost billions of dollars.
Visit our StartEngine raise page for more information regarding our technology and how you can join our rapidly growing community of early investors.
