What a fun project! Back in 1989 ES2 regularly shipped large chip data bases of up to 150MB from it's world wide design centers to the fab in France. Tape shipment was taking three days from the US where I ran the design center. Low cost 18K BPS PEP modems were becoming available but still far too slow for data bases of this size. ISDN and X.25 were pricey.

The largest data bases were fractured output from Dracula™. Looking at the CIF text files I noticed an extraordinary amount of redundancy in the data. The compression scheme I gradually developed in my spare time was designed to remove the redundancy and led to compression factors of up to 30:1 making it possible to modem the largest chip data at reasonable cost. The final system was developed jointly by Amal Reddy, Rod Widdowson and myself and was eventually installed in ES2 world wide to reliably transmit chip fabrication packages. We called the system Bulk Data Transfer (or BDT).

Here is an example of the results in CIF form before and after compression. Note the {}[]?+|& symbols have special meaning in the output stream:

             ARRAY OF BOXES BEFORE COMPRESSION:
 
             P 698640 394920 698640 395160 698880 395160 698880 394920;
             P 699360 394920 699360 395160 699600 395160 699600 394920;
             P 700080 394920 700080 395160 700320 395160 700320 394920;
             P 700860 394920 700860 395160 701100 395160 701100 394920;

             ARRAY OF BOXES AFTER COMPRESSION:

             P 698640 394920 & [240 [240|
             P {720?
             P+
             P {780?

Both contain identical data. I am still amazed how effective it is.

A patent was filed for the compression algorithms. As patents go it is reasonably easy to read! Click here for patent text.

You can look at the code here.

HomePage

Changes last made on:  11/17/06 .