Offshore / Compression
The compression techniques used in the Offshore Compact service are proprietary. Top secret.
For those who prefer pictures, here is the compression story for the Offshore Compact file:
------------
| | -----------
| | | |
| | | | --------
| | | | | |
| | | | | |
| | | | | |
------------ ----------- --------
Normal GRIB Other Offshore Compact
file Services file
Notice how much smaller the Offshore Compact file is!
For those who like a few details to go along with their pictures, read on.
The compression topic can be confusing, as so many claims are being made regarding how one service or another is getting fantastic compression! - often without any real detail attached to those claims. One service claims they get 90% compression, another 95% compression.
Some email services offer image compression so you can send and receive images while out at sea. Image compression is generally lossy, and depending on how lossy you tolerate, you can achieve huge reductions in image sizes. Some programs may indeed achieve 90% or higher compression on images.
Another example is text, such as an email message or a web page. Text is also easily compressed. For example, at the time this was written, this HTML page is 13,470 bytes and compresses down to 4,177, which is 70% compression.
There is a family of compression algorithms which have been in existence for a long time, and everybody has access to these algorithms. One example of this is the popular gzip program. There are others. The majority of web traffic world wide is compressed using gzip. The web servers used by LuckGrib automatically compress all of the HTML pages you view, and your browser automatically uncompresses them. This is all quite ordinary.
Here’s the scoop. Weather data is not so easy. The GRIB format is binary data and already has a certain level of compression applied.
What LuckGrib is doing in the implementation of the Offshore Data Compact file is much better than what a standard compression algorithm applied to GRIB data can achieve, and there are examples of that in the table below. LuckGrib is claiming that its proprietary compression algorithm is the best available, for weather data, and is offering evidence of that. The LuckGrib Offshore Data Compact file implements a proprietary domain specific compression algorithm.
The most important point to be understood is this: the compression techniques developed for this project work. In fact, they work very, very well. When other services make great compression promises you should investigate the details to see if the promises being made will apply to what you are doing. There are no other weather services achieving the compression results that the LuckGrib Offshore Data service is achieving.
The following table lists a variety of real files which are typical for offshore sailors. Click any of the filenames below for more detail along with the ability to download the files for examination and verification. For any of these examples, you can download both the original GRIB file and the compressed Offshore file, and then compare them. The two files are functionally the same.
| File Name | Original size | Offshore size | Saved |
| 063544.ww3_wind_3.grb | 15,744 | 7,593 | 52% |
| 093948.ww3_wind_5.grb | 98,152 | 44,391 | 55% |
| 20151001_002917_.grb | 41,087 | 11,542 | 72% |
| ECMWF_20180507_12.grb | 36,898 | 13,762 | 63% |
| GFS.050.20190113.12.grb2 | 132,167 | 55,716 | 58% |
| GFS.075.20190113.12.grb2 | 66,396 | 28,888 | 56% |
| GFS.100.20190113.12.grb2 | 43,890 | 18,316 | 58% |
| GFS20160511040500755.grb | 12,940 | 6,506 | 50% |
| GFS_20181115_00.grb | 26,496 | 12,706 | 52% |
| PWC_20160924_09.grb | 63,992 | 28,729 | 55% |
| WW3.20180813.06.grb2 | 29,454 | 5,605 | 81% |
| gfs20180802210007058.grb | 6,838 | 3,507 | 49% |
| gfs20180916065401993.grb | 16,824 | 7,505 | 55% |
| gfs20190110185803667.grb | 27,602 | 12,682 | 54% |
When you click into one of the files, you will see more detail. For example:
| Original GRIB file: | GFS20160511040500755.grb |
| Contents: | PRMSL, Wind-UV |
| Grid: | 44S 32S 176W 152W |
| Grid size: | 17 x 9 = 153 points |
| Resolution: | 1.5° x 1.5° |
| Original size: | 12,940 bytes |
| gzip size: | 9,856 bytes |
| bzip2 size: | 10,378 bytes |
| Offshore Compact size: | 6,506 bytes |
| Offshore Compact savings: | 50%, 6,434 bytes |
| Offshore Compact file: | GFS20160511040500755.lgrb |
This file was downloaded while offshore on a passage between New Zealand and Hawaii. It came from Saildocs, which is a well respected provider of GRIB data worldwide. The file was downloaded using Sailmail through a SSB/Pactor modem connection. This file size was typical for the files I was downloading on that passage.
The gzip and bzip2 file sizes are provided to give a reasonable estimate of the file size you would download when obtaining data from other providers.
As you can see, the Compact Offshore file is much smaller than the other two compression techniques.
If I had had access to the Offshore Data system on that passage, I could have increased the resolution of my downloads while still being within my budget of around 14KB for downloads on a SSB/Pactor. Alternatively, I could have increased the coverate area of this download, or asked for a longer forecast. How you use the space savings achieved by this service is up to you.
If you have a GRIB file you consider typical of your usage and you believe none of these examples represent it, please forward the file to craig@luckgrib.com and it can be added to this list of examples.
The bottom line of this compression discussion is: the Offshore Compact files offer the best compression in the industry, by a wide margin.