IV. Data Compression Protocols

Besides error control protocols, all current high-speed modems also support data compression protocols. That means the sending modem will compress the data on-the-fly and the receiving modem will decompress the data to its original form.

IV.1. MNP5 and V.42bis

There are two standards for data compression protocols, MNP5 and CCITT V.42bis. Some modems also use proprietary data compression protocols.

A modem cannot support data compression without utilizing an error control protocol, although it is possible to have a modem that only supports an error control protocol but not any data compression protocol. A MNP5 modem requires MNP 4 error control protocol and a V.42bis modem requires V.42 error control protocol.

Also note that although V.42 include MNP4, V.42bis does not include MNP5. However, virtually all high-speed modems that support CCITT V.42bis also incorporate MNP5.

The maximum compression ratio that a MNP5 modem can achieve is 2:1. That is to say, a 9600 bps MNP5 modem can transfer data up to 19200 bps. The maximum compression ratio for a V.42bis modem is 4:1. That is why all those V.32 modem manufacturers claim that their modems provide throughput up to 38400 bps.

There are some 2400-bps modems advertised as having MNP5 but are not real MNP5 modems. These modems do not have MNP5 implemented in the modems themselves, but rather depend on the communications software (e.g. Bitcom Deluxe) to do the tricks. Besides being slower than the real MNP5 modems, these modems will not provide an error-free connection unless you use the accompanying software. If you buy one of these modems and decide to use your own software (e.g. Procomm Plus), you have to treat the modem as a plain vanilla 2400-bps modem.

IV.2. Are MNP5 and V.42bis useful?

Don't be fooled by the claim. It is extremely rare, if ever, that you will be able to transfer files at 38400 bps. In fact, V.42bis and MNP5 are not very useful when you are downloading files from online services. Why?

How well the modem compression works depends on what kind of files are being transferred. In general, you will be able to achieve twice the speed for transferring a standard text file (like the one you are reading right now). Decreasing by 50% means that you can double the throughput on the line so that a 9600 bps modem can effectively transmit 19200 bps.

However, V.42bis and MNP5 modem cannot compress a file which is already compressed by software. In the case of MNP5, it will even try to compress a precompressed file and actually expand it, thus slow down the file transfer! Here are the test results obtained by downloading the three compressed files using (1) MNP4 without data compression, (2) MNP5, (3) V.42 without data compression, and (4) V.42bis.

    Filename        MNP4        MNP5        V.42            V.42bis
    ----------------------------------------------------------------
    dayrpt.arc      1023 cps    946 cps     1002 cps        1010 cps
    sunset.arc       971 cps    935 cps      953 cps         950 cps
    text109k.arc    1085 cps    988 cps     1064 cps        1053 cps

If you have ever downloaded files from a BBS or online service, you know that almost all files are in a compressed format. Therefore, you should only expect to see an actual throughput between 950 to 1100 cps even if your V.32/V.42bis modem is supposed to offer throughput "up to" 38400 bps.

Most PC files are in the ZIP format. Macintosh files are typically in the .SIT (Stuffit) or .CPT (Compact Pro) format. Amiga files are usually in the ZOO, ARC or LZH format. Note that GIF files are also in a compressed format.

IV.3. Compression Software vs. MNP5/V.42bis

There are several reasons why compression software programs (such as PKZIP or Stuffit) are superior to MNP5 or V.42bis.

1. Compressed files save disk storage space.

2. Compression software programs are more versatile. Most of them allow you to group several files in a compressed file archive to ensure that all the related files get transferred at the same time.

3. Software compression is more efficient than on-the-fly modem compression. In the case of a small file, this may not make much difference. But the difference can be significant when you are transferring large files.

    Filename        Size            Time            Throughput
    ----------------------------------------------------------
    the-wave.txt    143579 bytes    43 seconds      3296 cps
    dayrpt.arc        8423 bytes     8 seconds      1010 cps
    dayrpt.wks       19712 bytes     8 seconds      2337 cps
    sunset.arc        5084 bytes     5 seconds       950 cps
    sunset.pic       16391 bytes     6 seconds      2643 cps
    text109k.arc    29775 bytes     28 seconds      1053 cps
    text109k.txt    111386 bytes    39 seconds      2822 cps

As we can see from the test results, it is about 30% faster to transfer the compressed file text109k.arc than to download the text file with V.42bis.

Hayes BBS does not provide a compressed version for the file the-wave.txt. Using PKZIP (for PC) and Stuffit (for Macintosh), we obtain the following results:

    the-wave.zip:           6812 bytes (PKZIP)
    the-wave.sit:           6081 bytes (Stuffit)

Assuming a transfer speed of 1000 cps, the compressed file can be downloaded in 7 seconds. That's six times faster than downloading the text file with V.42bis!

Here is another example. One of my local BBS has a Macintosh TIFF file (206,432 bytes) which can be downloaded in 56 seconds (with an effective throughput of 3745cps) with a V.32/V.42bis modem.

The result may seem impressive at first. However, the file can be compressed to 6065 bytes (with Compact Pro) or 7385 bytes (with Stuffit). Assuming a transfer speed of 1000 cps, it would only take 68 seconds to transfer. Again, it is seven to nine times faster than downloading the file with V.42bis.

On-the-fly modem compression does have one advantage. It is more convenient. You can send a file without compressing it first and the recipient does not need to decompress the file.

IV.4. Local Flow Control and Data Buffering

To get the most from a modem with data compression, you'll want to send data from your PC to the modem as quickly as possible. If the modem is idle and waiting for the computer to send data, you are not getting the maximum performance from the modem.

For example, you have a V.32/V.42bis modem and you want to send a text file to a remote system which also has a V.32/V.42bis modem. Let's assume the modem is able to send the file at 20000 bps using V.42bis. If your computer is sending data to your modem at 9600 bps, your modem will have to stop and wait to receive data from your computer.

To get the maximum performance, you want to set the computer to send data to the modem at 38400 bps (the maximum a V.32/V.42bis modem can achieve). Since the modem can only send the file to the other modem at 20000 bps, it will never have to wait.

Here are the test results for downloading the text file thewave.txt by setting the communication port at different speeds (usually referred to as "DTE speed"):

the-wave.txt:    946 cps        (modem port speed 9600 bps)
                1885 cps        (modem port speed 19200 bps)
                3296 cps        (modem port speed 38400 bps)

However, there is a new problem. Since your computer is sending data faster than the modem can handle, there needs to be some ways for the modem to ask the computer to stop sending data. Otherwise, data loss is sure to occur. This is where local flow control comes into play.

A high-speed modem typically supports two kinds of local flow control: hardware handshaking (CTS/RTS) and software handshaking (XON/XOFF). Of the two, hardware flow control is the preferred method. We have mentioned earlier that there are three links involved when you are connected to a remote system:

The link between your computer and your modem
The link between the modems
The link between the remote modem and the remote computer

Local flow control is used for the first and third links. Notice that the first link may not use the same kind of flow control as the third link.

Hardware flow control (or hardware handshaking) works by altering voltage levels on the RTS (Request To Send) and CTS (Clear To Send) signal lines at the RS232 serial interface between the modem and the computer.

CTS is used by the modem on the sending end of a transmission. When the local modem is ready to receive data, it sends the CTS signal to the local computer and the computer starts transferring data. If the modem is unable to accept the data as fast as it is received from the computer, the modem will disable the CTS to inform the computer that the modem buffer is almost full (A high- speed modem typically contains a small amount of RAM which is used to provide data buffers). The computer will then suspend data transfer. Once the local modem has emptied its buffer by transmitting data to the remote modem, it will enable CTS again.

RTS is used by the computer on the receiving end of a transmission. When the computer cannot accept data at the rate at which the modem is passing data, it will disable RTS. The computer enables RTS again when it is ready to resume receiving data from the modem.

Software flow control (or software handshaking) is achieved by embedding control character in the data stream. XON and XOFF are the most commonly used control characters. XON is also known as ControlQ or DC3 (ASCII 19) while XOFF is known as ControlS or DC1 (ASCII 17).

The use of XON and XOFF during data transfer can create problems when a binary file contain the ControlS (^S) character as a legitimate part of the data. Do not use this method if ^S and ^Q are part of the transmitted data.

IV.5. Macintosh & High-speed Modems

If you use a Macintosh with a high-speed modem, you will need a special modem cable that is wired correctly to support hardware handshaking. You can order the cable from most mailorder companies that sell high-speed modems. I got mine from Maya Computer (800-541-2318) for $10 (plus $2.50 for shipping & handling).

Unfortunately, the cable did not work with my SE. The cable is good since it worked fine on a Mac IIsi. It just refused to work on my SE. I was disappointed but not surprised. After all, my SE is equipped with a 25 Mhz 68030 accelerator. (Well, it is actually both an accelerator and a video adapter for a 19 inch dualpage monitor.) Since I will never want to run my SE without the accelerator, I have no choice but to use software handshaking.

IV.6. PC & UART

Your PC's serial port has a UART (Universal Asynchronous Receiver/Transmitter) chip to control the input/output. The XT usually has an 8250 UART, the AT usually has a 16450 UART. If you are running Windows, Desqview, OS/2 or any other multitasking environment, you should upgrade your UART with the 16550 (if your PC does not already have one). The 16550 is standard in most IBM PS/2 and many 386based computers. The 16550 UART has a 16 bytes FIFO (first in, first out) buffer that helps to prevent degradation when several programs are running at the same time.

If you use an external modem, the UART is in your computer (either on the motherboard or on an I/O card that has the serial port). If you use an internal modem, the UART is on the modem. (Both internal modems from Practical Peripherals and Zoom use the 16550 UART. The Twincom 96/42 uses a 16450. The CompuCom SpeedModem Champ, due to its unique design, does not use a standard UART.)

Even if you have a 16550 UART, the communication software that you use will need to support it. Fortunately, the most recent versions of popular communications programs are all designed to support the 16550 UART.

IV.7. Hayes ESP (Enhanced Serial Port)

Hayes makes an adapter called Enhanced Serial Port (ESP) that has two serial ports complete with an onboard coprocessor. The ESP can save your PC's CPU from having to manage the work load. If a 16550 UART is not good enough for you, the ESP may be the only answer.