FlexFAX Modem Bakeoff
November 2, 1993

Samuel J. Leffler

Silicon Graphics Computer Systems
2011 N. Shoreline Blvd.
P.O. Box 7311
Mountain View, CA 94039-7311

ABSTRACT

This paper reports on an experiment to evaluate the performance of a number of commonly used contemporary facsimile modems. Nine modems were used to send a 3-page facsimile to 37 different facsimile machines and modems located in North America and Europe. The results are discussed and a metric is presented for comparing the modems.

[This document is converted from the original paper and is available in a PostScript format that is more condusive to the presentation of figures, tables, and equations.]

This report is Copyright Sam Leffler and Silicon Graphics, Inc.

Permission to copy this document for non-profit use is hereby granted without fee, provided that (i) this document is reproduced unchanged and in its entirety, (ii) the above copyright notices and this permission notice appear in all copies, and (iii) the names of Sam Leffler and Silicon Graphics may not be used in any advertising or publicity relating to the document without the specific, prior written permission of Sam Leffler and Silicon Graphics.

1. Introduction

This paper reports on an experiment to evaluate the reliability and performance of a number of commonly used facsimile modems. Nine different modems were used with the FlexFAX software package to send a 3-page high resolution facsimile to 37 different facsimile devices around the world. The ability of the modems to establish and hold a connection with the remote device was studied along with the time required to successfully transmit the facsimile data. Based on the data collected from this experiment a simple metric was developed for comparing the different facsimile modems.

This document is organized in six sections. In Section 2 the test environment and methodology is discussed. Section 3 provides a primer on the communication protocol used by Group 3 facsimile devices. Section 4 discusses the test results, while Section 5 presents the metric that was derived from this experiment. In Section 6 the conclusions that were drawn from the experiment are presented together with some advise to buyers of facsimile modems.

2. Test Environment and Methodology

The basic experiment involved using FlexFAX version 2.2.0 to send a 3-page facsimile to a variety of sites using nine different facsimile modems. FlexFAX is a freely available software package that supports a wide variety of modems--basically any modem that conforms to the Class 1 or Class 2 host-modem communication specifications.

Only sending was evaluated for several reasons. First, the Group 3 facsimile protocol is designed so that the sender is mainly responsible for the reliable transfer of facsimile data. Personal experience indicates that a modem that does well at sending facsimile also does well at receiving facsimile, while the reverse is not true. Second, most uses of facsimile modems are significantly more oriented toward sending than receiving facsimile. Consequently modem buyers are more concerned with a modem's ability to send than receive. (It should be noted however that because of just this reason, some modem vendors do not put as much effort into testing the receive capability of their modems as they should.) Finally, the logistics of running receive testing were too difficult to surmount in the time available to do the experiment.

The experiment was run on a Silicon Graphics Indigo machine under version 4.0.5H of the IRIX operating system. [The system was modified in one important way; it included a bug fix to the UART driver to correct a problem with the handling of DCD on devices that require RTS/CTS flow control. Without this bug fix the USR Sportster modem cannot be used with an SGI system.] The 3-page facsimile shown in Appendix B was used for testing. The document includes three different types of data, as shown in Table 1. While many facsimile are either mostly text or simple graphics (e.g. charts of tables), the selected matter represents a more difficult test of the Group 3 encoding algorithms and, consequently, of the performance of the sender and receiver. The pages were encoded and transmitted exclusively using the CCITT Group 3 1D encoding algorithm. FlexFAX also supports the 2D encoding algorithm but it was not used because some fax machines have been observed to falsely advertise support for 2D-encoded data. Thus, rather than complicate the tests it was decided to use only 1D-encoded data. As shown in Table 1, a 2D-encoded test document is 12% smaller than a 1D-encoded version of the same data. This is significant and should be taken into account when evaluating a modem's capabilities since not all facsimile modems support the transmission of 2D-encoded data.

Page Size (bytes) Shrink 1D 2D --------------------------------------------------- 1/mixed text and graphics 135846 115289 .85 2/mostly text, some graphics 67857 52752 .78 3/single shaded graphic 105486 95354 .90 --------------------------------------------------- Total 299057 263395 .88 Table 1. Size of encoded facsimile data.

The sending site was located in Berkeley, California USA (+141552687xx) and the phone line was a normal unconditioned 2-wire hookup. [The phone numbers have been altered by replacing the last few digits with characters. The country and city/area codes have been retained.] The line is normally shared with a telephone, but for these tests it was isolated to avoid any possible interactions with the telephone set. All the tests were run at night or on the weekend in order to obtain lower utility rates.

2.1 Facsimile Modems Tested

The nine different Class 1 and Class 2 modems listed in Table 2 were used in the experiment. The modems were selected according to: availability, reputation, price, and correct functional operation with the FlexFAX software. Some modems were not tested because they fail to properly implement the draft Class 2 specification SP-2388-A of August 30, 1991. [In particular, the PM14400FX models from Practical Peripherals Inc. and the FM144V model from GVC do not implement the +FDIS command properly. These modems ignore the command after a connection is established, so it is not possible to set session parameters based on the remote facsimile machine's capabilities.] Manufacturer Model Class Firmware ---------------------------------------------------------------- AT&T Paradyne DataPort 14.4/FAX 1 C01.45.00 Digicom Systems Scout+ 1 3507/3506 Everex Systems EverFax 24/96D 2 911027 Multi Tech Systems MT1432BA 2 0110 Supra SupraFAXModem V.32bis 1, 2 V1.80-02 TR14-Jxxx-001 Telebit Systems T3000 2 T3000SA - Version LA7.02 Twincom 144/DF 1, 2 V1.270 TR14-Jxxx-001 Zero One Networking ZyXEL 1496E 2 V 6.01M US Robotics Sportster 1 Supervisor 4.1/DSP 10 Table 2. Facsimile modems tested.

It is critical to note the firmware revision that each modem was running during the testing. While some modem problems are due to hardware design limitations, many problems are caused by faulty firmware. Vendors routinely claim that problems that exist in one firmware revision are corrected in a subsequent version. This often is true, but when evaluating a modem one must always consider both the hardware platform and operating firmware together.

Class 2 modems differ from Class 1 modems in that more of the facsimile protocol is implemented in firmware in the modem. Class 1 modems provide four basic fax-specific operations: send/receive an HDLC protocol frame, and send/receive T.4-encoded message data. Class 2 modems do not expose these low-level calls, instead presenting higher-level interfaces that shield the host software from many of the timing critical aspects of the facsimile protocol. Class 1 modems place significant demands on the host computing resources while Class 2 modems are less sensitive to the performance of the host computer. Consequently Class 2 modems have been desirable for most UNIX systems. FlexFAX however supports both Class 1 and Class 2 modems and Class 1 modems function very well on some UNIX hosts. Modems that support both the Class 1 and Class 2 interfaces were used only in Class 2.

Table 3 shows the method by which the modems were connected to the host computer. All modems were run using RTS/CTS flow control whenever possible. Some modems were found to not correctly implement RTS/CTS flow control when sending facsimile or simply did not support hardware flow control for facsimile operation. Note that RTS/CTS flow control is not required by the Class 1 or Class 2 specifications. Using software flow control between a host and modem at rates above 9600 baud can be problematic.

Model DTE-DCE Rate Flow Control -------------------------------------------- AT&T DataPort 14.4/FAX 38400 RTS/CTS Digicom Scout+ 38400 RTS/CTS Everex 24/96D 19200 XON/XOFF Multi-Tech 1432BA 38400 RTS/CTS SupraFAXModem V.32bis 19200 XON/XOFF Telebit T3000 38400 RTS/CTS Twincom 14.4/DF 19200 XON/XOFF ZyXEL 1496E 38400 RTS/CTS USR Sportster 38400 RTS/CTS Table 3. Facsimile modem hardware configuration.

The Indigo serial port can operate up to 38.4 Kb/s without an external clock. All modems were run at the maximum possible rate. Many modems based on the Rockwell RC144DP lock the serial port rate to 19.2 Kb/s when sending or receiving facsimile; this is why the Supra and Twincom modems were run at 19.2 Kb/s. The Everex modem does not operate reliably at 38.4 Kb/s and so was run at 19.2 Kb/s.

2.2 Receiving Devices

Table 4 lists the devices that were used to receive facsimile during testing. Two different types of receivers are distinguished: a machine is a standalone facsimile machine that includes a scanner and printer, a modem is a standalone modem unit that interfaces to a computer and has no integral scanner or printer device. Destination BR 2D MST ECM Type Device -------------------------------------------------------------- +121282556xx 9600 Yes 20/10 Yes machine Panasonic Panafax PD-4200 +121522136xx 14400 Yes 0 modem SupraFAX v.32bis +130597073xx 9600 0 modem Hayes SmartFax +140328387xx 9600 0 modem Everex 24/96D +140874538xx 9600 Yes 10/5 Yes machine Fujitsu DEX 170 +141596576xx 9600 10 machine NEC BitFax +141623971xx 9600 Yes 20/10 machine OmniFax G66 +141636206xx 14400 Yes 10/5 Yes machine Xerox Telecopier 7033 +141696989xx 9600 Yes 10/5 machine Pitney-Bowes 9300 +150866368xx 14400 Yes 0 modem SupraFAX v.32bis +151084934xx 14400 Yes 0 modem ZyXEL 1496 +151292941xx 9600 20/10 machine XEROX Telecopier 7010 +151974615xx 9600 0 modem Telebit WorldBlazer +151974637xx 9600 Yes 10/5 Yes machine Lanier Model 120 +160278928yy 9600 Yes 10 Yes machine Ricoh Fax65 +160278928xx 9600 Yes 10/5 Yes machine Lanier FaxWriter 4000 +160278929xx 9600 0 modem Telebit WorldBlazer +160496057xx 14400 Yes 0 Yes machine Canon FAX-L785 +161723336xx 9600 0 modem Telebit WorldBlazer +161945506xx 9600 Yes 10 machine Ricoh 9200 +161954639xx 9600 Yes 20/10 Yes machine HP FAX-200 +170825003xx 14400 0 modem PPI PM14400FX +170836718xx 14400 Yes 0 modem ZyXEL 1496 +170889876xx 14400 Yes 0 modem USR Sportster +171864874xx 9600 20/10 machine Brother InstaFax 390 +181830557xx 4800 0 modem Intel SatisFaxion 100 +191858837xx 9600 Yes 10 Yes machine Ricoh FAX 3000L +41142227xx 14400 0 modem Multi-Tech 1432 +446127560xx 9600 Yes 0 Yes machine Canon FAX-L770 +446127561xx 14400 Yes 0 modem Zoom 14.4X +4681025xx 9600 Yes 10/5 Yes machine OKIFAX OF-110N +49428616xx 9600 20/10 machine GUIS ETFax 7 +495115613xx 9600 Yes 10/5 machine Canon 260E +495117178xx 14400 Yes 0 modem ZyXEL 1496 +4956180442xx 9600 20 machine Olivetti OFX 430 +4991318587xx 9600 0 modem Multi-Tech 1432 Table 4. Receiving devices and their capabilities.

There were 20 different facsimile machines and 17 modems, with 10 different modem models. The sites with modems were all running the FlexFAX software except for one site that was running Hayes SmartFax software on a 486 machine. The one non-FlexFAX site crashed early on in testing while receiving a facsimile and was unable to recover in time to participate in the majority of the testing. For this reason the tests include results for only 36 receivers: 20 machines and 16 modems.

Table 4 also lists some of the capabilities of the remote systems that can affect the results of the transmission tests. These capabilities are received by the sender in the initial facsimile protocol handshake. Several interesting trends can be observed from this data:

  1. Only 2 of the 20 facsimile machines support V.17 communication at 14400 b/s.
  2. Only 2 of the 20 facsimile machines support a minimum scanline time of 0 ms.
  3. Most facsimile machines support 2D-encoded data (15 of 20), but only 22 of 37 total receivers advertised themselves as capable of receiving 2D-encoded data.
  4. Only 10 of 37 receivers support the optional error correction mode (ECM); none of which are modem-based receivers.

Items 1 and 2 directly affect the transmission time of a facsimile in unavoidable ways. Obviously the higher the signaling rate the less time it takes to transfer the page data. Less obvious however is that using a V.17 modem can reduce other overhead in the protocol through the use of ``short training''; this is discussed in Section 3. The minimum scanline time is the minimum amount of time the receiver requires to process a row of pixels. Facsimile machines that immediately print received facsimile typically require a non-zero minimum scanline time so that they have time to mark and move the paper. 1D versus 2D encoding refers to the algorithm by which pixel data is encoded into a digital bit stream. 2D-encoding is more compact than 1D-encoding and is especially desirable for pages comprised of text and simple graphics. It is interesting that so many modems do not indicate a capability for the transfer of 2D-encoded data because doing so requires only trivial support in the modem firmware. [Unless the modem implements copy quality checking.] Item 4 is interesting from the perspective that using ECM can potentially reduce transmission time when poor line conditions are encountered. This issue was not explored in the experiment.

2.3 Test Methodology

Test data was collected by installing a modem on the sending machine, running the faxaddmodem script to configure the modem, and then submitting a facsimile job for each of the receivers. The FlexFAX software was then permitted to run normally until the queue was cleared. Each job was permitted five tries to reach the receiver before it was rejected. The FlexFAX software automatically rejects a job if three protocol errors occur in attempting to send the same page of a facsimile.

All communication was recorded using the standard FlexFAX session logging mechanism. Additional information logged via the system logging facility was also recorded as was the information recorded in the FlexFAX accounting file.

The participants were asked to collect the received facsimile and grade the quality of each page received based on a scale of 0-9, where 0 meant that a page was not received and 9 meant that a page was received completely and without any discernible errors. Some of the received facsimile were not available for this grading due to the usual circumstances associated with a communal facsimile device. For example, one person that was not involved with the experiment apparently decided there was no reason to keep duplicate pages and so discarded all but one good page for each of the pages that were received. Also some of the receivers were uncommunicative for various reasons (e.g. modems being turned off). This information was used only as a ``sanity check'' on the information collected at the sending site.

Note that the intent of the experiment was to simulate normal usage of FlexFAX. As such, manual intervention was minimized. The jobs were just entered into the system and permitted to run to completion.

3. Group 3 Facsimile Primer

In order to understand some of the test results it is important to have a basic understanding of the Group 3 facsimile protocol used for transmission. This section provides a simple primer on the subject. The reader is referred to [McConnell] and the [CCITTT.30] specification for more information. Other useful references are the [CCITTT.4] specification for a description of the data encoding algorithm, the EIA/TIA [Class1] specification for the Class 1 host-modem communication specification, and the draft Class 2 specification [SP-2388-A] for information on the Class 2 host-modem communication specification used by current Class 2 modems. [ A newly standardized version of this draft (``Class 2.0'') is slowly making its way into the marketplace. At this time however, there are no Class 2.0 modems. ]

Facsimile are transmitted through a fax modem device that accepts digital information, encodes it according to CCITT Recommendation T.4, and then modulates it to form an analog signal that is transmitted on a telephone line. Facsimile machines include, in addition to the modem and associated digital logic, a scanner that converts document pages to a digital form, and a printer that converts digital data to a printed form. Facsimile modems by themselves are not useful; they must either be combined with a computer or integrated into a facsimile machine.

Data is communicated between facsimile modems according to a set of rules specified in CCITT Recommendation T.30. These rules comprise what we call the Group 3 facsimile protocol. The protocol deals with negotiating acceptable characteristics for the transmitted page data (page width, page length, data encoding, output resolution, etc.) and the correct communication of the negotiated page data. In particular, the standard protocol includes an acknowledgement by the receiver that each page has been correctly received and is of acceptable quality.

The facsimile protocol actually operates using two different signaling schemes. A 300 b/s low speed carrier is used to carry handshaking information while a higher speed carrier is used to carry the T.4-encoded page data. The speed of the high speed carrier is determined during the handshaking procedure according to the capabilities of the sender and receiver and according to perceived quality of the telephone link between sender and receiver. All fax devices are required to support a V.27ter modem for sending page data at 4800 and 2400 b/s. Most devices also support a V.29 modem for transmitting page data at 7200 and 9600 b/s. Newer devices also support a V.17 modem which is a half-duplex trellis-coded modem that runs at 14400 b/s with fallback speeds of 12000, 9600, and 7200 b/s. Note that the techniques by which page data are transmitted by a facsimile modem are different from those of a normal data modem.

Figure 1. Group 3 transmit protocol (single page).

The basic flow of the Group 3 protocol is illustrated by the single page transfer shown in Figure 1. There are five phases of communication:

Phase A. Call establishment.
Phase B. Pre-message procedure.
Phase C. Message transmission.
Phase D. Post-message procedure.
Phase E. Call release.
In the following sections it is assumed that the sender and receiver interact using binary coded signaling rather than tonal procedures. The tonal mechanisms are required for backward compatibility with older facsimile equipment.

Phase A. Call establishment.

A facsimile call can be placed manually or automatically. We will not discuss manual call placement. With regard to automatic call establishment, the only thing to note is that the caller is required to transmit a CNG tone and the receiver is required to advertise its presence by transmitting a CED tone. The presence of a CNG tone is used by many combination facsimile and data modems to automatically determine if a caller is a facsimile modem, a data modem, or a voice caller. This automatic identification scheme is termed adaptive-answer in the Class 2 specification.

Phase B. Pre-message procedure.

In the pre-message stage of transmission the receiver identifies its capabilities by sending a Digital Identification Signal (DIS) at 300 b/s, after which the sender responds with a Digital Command Signal (DCS) specifying the set of capabilities to use for the message data that will follow in Phase C. The transmitter then sends a high speed training signal that is used to synchronize the two modems. A burst of data (1.5 seconds of zero data) is also transmitted in this procedure and the receiver is expected to analyze the received data and decide if the line quality is acceptable for transmission. If the receiver is satisfied with the signal quality it responds to the training with a Confirmation (CFR) message at 300 b/s and Phase C is entered. Otherwise, if the training failed, a Failure To Train (FTT) message is returned to the sender who has the option to continue or abort the call. The exact procedure to use when training fails is not part of the T.30 specification. Class 2 modems include this procedure as part of the firmware in the modem. Class 1 modems do not include this procedure; instead the host-based software is required to decide how to proceed. FlexFAX's Class 1 driver tries to do training twice at a given signaling rate before stepping down to a slower signaling rate. This corresponds to what many facsimile modems and machines do. Beware, however, that some Class 2 modems have firmware bugs that cause them to infinitely loop doing training under certain conditions.

Phase C. Message transmission.

The sender transmits a (usually short) training signal followed by a page of T.4-encoded image data. Data is sent on the high speed message carrier. The sender and receiver apply various timeouts to this process to insure that sender and receiver remain in step. For example, each scanline of data in a T.4-encoded image is punctuated by an End-Of-Line (EOL) code. Class 2 modems require that at least one EOL code be presented by the host every 5 seconds; otherwise the page transmission is aborted. The receiver places similar constraints on reception of page data.

There are two aspects to the image data transfer that are noteworthy. First, as part of the capabilities negotiation done in Phase B a minimum scanline time is selected. This value specifies the minimum amount of time that the receiver requires to process a row of image data. Receiving devices that immediately print a facsimile may need time to mark black on a page and advance the paper to the next row on the page. Senders must honor the requested minimum scanline time by zero-padding each scanline that so that it takes at least the requested time. Class 2 modems do this scanline padding automatically in the modem and only the raw scanline data must be transferred from the host. Class 1 modems typically do the zero-padding on the host and transfer the padded scanline data to the modem.

A second issue is the duration of the training signal sent prior to the page data. V.27 and V.29 modems use a longer training signal than V.17 modems. This also can affect the time spent in Phase C during a facsimile transmission.

Phase D. Post-message procedure.

The sender terminates a page of data transmitted on the high speed carrier with a Return To Control (RTC) code--six consecutive EOL codes. After RTC is transmitted the sender switches back to the low speed carrier and sends a Post-Page Message (PPM) to indicate whether this page is:

When the receiver receives RTC it listens on the low speed carrier for the post-page message and then responds with a message that acknowledges reception of the page or that indicates the page was received in error. In addition the Post-Page Response (PPR) returned by the receiver can request that the sender redo training if it believes that signal quality has degraded. Note that the sender is free to retrain at the same signaling rate or at a lower rate in response to a request to retrain.

The post-page handshake involves critical timing to insure the sender and receiver remain synchronized. If the sender does not receive a response to its post-page message within approximately 3 seconds it will retransmit the message. The sender will transmit the post-page message at most three times before aborting Phase D. Also, the delay between switching from the high speed message carrier to the low speed carrier is tricky. If this delay is made too short the receiver may miss the start of the post-page message and discard the data. If the delay is made too long the receiver may timeout waiting for a post-page message before the sender has had an opportunity to send its full complement of three messages. The timing characteristics of different facsimile devices varies widely in Phase D. Some machines are notorious for being slow to switch from the high speed message carrier to the low speed message carrier resulting in their only seeing the second or third transmit of the post-page message.

Phase E. Call release.

The caller sends a Disconnect (DCN) message and then both sides are free to hangup the line.

4. Test Results

In this section the test results are presented and analyzed from the perspective of the sender and the receiver. From the sender perspective, the modems are evaluated in terms of how well they communicate with the various receivers. Our examination of the receivers is done to look for trends in the performance of the receiving devices that may be used to understand anomalous results.

Results are summarized in a number of tables that have the following column headings: The Tries column gives the number of times that FlexFAX attempted to send a facsimile to the destination, while the Calls column indicates the number of times the modem reported that the receiver picked up the phone. These numbers are different when calls failed because the line was busy, because the receiver did not answer, or because the sending and receiving modems were unable to establish carrier. ConnTime is the total amount of time spent on the phone processing facsimile jobs--this number should be identical to the connect time charged by the phone company. Note that connect time does not include the time spent on the phone waiting for the phone to be answered. [ The software was configured to wait up to 60 seconds for carrier. Busy signals were recognized in significantly less than this time. Modems that reported ambiguous results for dialing were assumed to reach the receiver but be unsuccessful establishing carrier. ] The Pages column indicates the total number of pages that were successfully transmitted. There were initially 37 receivers but four were dropped from the experiment because of repeated failures. This left 33 receivers, so the total number of pages that a sender transmits successfully should be 3 x 33 = 99. (And each receiver should receive 3 pages from 10 different modems for 3 x 10 = 30 pages.) The final column marked TypRate is the most frequent signaling rate used to transmit page data.

4.1 Receiver Test Results

Tables 5a and 5b show the results of the transmission tests from the perspective of the receiving devices. Destination Tries Calls Pages TypRate ConnTime Device ---------------------------------------------------------------------- +121522136xx 26 15 24 14400 53:41 SupraFAX v.32bis +130597073xx 34 2 6 9600 25:55 Hayes SmartFax +140328387xx 10 10 30 9600 51:18 Everex 24/96D +150866368xx 17 17 24 9600 53:09 SupraFAX v.32bis +151084934xx 16 9 27 14400 40:11 ZyXEL 1496 +151974615xx 21 13 18 9600 1:00:16 Telebit WorldBlazer +160278929xx 41 7 3 9600 31:28 Telebit WorldBlazer +161723336xx 12 12 27 9600 56:13 Telebit WorldBlazer +170825003xx 10 10 27 14400 38:51 PPI PM14400FX +170836718xx 23 14 24 14400 47:07 ZyXEL 1496 +170889876xx 23 14 27 14400 47:30 USR Sportster +181830557xx 17 15 30 4800 2:06:17 Intel SatisFaxion 100 +358045529xx 2 0 0 9600 0:45 Multi-Tech 1432 +41142227xx 26 21 9 9600 59:03 Multi-Tech 1432 +446127561xx 23 14 18 9600 49:12 Zoom 14.4X +495117178xx 18 13 24 14400 49:26 ZyXEL 1496 +4991318587xx 18 13 24 9600 1:03:48 Multi-Tech 1432 Table 5a. Summary of transmission tests by receiver modem. The Ricoh Fax65 at +160278928yy and the Multi-Tech 1432 at +358045529xx were not called for all test runs for various reasons and their results have been discounted. The Telebit modem at +160278929xx also was dropped because it appeared to either be misconfigured or unavailable during most of the testing. As noted previously, the Hayes SmartFax system at +130597073xx crashed early on in the testing and did not respond to later calls. Otherwise, all devices were assumed to be functional and operating normally. Destination Tries Calls Pages TypRate ConnTime Device ---------------------------------------------------------------------- +121282556xx 11 11 30 9600 58:53 Panasonic Panafax PD-4200 +140874538xx 13 10 30 9600 57:20 Fujitsu DEX 170 +141596576xx 16 16 27 9600 1:08:47 NEC BitFax +141623971xx 12 12 29 9600 1:06:17 OmniFax G66 +141636206xx 11 11 30 9600 51:43 Xerox Telecopier 7033 +141696989xx 18 17 21 9600 1:07:58 Pitney-Bowes 9300 +151292941xx 11 10 30 9600 1:06:29 XEROX Telecopier 7010 +151974637xx 11 11 30 9600 57:15 Lanier Model 120 +160278928yy 3 1 1 9600 7:18 Ricoh Fax65 +160278928xx 11 11 30 9600 57:41 Lanier FaxWriter 4000 +160496057xx 11 11 30 14400 46:33 Canon FAX-L785 +161945506xx 12 12 28 9600 1:02:51 Ricoh 9200 +161954639xx 13 13 29 9600 1:03:20 HP FAX-200 +171864874xx 13 12 27 9600 53:52 Brother InstaFax 390 +191858837xx 15 10 27 9600 52:35 Ricoh FAX 3000L +446127560xx 21 13 20 9600 57:21 Canon FAX-L770 +4681025xx 12 11 30 9600 56:36 OKIFAX OF-110N +49428616xx 16 15 30 9600 1:12:18 GUIS ETFax 7 +495115613xx 10 10 30 9600 58:43 Canon 260E +4956180442xx 20 12 25 9600 1:02:50 Olivetti OFX 430 Table 5b. Summary of transmission tests by receiver machine.

Some observations can be made regarding the performance of the receiving devices:

  1. The facsimile machines functioned significantly better than the standalone facsimile modems. Of course it is difficult to fully assess this as the quality of the modems is questionable (especially when the results of this experiment are taken into account) and their operation is open to confusion--i.e. a facsimile machine is significantly easier to operate correctly than a modem. Nonetheless one should recognize that since a facsimile machine is a self-contained system that can be fully tested for correct operation, it is possible to achieve a higher degree of reliability than a standalone modem unit that functions with a variety of hardware and software systems.
  2. International phone calls were significantly more prone to failure. In 58% of the international calls, carrier was successfully established; compared to 71% of the continental calls (83% if we discount +130597073xx and +160278929xx).
Destination A B C[1] D[1] C[2] D[2] C[3] D[3] Total ------------------------------------------------------------------------------------- +121522136xx 20.8 4.2 1:23.7 7.4 42.1 7.2 1:06.5 6.7 3:58.6 +140328387xx 15.1 2.8 1:53.9 5.4 57.4 6.6 1:28.7 5.6 4:55.5 +150866368xx 21.7 5.1 1:30.3 6.0 46.5 6.2 1:11.9 6.2 4:13.9 +151084934xx 17.7 3.5 1:28.7 5.1 44.6 5.2 1:09.0 5.2 3:59.0 +151974615xx 14.4 3.3 1:53.8 5.9 57.6 6.2 1:29.0 6.2 4:56.4 +161723336xx 13.7 2.8 1:54.0 5.4 57.5 5.6 1:28.8 5.6 4:53.4 +170825003xx 17.6 2.9 1:30.4 5.6 45.4 5.7 1:10.2 5.8 4:03.6 +170836718xx 16.7 3.9 1:27.4 5.4 45.2 5.5 1:10.1 6.2 4:00.4 +170889876xx 13.5 3.0 1:30.2 5.1 45.2 5.3 1:10.0 5.3 3:57.6 +181830557xx 23.4 4.2 4:07.0 7.6 2:04.6 7.1 3:10.1 7.1 10:11.1 +41142227xx 25.0 8.6 1:22.1 12.9 49.6 9.5 1:17.5 9.9 4:35.1 +446127561xx 31.3 3.9 1:28.3 6.6 47.2 6.8 1:13.3 6.9 4:24.3 +495117178xx 29.7 4.1 1:27.6 5.4 45.2 5.6 1:10.1 5.6 4:13.3 +4991318587xx 23.0 5.6 1:54.0 5.0 1:12.4 5.0 1:20.3 5.5 5:10.8 Table 5c. Summary of protocol times by receiver modems. Destination A B C[1] D[1] C[2] D[2] C[3] D[3] Total ------------------------------------------------------------------------------------- +121282556xx 14.1 3.6 1:55.9 10.4 1:03.2 10.9 1:37.8 10.7 5:26.6 +140874538xx 19.0 3.0 1:56.1 6.0 1:03.5 5.8 1:35.9 5.9 5:15.2 +141596576xx 17.5 3.9 1:55.4 7.2 1:03.5 7.4 1:38.6 8.5 5:22.0 +141623971xx 20.2 4.0 2:12.7 7.0 1:11.3 7.6 1:48.9 6.5 5:58.2 +141636206xx 18.2 4.6 1:37.1 5.6 56.0 6.2 1:23.0 6.3 4:37.0 +141696989xx 20.5 5.5 1:55.0 9.4 1:04.0 7.4 1:37.0 7.6 5:26.4 +151292941xx 19.6 4.1 1:55.8 5.7 1:03.2 6.3 1:51.8 6.1 5:32.6 +151974637xx 14.7 3.2 1:56.2 10.7 1:04.4 10.5 1:35.9 6.1 5:21.7 +160278928xx 21.6 3.2 1:56.2 5.4 1:03.6 5.9 1:35.9 6.0 5:17.8 +160496057xx 25.0 3.5 1:27.8 5.0 44.6 5.2 1:09.0 5.2 4:05.3 +161945506xx 16.5 3.8 1:56.0 6.4 1:06.0 9.7 1:38.9 6.2 5:23.5 +161954639xx 18.4 3.6 1:56.0 11.0 1:03.8 8.7 1:38.3 11.1 5:30.9 +171864874xx 31.6 2.9 1:56.4 6.5 1:03.5 6.4 1:37.3 6.7 5:31.3 +191858837xx 12.8 3.1 1:54.4 3.2 1:04.0 4.1 1:40.2 3.5 5:05.3 +446127560xx 26.5 3.9 1:53.8 5.8 57.5 6.0 1:28.8 6.5 5:08.8 +4681025xx 19.1 4.0 1:56.2 5.6 1:03.5 6.0 1:35.9 6.2 5:16.5 Table 5d. Summary of protocol times by receiver machines.

Tables 5c and 5d show the average time spent in each phase of the Group 3 facsimile protocol, collated by receiver. Phases C and D are broken down by page: ``C[2]'' is the time spent in Phase C delivering page 2 of the test facsimile. The numbers were calculated from the timestamps recorded in the FlexFAX session log files. Only phases that were completed successfully were counted in the calculations. Phase E was not calculated because it is difficult to deduce the timing for Class 2 modems. Note that the per-phase timings are based on the time at which status messages were received by the host from the modem. If a modem, in particular a Class 2 modem, delays the delivery of a status message then these numbers will be skewed. Per-phase times are averaged over all the tests and given in ``minutes:seconds'' (beware that the calculation of the per-phase times effectively average transfer rates for V.29 and V.17 modems.) Appendix A shows the complete set of per-phase times for each modem and receiver.

4.2 Sender Test Results

Table 6 gives a synopsis of the transmission test results from the point of view of the sending modem. As described in section 4.1, only the data for 33 receivers was used. Modem Tries Calls ConnTime Pages TypRate ------------------------------------------------------------------- AT&T DataPort 14.4/FAX 36 35 3:10:50 99 9600 Multi-Tech 1432BA 36 34 2:48:55 96 9600 Everex 24/96D 51 36 3:38:33 93 9600 Digicom Scout+/V.17 54 45 3:13:29 90 9600 Zyxel 1496E 53 45 3:14:48 89 9600 SupraFAXModem V.32bis 52 49 3:23:29 87 9600 Twincom 14.4/DF 63 57 4:01:09 83 9600 Telebit T3000 50 39 2:48:02 81 9600 USR Sportster 50 45 3:07:44 81 9600 Digicom Scout+/V.29 72 32 2:48:25 67 9600 Table 6. Summary of transmission tests by sending modem.

There are two entries in Table 6 for the Digicom Scout+ modem. These entries correspond to the modem operating with two different versions of firmware: one that supports V.27, V.29, and V.17; and one that supports only V.27 and V.29. Section 4.3.2 explains the reason for the additional tests.

4.3 Sending Modem In-depth Results

The following sections present the individual results for each modem and explore some of the problems that were encountered. Each section contains a table similar to Table 6 that indicates only the receivers that required some unusual effort to complete the transmission. All receivers that are not listed in these tables received their test facsimile at the maximum signaling rate the sender and receiver were capable of, and without anomalous events.

4.3.1 AT&T DataPort 14.4/FAX

Table 7 shows the anomalous jobs for the AT&T DataPort. The first call to the Supra modem at +121522136xx failed to establish carrier; a subsequent call succeeded without a problem. The Telebit modem at +151974615xx failed on the first call to respond to the PPM sent after the first page; a subsequent call succeeded without a problem. Both Ricoh facsimile machines required page retransmissions at a lower signaling rate. The GUIS ETFax machine at +49428616xx required a second call to transfer the third page; the first call failed when the receiver did not acknowledge the EOP post-page message. The Multi-Tech modem at +4991318587xx took excessive time to transfer the 3-page document because each of the first two pages required retransmission (though at the same signaling rate). Destination Tries Calls ConnTime Pages TypRate Device -------------------------------------------------------------- +121522136xx 2 2 6:13 3 9600 SupraFAX v.32bis +151974615xx 2 2 7:27 3 9600 Telebit WorldBlazer +161945506xx 1 1 9:07 3 7200 Ricoh 9200 +191858837xx 1 1 8:16 3 7200 Ricoh FAX 3000L +49428616xx 2 2 8:02 3 9600 GUIS ETFax 7 +4991318587xx 1 1 8:08 3 9600 Multi-Tech 1432 Table 7. Transmission anomalies for DataPort 14.4/FAX.

4.3.2 Digicom Systems Scout+

Table 8 shows the anomalous jobs for the Digicom Scout+. The Scout+ had a decided problem establishing carrier with both modems and machines. Frequently the modem would return a +FCERROR status when calling, indicating that an incorrect carrier was encountered. For example, Oct 04 03:45:05.62: [ 1504]: DIAL 130597073xx Oct 04 03:45:05.62: [ 1504]: <-- [21:ATS24=2DT130597073xx@] Oct 04 03:45:40.53: [ 1504]: --> [8:+FCERROR] Oct 04 03:45:55.55: [ 1504]: --> [10:NO CARRIER] Oct 04 03:45:55.55: [ 1504]: SEND FAILED: No carrier detected This is the reason for many of the unsuccessful calls to the: Fujitsu DEX 170 at +140874538xx, Pitney-Bowes 9300 at +141696989xx, and XEROX Telecopier 7010 at +151292941xx. Destination Tries Calls ConnTime Pages TypRate Device -------------------------------------------------------------- +140874538xx 4 1 8:02 3 9600 Fujitsu DEX 170 +141696989xx 4 3 9:08 0 2400 Pitney-Bowes 9300 +150866368xx 2 2 6:14 3 14400 SupraFAX v.32bis +151292941xx 2 1 6:12 3 9600 XEROX Telecopier 7010 +170836718xx 2 2 4:39 3 14400 ZyXEL 1496 +170889876xx 5 2 6:46 3 14400 USR Sportster +181830557xx 3 2 13:16 3 4800 Intel SatisFaxion 100 +41142227xx 3 3 5:29 0 14400 Multi-Tech 1432 +446127561xx 3 3 6:49 0 2400 Zoom 14.4X +4956180442xx 2 2 12:42 3 4800 Olivetti OFX 430 +4991318587xx 2 2 7:25 3 9600 Multi-Tech 1432 Table 8. Transmission anomalies for Scout+ with V.17 support.

The calls to the Pitney-Bowes 9300 at +141696989xx and the Multi-Tech 1432 at +41142227xx failed because no response was received to the PPM sent after the first page. After three successive attempts to send a page FlexFAX will abort the job and return the untransmitted contents to the sender. The call to the Zoom 14.4X at +446127561xx failed because the two modems were unable to succeed in training.

The first call to the Supra modem at +150866368xx, the ZyXEL modem at 170836718xx, and the Sportster at +170889876xx, failed because of training; with the subsequent call going perfectly. Other calls failed either because of failure to train the modems or because of failure to receive a response to the post-page message for the first page. The call to the Olivetti OFX 430 at +4956180442xx took a long time because of retraining and having to make an extra call to retransmit the last page.

In general it appeared that the Scout+ performed poorly in the face of line noise and/or low signal quality. Discussions with informed parties confirmed this observation: firmware revision 3507/3506 has problems with line noise and poor signal quality. For this reason the tests were rerun with a previous version of the firmware that did not include V.17 support but which was thought to perform well otherwise. The results for firmware revision 2A19/2931 are shown in Table 8a. Problems with misrecognized carrier (+FCERROR) were encountered for many of the same receivers. Otherwise the older firmware did a better job of communicating with facsimile machines (as opposed to modems); but performed worse than the newer firmware that includes support for V.17.

Destination Tries Calls ConnTime Pages TypRate Device -------------------------------------------------------------- +121522136xx 5 0 3:29 0 0 SupraFAX v.32bis +151084934xx 5 0 1:30 0 0 ZyXEL 1496 +151974615xx 5 1 7:06 0 9600 Telebit WorldBlazer +170836718xx 5 0 2:51 0 0 ZyXEL 1496 +170889876xx 3 2 8:11 3 9600 USR Sportster +41142227xx 5 0 3:36 0 0 Multi-Tech 1432 +446127560xx 3 3 8:31 0 9600 Canon FAX-L770 +446127561xx 5 1 3:55 0 9600 Zoom 14.4X +495117178xx 5 1 4:48 0 2400 ZyXEL 1496 +4956180442xx 5 3 8:08 1 9600 Olivetti OFX 430 +4991318587xx 5 0 2:56 0 0 Multi-Tech 1432 Table 8a. Transmission anomalies for Scout+ without V.17 support.

4.3.3 Everex Systems EverFax 24/96D

The Everex modem is the oldest modem that was tested; in fact it is no longer sold. The older hardware design was seen in the lower transmission performance. Table 9 shows the anomalous jobs for the EverFax 24/96D. Destination Tries Calls ConnTime Pages TypRate Device -------------------------------------------------------------- +121522136xx 4 3 8:49 3 9600 SupraFAX v.32bis +141696989xx 2 2 9:57 3 9600 Pitney-Bowes 9300 +170836718xx 3 1 5:51 3 9600 ZyXEL 1496 +181830557xx 2 2 14:42 3 4800 Intel SatisFaxion 100 +191858837xx 5 0 0:00 0 0 Ricoh FAX 3000L +49428616xx 2 2 11:17 3 9600 GUIS ETFax 7 +4956180442xx 5 0 0:00 0 0 Olivetti OFX 430 Table 9. Transmission anomalies for EverFax 24/96D.

Typically when the modem was able to establish a connection communication was very reliable. Calls to the GUIS ETFax 7 at +49428616xx, the Pitney-Bowes 9300 at +141696989xx, and the Intel SatisFaxion 100 at +181830557xx required an additional call because the receiver failed to respond to a post-page message.

4.3.4 Multi Tech Systems MT1432BA

Table 10 shows the anomalous jobs for the MT1432BA. Calls to the Telebit modem at +151974615xx failed because no response was received to the post-page message; subsequent calls were not answered and the jobs were prematurely removed from the queue. The call to the Ricoh 9200 at +161945506xx required excessive time because the third page had to be retransmitted. The GUIS ETFax at +49428616xx required a second call to retransmit the third page; the first try failed when no acknowledgement was received to the post-page message. The Multi-Tech modem at +4991318587xx took extra time because the first page had to be retransmitted. Destination Tries Calls ConnTime Pages TypRate Device -------------------------------------------------------------- +151974615xx 2 1 3:22 0 9600 Telebit WorldBlazer +161945506xx 1 1 7:05 3 9600 Ricoh 9200 +49428616xx 2 2 8:11 3 9600 GUIS ETFax 7 +4991318587xx 1 1 6:32 3 9600 Multi-Tech 1432 Table 10. Transmission anomalies for MT1432BA.

4.3.5 Supra SupraFAXModem V.32bis

Table 11 shows the anomalous jobs for the Supra modem. There are some interesting results here. The OmniFax G66 at +141623971xx required extra time because the modem retrained down to 7200 baud from the requested 9600 baud. The XEROX Telecopier 7010 at +151292941xx took extra time because the third page had to be retransmitted. The Canon 260E at +495115613xx also required extra time because of retraining; though this time no page retransmission was requested. The remaining problems were reported as +FHNG: 51 or +FHNG: 52; the latter indicates ``No response to MPS repeated 3 times'' (i.e. no response to the MPS post-page message), while the former means ``RSPREC error/got DCN'', or disconnect was received when waiting for the post-page response. These sort of failures are indicative of a modem that does not do a good job handling line noise and poor signal quality. The Supra modem is based on the Rockwell RC144DP, a part that is considered to be susceptible to poor line conditions. Destination Tries Calls ConnTime Pages TypRate Device -------------------------------------------------------------- +141596576xx 2 2 8:17 3 9600 NEC BitFax +141623971xx 1 1 7:07 3 7200 OmniFax G66 +141636206xx 2 2 6:41 3 14400 Xerox Telecopier 7033 +150866368xx 2 2 4:48 3 14400 SupraFAX v.32bis +151292941xx 1 1 7:26 3 9600 XEROX Telecopier 7010 +160278928xx 2 2 8:41 3 9600 Lanier FaxWriter 4000 +160496057xx 2 2 6:11 3 14400 Canon FAX-L785 +170836718xx 3 3 5:28 0 14400 ZyXEL 1496 +170889876xx 5 3 2:39 0 14400 USR Sportster +181830557xx 2 2 14:31 3 4800 Intel SatisFaxion 100 +191858837xx 2 2 7:59 3 9600 Ricoh FAX 3000L +41142227xx 3 3 6:37 0 14400 Multi-Tech 1432 +446127561xx 2 2 6:26 3 14400 Zoom 14.4X +495115613xx 1 1 7:05 3 9600 Canon 260E +495117178xx 4 3 7:46 0 14400 ZyXEL 1496 Table 11. Transmission anomalies for SupraFAXModem V.32bis.

4.3.6 Telebit Systems T3000

Table 12 shows the anomalous jobs for the Telebit T3000. Several jobs were prematurely terminated which accounts for their having only three or four tries instead of the expected five. Calls to two different ZyXEL modems failed the first time to establish carrier but were successful on the second try--this may be related to the ZyXEL's adaptive-answer scheme (it is not known if all the ZyXEL modems involved were configured to use adaptive-answer). The Pitney-Bowes 9300 at +141696989xx failed because of a failure to receive a response to the MPS post-page message. Destination Tries Calls ConnTime Pages TypRate Device -------------------------------------------------------------- +121522136xx 4 0 3:36 0 9600 SupraFAX v.32bis +141696989xx 3 3 8:28 0 9600 Pitney-Bowes 9300 +171864874xx 4 3 2:28 0 9600 Brother InstaFax 390 +41142227xx 3 3 2:48 0 9600 Multi-Tech 1432 +446127560xx 4 0 2:36 0 9600 Canon FAX-L770 +4991318587xx 3 3 2:43 0 9600 Multi-Tech 1432 Table 12. Transmission anomalies for Telebit T3000.

The Pitney-Bowes 9300 at +141696989xx, the Multi-Tech 1432 at +41142227xx, and the Canon FAX-L770 at +446127560xx failed with what looks to be an oversight in the modem firmware. The modem sent a ``NO CARRIER'' result string without first sending a Class 2 ``+FHNG:'' result:

Oct 03 21:16:33.34: [19850]: <-- [6:AT+FDT] Oct 03 21:16:34.50: [19850]: --> [74:+FHT:FF C0 C2 04 04 04 04 ... Oct 03 21:16:34.50: [19850]: --> [26:+FHT:FF C8 C1 00 62 1F 00 ] Oct 03 21:16:39.93: [19850]: --> [74:+FHR:FF C0 02 0C CC EC 4C ... Oct 03 21:16:40.29: [19850]: --> [26:+FHR:FF C8 01 00 76 5F 00 ] Oct 03 21:16:41.31: [19850]: --> [74:+FHT:FF C0 C2 04 04 04 04 ... Oct 03 21:16:41.31: [19850]: --> [26:+FHT:FF C8 C1 00 62 1F 00 ] Oct 03 21:16:45.47: [19850]: --> [10:NO CARRIER] The Class 2 specification is not clear on whether or not this is permissible; about all that can be said is that the Telebit modem is unique in its operation in this respect.

The calls to the Brother InstaFax 390 at +171864874xx failed with another problem that is definitely a firmware bug:

Oct 03 23:32:40.14: [25845]: DIAL 171864874xx Oct 03 23:32:40.15: [25845]: <-- [15:ATDT171864874xx] Oct 03 23:33:11.98: [25845]: --> [74:+FHR:FF C0 02 9C 2C 2C EC ... Oct 03 23:33:12.27: [25845]: --> [26:+FHR:FF C8 01 00 72 1D 00 ] Oct 03 23:33:12.27: [25845]: --> [28:+FCSI:" 718 648 74XX"] The modem did not send the host the expected +FCON message that indicates a facsimile connection was established.

4.3.7 Twincom 14.4/DF

The Twincom modem was running old firmware. The vendor was contacted several weeks before these tests were run, but newer firmware was not available for testing. Table 13 shows the anomalous jobs for the Twincom modem. Like the Supra modem the Twincom is based on the Rockwell RC144DP; a component that is considered to be susceptible to poor line conditions. Destination Tries Calls ConnTime Pages TypRate Device -------------------------------------------------------------- +121522136xx 5 5 11:21 3 14400 SupraFAX v.32bis +141596576xx 2 2 8:55 3 9600 NEC BitFax +141623971xx 1 1 9:53 3 9600 OmniFax G66 +141696989xx 3 3 8:24 0 9600 Pitney-Bowes 9300 +150866368xx 3 3 4:09 0 14400 SupraFAX v.32bis +151292941xx 1 1 14:10 3 9600 XEROX Telecopier 7010 +151974615xx 3 3 10:27 0 9600 Telebit WorldBlazer +151974637xx 2 2 7:44 3 9600 Lanier Model 120 +161723336xx 3 3 10:22 0 9600 Telebit WorldBlazer +170825003xx 2 2 4:37 3 14400 PPI PM14400FX +170836718xx 3 3 5:34 3 14400 ZyXEL 1496 +170889876xx 3 1 3:37 3 14400 USR Sportster +181830557xx 1 1 18:11 3 7200 Intel SatisFaxion 100 +41142227xx 3 3 12:12 0 14400 Multi-Tech 1432 +446127560xx 5 1 5:29 2 9600 Canon FAX-L770 +4681025xx 2 2 7:48 3 9600 OKIFAX OF-110N +49428616xx 3 3 10:31 3 9600 GUIS ETFax 7 +495117178xx 2 2 4:57 3 14400 ZyXEL 1496 +4991318587xx 2 2 11:23 3 9600 Multi-Tech 1432 Table 13. Transmission anomalies for Twincom 14.4/DF.

A significant test was to the Multi-Tech 1432 at +41142227xx. Two different calls to this receiver left the modem in a state where it held carrier but did not respond to DTR. Manual intervention was required to hangup the line. This is a serious problem that alone makes the modem unacceptable for unmonitored operation.

The Pitney-Bowes 9300 at +141696989xx and the Supra at +150866368xx failed because there was no response to the post-page message. The Telebit modems at +151974615xx and +161723336xx, and the Multi-Tech 1432 at +41142227xx failed because the modem locked up during Phase B/C; for example:

Oct 03 18:58:09.40: [17733]: <-- [23:AT+FDIS=1,3,0,2,0,0,0,0] Oct 03 18:58:09.68: [17733]: --> [2:OK] Oct 03 18:58:09.68: [17733]: <-- [6:AT+FDT] Oct 03 19:01:09.68: [17733]: TIMEOUT: reading line from modem FlexFAX times out the AT+FDT command after three minutes to insure the sender does not leave the phone off hook due to a modem problem. This timeout is configurable and scaled according to the signaling rate used for the high speed message carrier. In practice three minutes is significantly longer than could possibly be needed to carry out the Phase B handshaking.

4.3.8 Zero One Networking ZyXEL 1496E

Table 14 shows the anomalous jobs for the ZyXEL 1496E. The ZyXEL firmware has a bad problem of not reporting useful hangup codes when problems occur. Virtually any problem is reported to the host as ``+FHNG: 50'' which means ``Unspecified Transmit Phase D error, including +FPHCTO timeout between data and +FET command.'' Consequently it is difficult to interpret many of the results. Destination Tries Calls ConnTime Pages TypRate Device -------------------------------------------------------------- +121282556xx 2 2 8:00 3 9600 Pan. Panafax PD-4200 +141596576xx 3 3 10:29 3 9600 NEC BitFax +141623971xx 3 3 10:24 2 9600 OmniFax G66 +150866368xx 2 2 5:12 3 14400 SupraFAX v.32bis +161945506xx 3 3 8:14 1 9600 Ricoh 9200 +161954639xx 4 4 12:20 2 9600 HP FAX-200 +41142227xx 3 3 6:56 0 14400 Multi-Tech 1432 +446127561xx 5 0 0:00 0 0 Zoom 14.4X Table 14. Transmission anomalies for ZyXEL 1496E.

The Panasonic Panafax PD-4200 at +121282556xx required an extra call because the response to the EOP was apparently not received. Unfortunately the modem does not indicate this directly but instead reports:

Oct 02 10:18:49.33: [ 579]: <-- [8:AT+FET=2] Oct 02 10:18:55.38: [ 579]: --> [10:NO CARRIER] without a hangup code. This is a firmware bug since the modem should present a +FHNG: status message.

Three tries were required to transmit the third page to the NEC BitFax at +141596576xx. The first try failed with the above problem, while the second try returned:

Oct 02 12:43:42.00: [ 579]: <-- [8:AT+FET=2] Oct 02 12:43:58.24: [ 579]: --> [8:+FHNG:50]

The OmniFax G66 at +141623971xx failed with problems similar to the above two jobs. The first two pages went through without a problem, but the third page was unsuccessful with either no hangup code or with hangup code 50.

The Supra modem at +150866368xx required two calls because the first failed with a ``Unspecified Transmit Phase B error'' (hangup code 20). Once again a better hangup code would help to understand what actually took place.

The Ricoh 9200 at +161945506xx failed after three attempts to send the second page. As above, a generic error code (50) was returned to the host.

The results are indicative of a modem that has problems with line noise. This is surprising since the modem is reputed to do well as data modem.

4.3.9 US Robotics Sportster

Table 15 shows the anomalous jobs for the Sportster modem. A significant test was to the Zoom modem at +446127561xx. On one occasion the modem entered a state where it held carrier but did not respond to DTR. Manual intervention was required to hangup the line (after over an hour of connect time!) This is a serious problem that alone makes the modem unacceptable for unmonitored operation. Destination Tries Calls ConnTime Pages TypRate Device -------------------------------------------------------------- +121522136xx 2 1 4:54 3 14400 SupraFAX v.32bis +141596576xx 3 3 7:49 0 9600 NEC BitFax +150866368xx 3 3 7:37 0 0 SupraFAX v.32bis +151974615xx 4 1 5:59 0 9600 Telebit WorldBlazer +181830557xx 4 3 17:52 3 4800 Intel SatisFaxion 100 +41142227xx 3 3 6:12 0 14400 Multi-Tech 1432 +446127560xx 3 3 8:22 0 9600 Canon FAX-L770 +446127561xx 3 3 7:41 0 14400 Zoom 14.4X +4991318587xx 1 1 6:06 3 9600 Multi-Tech 1432 Table 15. Transmission anomalies for USR Sportster.

The Supra modem at +121522136xx took a little extra time because of some handshaking messages had to be retransmitted. The NEC BitFax at +141596576xx failed when three consecutive attempts to send the first page were unsuccessful because no response was received to the post-page message. Communication with the Supra modem at +150866368xx failed because the modems were unable to establish training--specifically the sender never received a CFR/FTT response to training. The Telebit modem at +151974615xx failed when no response was received to the first post-page message; subsequent calls failed when no carrier was established. The Intel modem at +181830557xx required three tries to successfully transfer the first page, after which the remainder of the document went through without a problem. The Multi-Tech modem at +4991318587xx required extra time because the second page had to be retransmitted.

5. A Simple Metric

The data presented in Section 4 and Appendix A is too difficult for most people to understand and use in comparing facsimile modems. For better or worse, people want a single number that they can use to guide them in comparing the modems. Creating a single number is however problematic. The results of Section 4 indicate there are at least two distinct characteristics that are important in rating a modem: reliability and performance. Reliability refers to whether or not a modem accomplishes a requested task--in this case delivering the test facsimile document to a receiving device. Performance is a measure of how well the modem carries out the set of tasks it has been assigned. Some modems support V.27, V.29, and V.17, but do not handle poor line conditions well or have difficulties establishing carrier under certain conditions. These modems may perform well in limited use but not be as reliable as other modems. Other modems may excel under poor line conditions but not support the higher signaling rate of V.17. These modems should score high for reliability but may not rate as the highest performing modems available. Our measure for comparing modems rates reliability and performance separately, combining them afterwards for the purpose of defining a single quality rating.

The reliability of a modem, Rm, is given by Equation 5.1:

    Rm = # pages delivered / total # pages				(5.1)
Remember that a page is not considered delivered unless an acknowledgement is received from the receiving device. Note also that the reliability of a Class 1 modem is heavily dependent on the host software that implements the Group 3 facsimile protocol. This is much less true for Class 2 modems. In our tests the same software was used for all tests. We assume that it functions correctly and efficiently. The results tend to support this premise. When all pages are delivered, Rm reduces to:
    Rm = 1

The performance of a modem, Pm, is given Equation 5.2: .EQ I (5.2)

    Pm = Sum {r:Receivers} Sum {p:Phase A-Phase D} T(r,p)		(5.2)
where T(r,p) is the average time spent in phase p of the Group 3 protocol communicating with receiver r. If a modem does not successfully complete a phase then the average time for a modem of its capabilities is substituted. That is, if a modem that supports only V.27 and V.29 is unable to successfully transmit page 3 to a particular receiver, then the average time for similar modems for each unexecuted phase of the transfer is used when calculating Pm.

The above equations measure how well a modem does under error-free conditions. When a modem does not transmit a facsimile on the first call it should be penalized. Also, when a modem sends multiple copies of a page it should be penalized. Thus we define an error metric for a modem as Em:

    Em = Wep (# extra pages transmitted / maximum # extra pages) x	(5.3)
	 Wec (# extra calls / (.67 x maximum # extra calls))
where Wep and Wec are weights that reflect the relative importance of each term in Em. The .67 term in Equation 5.3 is present to balance the two terms because the maximum number of extra pages is 2 x 3 x 33 = 198 and the maximum number of extra calls is 4 x 33 = 132.

Given the above definitions, we define a quality rating Qm by:

    Qm = Wr x R m + Wp x (Tbase / Pm) + Em				(5.4)
.EN where Wr and Wp are weights that are used to reflect the relative importance of the Rm and Pm terms and Tbase is a baseline time that is combined with Wp to normalize the performance numbers to a range that is more consistent with the other terms.

For a modem that transmits all the test data without extraneous phone calls or page retransmissions the metric reduces to:

    Qm = Wr x 1 + Wp x (Tbase / Pm) - 0
       = Wr     + Wp x (Tbase / Pm)

So long as a modem is able to transmit all the test pages its rating is controlled by its performance metric Pm and any error costs. A failed call incurs cost through an extra phone call and possibly an extra page transmission. This cost appears in the error term Em. Modems are usually double-penalized for pages that are never successfully transmitted because the failed page appears both in Rm and indirectly in Em as extra calls and (probably) extraneous transmitted pages.

Table 16 shows the parameters for Qm derived from the test data. The performance metric Pm was calculated as described above. Because there were relatively few modems that supported V.29 but not V.17, the results for the Everex 24/96D were not used in calculating ``fill in'' entries for failed phases (since the Everex times were significantly skewed from the norm).

Modem Pages ExtraPages ExtraCalls Pm --------------------------------------------------------------- AT&T DataPort 14.4 99 4 3 2:48:55 Digicom Scout+/V.17 90 2 20 2:39:19 Everex 24/96D 93 3 6 3:27:41 Multi-Tech 1432BA 96 4 3 2:35:55 SupraFAXModem V.32bis 87 12 17 2:44:40 Telebit T3000 81 2 17 2:49:41 Twincom 14.4/DF 83 20 24 2:51:04 USR Sportster 81 11 17 2:33:59 ZyXEL 1496E 89 13 18 2:43:43 Table 16. Modem quality rating parameters.

To complete the calculations for the Qm we need to select a baseline time and values for the different weights. Choosing weights requires selecting ranges for the different factors that reflect desired characteristics of a ``good modem''. Our criteria weights reliability significantly higher than performance, with extra calls more important than extraneous pages. In addition we want quality metrics for a good modem to be around 100. A possible set of weights that meets our criteria are: Wr of 80, Tbase of 2:23:11, Wp of 16, Wep of 8, and Wec of 20. [The value of Tbase is based on an analysis of the T.4-encoded data, the average phase times, and the receiver capabilities. It represents the best expected time that a sender can do.] Using these weights we define a FaxStone to be:

    FaxStone[m] = Qm[Wr=80, Wp=16, Tbase=2:23:11, Wep=8, Wec=20]	(5.5)
	        = 80 x Rm + 16 (2:23:11 / Pm)
			  + [-8 (extra pages/132) - 20 (extra calls/132)]

This set of parameters has many good intuitive properties. The reliability term is in the range [0,80] while the performance term is in the approximate range [0,16]. This weights reliability over performance by a factor of five-as desired. The error metric is in the range [-28,0] with the majority of the error metrics in the range [-5,0]. The overall range for the quality rating is large enough to distinguish between modems but not so wide as to make comparisons difficult. Differences in modem reliability are distinguishable and it is possible to identify raw performance by looking solely at the Pm term. Clearly other weights may be selected in order to emphasize other performance characteristics of the modems.

Table 17 shows the FaxStones calculated from our test data. The columns marked R, P, and E are the weighted reliability, performance, and error terms, respectively.

Modem Class 2D BR Price R P E FaxStones ----------------------------------------------------------------------------------------- AT&T DataPort 14.4 1 Yes V.29 200 80.0 13.6 -0.84 92.7 Multi-Tech 1432BA 2 No V.17 450 77.6 14.7 -0.84 91.4 Everex 24/96D 2 Yes V.29 N/A 75.2 11.0 -1.48 84.7 Digicom Scout+/V.17 1 Yes V.17 185 72.7 14.4 -4.60 82.5 ZyXEL 1496E 2 Yes V.17 325 71.9 14.0 -4.60 81.3 SupraFAXModem V.32bis 1,2 Yes V.17 225 70.3 13.9 -4.33 79.9 USR Sportster 1 Yes V.17 185 65.5 14.9 -4.29 76.0 Telebit T3000 2 No V.29 600 65.5 13.5 -3.03 75.0 Twincom 14.4/DF 1,2 Yes V.17 225 67.1 13.4 -6.24 74.2 Table 17. Quality metric results for test modems.

6. Conclusions

Two modems were clearly the best performers: the AT&T DataPort 14.4/FAX and the Multi-Tech MT1432BA. The modems were extremely reliable. The DataPort lagged noticeably behind the Multi-Tech modem in performance because it did not support V.17, however this was not especially significant in the results because the majority of the receiving devices also did not support V.17.

The Everex 24/96D proved to be very reliable also, though it was unable to connect to several of the more difficult receivers. Its performance lagged significantly behind all the other modems. This is due to it being based on old hardware technology.

The remaining modems all rated fairly close together: Digicom Scout+, ZyXEL 1496E, USR Sportster, SupraFAXModem v.32bis, Telebit T3000, and Twincom 14.4/DF. The Scout+ had a significant problem establishing carrier with a variety of receivers. The ZyXEL 1496E was particularly disappointing because of the relatively low cost of the modem and its excellent reputation as a data modem. The test results however indicate that the firmware has problems and the modem may be susceptible to line noise during facsimile operation. The USR Sportster modem had the highest performance rating of any modem but rated very unreliable in testing. Also, as reported earlier, the Sportster had a serious firmware problem that makes unattended use uncertain. The Telebit modem had at least one firmware problem that when corrected will significantly raise the modem's FaxStone rating. This modem also appears to be one of the quickest modems and also appears to handle poor line conditions well. The Twincom 14.4/DF rated last in the tests; it also exhibited serious firmware problems.

Table 17 includes price estimates in US dollars for the modems that were tested. Given these prices, the best buy is the AT&T DataPort 14.4/FAX. The one complaint that can be made about the modem is that it does not support V.17. Since Telebit offers fax upgrades to existing T3000 and WorldBlazer modems for less than US$200 this may also be an excellent route to take for people that already own a Telebit modem, once the Telebit firmware problems are resolved. The Multi-Tech modem looks to be an excellent modem for people that require a Class 2 modem. The only complaint one can make about the MT1432BA is that it does not support 2D-encoded data (though this should be trivial to add to the firmware).

Future Work

This experiment used a very pragmatic approach to evaluating each modem's performance. Another valid approach is to test each modem using equipment that can simulate varying line conditions and protocol errors. This would be useful in understanding exactly how well the modems do under poor line conditions rather than guessing at such behaviour based on the way the modem operates.

Another worthwhile area of work is to get a better understanding of the value of the optional T.30 Error Correction Mode (ECM). ECM could only be used to communicate with the facsimile machines in this experiment, but it is potentially a worthwhile mechanism for improving the reliability of facsimile communication when poor line conditions are encountered.

Finally, the set of receivers used in this experiment were all located in highly industrialized areas of the world. It is important to recognize that other parts of the world may not have such high quality phone systems. In such areas devices such as echo-suppressors may have more of an effect on communication. The test results reported here are likely not to reflect how well the modems work under such conditions.

Acknowledgements

John Banning provided a lot of useful feedback and corrected my wayward thinking when constructing the various metrics. Ed McCreight helped understand many of the issues involved in comparing the modems.

The following people volunteered their time and equipment; their help is greatly appreciated.

Adrian Collins	    Bill Morrow		Brian Katzung	    C. Harald Koch
Carsten Koch	    Charlie Slater	Dan Busarow	    David Vrona
Dirk Husemann	    Heiner Meuser	Ian Darwin	    James Stansell
Jim Patterson	    Jochen Ruhland	John R. Koehring    John Schwegler
John Sellens	    John T. Kohl	Joseph E. Sacco	    L. Todd Masco
Larry Williamson    Lyndon Nerenberg	Martin Lichtin	    Paul Lupa
Peter Shipley	    Rickard Schoultz	Robert Weiner	    Vance Shipley

Certain vendors were kind enough to provide me with modems for use in testing; their assistance was critical: Digicom Systems (Wolfgang Henke), Multi-Tech (Pete Hanlon), Twincom (John O'Connell). and US Robotics (Pete Frankiewicz). Thanks also to Nick Doshi at Zero One Networking and Eric Ziegast of UUNET for providing firmware updates for existing modems.

Availability of Materials

FlexFAX version 2.2.1 is available by public ftp from the host sgi.com in the directory ~ftp/sgi/fax. The PostScript\(tm source files for the test document are available also by public ftp from sgi.com in the directory ~ftp/sgi/fax/bakeoff. A PostScript version of Appendix A is also available in this directory. The session logs are available to modem vendors by request.

References

[McConnell]
Kenneth R. McConnell, Dennis Bodson, and Richard Schaphorst, FAX: Digital Facsimile Technology and Applications; Artech House, 1992.
[CCITT T.30]
Procedures for Document Facsimile Transmission in the General Switched Telephone Network. CCITT Recommendation T.30
[CCITT T.4]
Standardization of Group 3 Facsimile Apparatus for Document Transmission.
[Class1]
EIA/TIA-578. Asynchronous Facsimile DCE Control Standard (Service Class 1). November 1990.
[SP-2388-A]
Asynchronous Facsimile DCE Control Standard (Service Class 2). October 30, 1991.

Appendix A - Detailed Group 3 Protocol Transfer Times

(This table is available separately; it was deleted to save space.)

Appendix B - Test Facsimile

(This material is available separately: cover page, page 1, and page 2; it was deleted to save space.)