Silicon Graphics Computer Systems
2011 N. Shoreline Blvd.
P.O. Box 7311
Mountain View, CA 94039-7311
ABSTRACT
This paper reports on an experiment to evaluate the
performance of a number of commonly used contemporary
facsimile modems.
Nine modems were used to
send a 3-page facsimile to 37 different
facsimile machines and modems located in North America and Europe.
The results are discussed and a metric is presented for comparing
the modems.
[This document is converted from the original paper and is
available in a
PostScript format
that is more condusive to
the presentation of figures, tables, and equations.]
This report is Copyright Sam Leffler and Silicon Graphics, Inc.
Permission to copy this
document for non-profit use is hereby granted without fee, provided
that (i) this document is reproduced unchanged and in its entirety,
(ii) the above copyright notices and this permission notice appear in
all copies, and (iii) the names of
Sam Leffler and Silicon Graphics may not be used in any advertising or
publicity relating to the document without the specific, prior written
permission of Sam Leffler and Silicon Graphics.
This paper reports on an experiment to evaluate the
reliability and performance of a number of commonly used facsimile modems.
Nine different modems were used with the
FlexFAX software package
to send a 3-page high resolution facsimile to
37 different facsimile devices around the world.
The ability of the modems to establish and hold a connection with the
remote device was studied along with the time required to successfully
transmit the facsimile data.
Based on the data collected from this experiment a simple
metric was developed for comparing the different facsimile modems.
This document is organized in six sections.
In
Section 2
the test environment and methodology is discussed.
Section 3
provides a primer on the communication protocol used by
Group 3 facsimile devices.
Section 4
discusses the test results, while
Section 5
presents the metric that was derived from this experiment.
In
Section 6
the conclusions that were drawn
from the experiment are presented together with
some advise to buyers of facsimile modems.
The basic experiment involved using FlexFAX version 2.2.0 to send a 3-page
facsimile to a variety of sites using nine different facsimile modems.
FlexFAX is a freely available software package that
supports a wide variety of modems--basically any modem that
conforms to the Class 1 or Class 2 host-modem communication
specifications.
Only sending was evaluated for several reasons.
First, the Group 3 facsimile protocol is designed so that the
sender is mainly responsible for the reliable transfer of facsimile data.
Personal experience indicates that a modem that does well at sending facsimile
also does well at receiving facsimile, while the reverse is not true.
Second, most uses of facsimile modems are significantly more
oriented toward sending than receiving facsimile.
Consequently modem buyers are more concerned with a modem's ability
to send than receive.
(It should be noted however that because of just this reason, some
modem vendors do not put as much effort into testing the
receive capability of their modems as they should.)
Finally, the logistics of running receive testing were too
difficult to surmount in the time available to do the experiment.
The experiment was run on a Silicon
Graphics Indigo machine under version 4.0.5H of the
IRIX
operating system.
[The system was modified in one important way; it included a bug fix to the
UART
driver to correct a problem with the handling of
DCD
on devices that require
RTS/CTS
flow control.
Without this bug fix the USR Sportster modem cannot be used with an
SGI
system.]
The 3-page facsimile shown in Appendix B was used for testing.
The document includes three different types of data, as shown in Table 1.
While many facsimile are either mostly text or simple graphics (e.g. charts
of tables), the selected matter represents a more difficult test of the
Group 3 encoding algorithms and, consequently, of the performance of the
sender and receiver.
The pages were encoded and transmitted exclusively using the
CCITT
Group 3 1D encoding algorithm.
FlexFAX also supports the 2D encoding algorithm but it was not used
because some fax machines have been observed to falsely
advertise support for 2D-encoded data.
Thus, rather than complicate the tests it was decided to use only
1D-encoded data.
As shown in Table 1, a 2D-encoded test document is 12% smaller than
a 1D-encoded version of the same data.
This is significant and should be taken into account when evaluating
a modem's capabilities since not all facsimile modems support
the transmission of 2D-encoded data.
Page Size (bytes) Shrink
1D 2D
---------------------------------------------------
1/mixed text and graphics 135846 115289 .85
2/mostly text, some graphics 67857 52752 .78
3/single shaded graphic 105486 95354 .90
---------------------------------------------------
Total 299057 263395 .88
Table 1. Size of encoded facsimile data.
The sending site was located in Berkeley, California USA (+141552687xx)
and the phone line was a normal unconditioned 2-wire hookup.
[The phone numbers have been altered by replacing the last few digits
with characters.
The country and city/area codes have been retained.]
The line is normally shared with a telephone, but for these tests
it was isolated to avoid any possible interactions
with the telephone set.
All the tests were run at night or on the weekend in order to obtain
lower utility rates.
The nine different Class 1 and Class 2 modems listed in Table 2
were used in the experiment.
The modems were selected according to: availability, reputation, price,
and correct functional operation with the FlexFAX software.
Some modems were not tested because they fail to properly
implement the draft Class 2 specification SP-2388-A of August 30, 1991.
[In particular, the
PM14400FX
models from Practical Peripherals Inc.
and the FM144V model from GVC do not implement the
+FDIS
command properly.
These modems ignore the command after a connection is
established, so it is not possible to set session parameters based
on the remote facsimile machine's capabilities.]
Manufacturer Model Class Firmware
----------------------------------------------------------------
AT&T Paradyne DataPort 14.4/FAX 1 C01.45.00
Digicom Systems Scout+ 1 3507/3506
Everex Systems EverFax 24/96D 2 911027
Multi Tech Systems MT1432BA 2 0110
Supra SupraFAXModem V.32bis 1, 2 V1.80-02 TR14-Jxxx-001
Telebit Systems T3000 2 T3000SA - Version LA7.02
Twincom 144/DF 1, 2 V1.270 TR14-Jxxx-001
Zero One Networking ZyXEL 1496E 2 V 6.01M
US Robotics Sportster 1 Supervisor 4.1/DSP 10
Table 2. Facsimile modems tested.
It is critical to note the firmware revision that each modem was
running during the testing.
While some modem problems are due to hardware design limitations,
many problems are caused by faulty firmware.
Vendors routinely claim that problems that exist in one firmware revision
are corrected in a subsequent version.
This often is true, but when evaluating a modem one must always
consider both the hardware platform and operating firmware together.
Class 2 modems differ from Class 1 modems in that more of the
facsimile protocol is implemented in firmware in the modem.
Class 1 modems provide four basic fax-specific
operations: send/receive an
HDLC
protocol frame, and
send/receive T.4-encoded message data.
Class 2 modems do not expose these low-level calls, instead
presenting higher-level interfaces that shield the host software
from many of the timing critical aspects of the facsimile protocol.
Class 1 modems place significant demands on the host
computing resources while Class 2 modems are less sensitive to
the performance of the host computer.
Consequently Class 2 modems have been desirable for most
UNIX
systems.
FlexFAX however supports both Class 1 and Class 2 modems and Class 1
modems function very well on some
UNIX
hosts.
Modems that support both the Class 1 and Class 2 interfaces were
used only in Class 2.
Table 3 shows the method by which the modems were connected to
the host computer.
All modems were run using
RTS/CTS
flow control whenever possible.
Some modems were found to not correctly implement
RTS/CTS
flow control
when sending facsimile or simply did not support
hardware flow control for facsimile operation.
Note that
RTS/CTS
flow control is not required by the Class 1 or Class 2 specifications.
Using software flow control between a host and modem at
rates above 9600 baud can be problematic.
The Indigo serial port can operate up to 38.4 Kb/s
without an external clock.
All modems were run at the maximum possible rate.
Many modems based on the Rockwell
RC144DP
lock the serial port rate
to 19.2 Kb/s when sending or receiving facsimile; this is why
the Supra and Twincom modems were run at 19.2 Kb/s.
The Everex modem does not operate reliably at 38.4 Kb/s and so
was run at 19.2 Kb/s.
Table 4 lists the devices that were used to receive
facsimile during testing.
Two different types of receivers are distinguished:
a machine is a standalone facsimile machine that
includes a scanner and printer, a modem is a standalone
modem unit that interfaces to a computer and has no integral
scanner or printer device.
Destination BR 2D MST ECM Type Device
--------------------------------------------------------------
+121282556xx 9600 Yes 20/10 Yes machine Panasonic Panafax PD-4200
+121522136xx 14400 Yes 0 modem SupraFAX v.32bis
+130597073xx 9600 0 modem Hayes SmartFax
+140328387xx 9600 0 modem Everex 24/96D
+140874538xx 9600 Yes 10/5 Yes machine Fujitsu DEX 170
+141596576xx 9600 10 machine NEC BitFax
+141623971xx 9600 Yes 20/10 machine OmniFax G66
+141636206xx 14400 Yes 10/5 Yes machine Xerox Telecopier 7033
+141696989xx 9600 Yes 10/5 machine Pitney-Bowes 9300
+150866368xx 14400 Yes 0 modem SupraFAX v.32bis
+151084934xx 14400 Yes 0 modem ZyXEL 1496
+151292941xx 9600 20/10 machine XEROX Telecopier 7010
+151974615xx 9600 0 modem Telebit WorldBlazer
+151974637xx 9600 Yes 10/5 Yes machine Lanier Model 120
+160278928yy 9600 Yes 10 Yes machine Ricoh Fax65
+160278928xx 9600 Yes 10/5 Yes machine Lanier FaxWriter 4000
+160278929xx 9600 0 modem Telebit WorldBlazer
+160496057xx 14400 Yes 0 Yes machine Canon FAX-L785
+161723336xx 9600 0 modem Telebit WorldBlazer
+161945506xx 9600 Yes 10 machine Ricoh 9200
+161954639xx 9600 Yes 20/10 Yes machine HP FAX-200
+170825003xx 14400 0 modem PPI PM14400FX
+170836718xx 14400 Yes 0 modem ZyXEL 1496
+170889876xx 14400 Yes 0 modem USR Sportster
+171864874xx 9600 20/10 machine Brother InstaFax 390
+181830557xx 4800 0 modem Intel SatisFaxion 100
+191858837xx 9600 Yes 10 Yes machine Ricoh FAX 3000L
+41142227xx 14400 0 modem Multi-Tech 1432
+446127560xx 9600 Yes 0 Yes machine Canon FAX-L770
+446127561xx 14400 Yes 0 modem Zoom 14.4X
+4681025xx 9600 Yes 10/5 Yes machine OKIFAX OF-110N
+49428616xx 9600 20/10 machine GUIS ETFax 7
+495115613xx 9600 Yes 10/5 machine Canon 260E
+495117178xx 14400 Yes 0 modem ZyXEL 1496
+4956180442xx 9600 20 machine Olivetti OFX 430
+4991318587xx 9600 0 modem Multi-Tech 1432
Table 4. Receiving devices and their capabilities.
There were 20 different facsimile machines
and 17 modems, with 10 different modem models.
The sites with modems were all running the FlexFAX software
except for one site
that was running Hayes SmartFax software on a 486 machine.
The one non-FlexFAX site crashed early on in testing while
receiving a facsimile and was unable to recover in time to
participate in the majority of the testing.
For this reason the tests include results for only 36 receivers:
20 machines and 16 modems.
Table 4 also lists some of the capabilities of the remote systems that
can affect the results of the transmission tests.
These capabilities are received by the sender in the
initial facsimile protocol handshake.
Several interesting trends can be observed from this data:
Only 2 of the 20 facsimile machines support V.17 communication at 14400 b/s.
Only 2 of the 20 facsimile machines support a minimum scanline time of 0 ms.
Most facsimile machines support 2D-encoded data (15 of 20), but only
22 of 37 total receivers advertised themselves as capable of receiving
2D-encoded data.
Only 10 of 37 receivers support the optional error correction mode
(ECM); none of which are modem-based receivers.
Items 1 and 2 directly affect the transmission time of a facsimile in
unavoidable ways.
Obviously the higher the signaling rate the less time it takes to
transfer the page data.
Less obvious however is that using a V.17 modem can reduce other
overhead in the protocol through the use of ``short training'';
this is discussed in Section 3.
The minimum scanline time is the minimum amount of time the receiver
requires to process a row of pixels.
Facsimile machines that immediately print received facsimile typically
require a non-zero minimum scanline time so that they have time to
mark and move the paper.
1D versus 2D encoding refers to the algorithm by which pixel data is
encoded into a digital bit stream.
2D-encoding is more compact than 1D-encoding and is
especially desirable for pages comprised of text and simple graphics.
It is interesting that so many modems do not indicate a capability for
the transfer of 2D-encoded data because doing so requires only trivial
support in the modem firmware.
[Unless the modem implements copy quality checking.]
Item 4 is interesting from the perspective that using
ECM
can potentially
reduce transmission time when poor line conditions are encountered.
This issue was not explored in the experiment.
Test data was collected by installing a modem on the sending machine,
running the
faxaddmodem
script to configure the modem, and then submitting a facsimile job for
each of the receivers.
The FlexFAX software was then permitted to run normally
until the queue was cleared.
Each job was permitted five tries to reach the receiver before it was
rejected.
The FlexFAX software automatically rejects a job if three
protocol errors occur in attempting to send the same page of a facsimile.
All communication was recorded using the standard FlexFAX session
logging mechanism.
Additional information logged via the system logging facility was also
recorded as was the information recorded in the FlexFAX accounting file.
The participants were asked to collect the received facsimile and
grade the quality of each page received based on a scale of 0-9,
where 0 meant that a page was not received and 9 meant that a
page was received completely and without any discernible errors.
Some of the received facsimile were not available for this grading due
to the usual circumstances associated with a communal facsimile device.
For example, one person that was not involved with the experiment
apparently decided there was no reason to keep duplicate pages and so
discarded all but one good page for each of the pages that were received.
Also some of the receivers were uncommunicative for various reasons
(e.g. modems being turned off).
This information was used only as a ``sanity check'' on the information
collected at the sending site.
Note that the intent of the experiment was to simulate normal usage
of FlexFAX. As such, manual intervention was minimized.
The jobs were just entered into the system and permitted to run to completion.
In order to understand some of the test results it is important
to have a basic understanding of the Group 3 facsimile protocol used
for transmission.
This section provides a simple primer on the subject.
The reader is referred to [McConnell] and the
[CCITTT.30]
specification for more information.
Other useful references are the
[CCITTT.4]
specification for a description
of the data encoding algorithm, the
EIA/TIA
[Class1] specification for
the Class 1 host-modem communication specification, and
the draft Class 2 specification [SP-2388-A] for
information on the Class 2 host-modem communication specification
used by current Class 2 modems.
[
A newly standardized version of this draft (``Class 2.0'')
is slowly making its way into the marketplace.
At this time however, there are no Class 2.0 modems.
]
Facsimile are transmitted through a fax modem device that accepts digital
information, encodes it according to
CCITT
Recommendation T.4, and then
modulates it to form an analog signal that is transmitted on a telephone line.
Facsimile machines include, in addition to the modem and associated
digital logic, a scanner that converts document pages to a digital
form, and a printer that converts digital data to a printed form.
Facsimile modems by themselves are not useful; they must either be combined
with a computer or integrated into a facsimile machine.
Data is communicated between facsimile modems according to a set of rules
specified in
CCITT
Recommendation T.30.
These rules comprise what we call the Group 3 facsimile protocol.
The protocol deals with negotiating acceptable characteristics for the
transmitted page data (page width, page length, data encoding,
output resolution, etc.) and the correct communication of the negotiated
page data.
In particular, the standard protocol includes an acknowledgement
by the receiver that each page has been correctly received and is of
acceptable quality.
The facsimile protocol actually operates using two different signaling
schemes.
A 300 b/s low speed carrier is used to carry handshaking information
while a higher speed carrier is used to carry the T.4-encoded page data.
The speed of the high speed carrier is determined during the handshaking
procedure according to the capabilities of the sender and receiver and
according to perceived quality of the telephone link between sender
and receiver.
All fax devices are required to support a V.27ter modem
for sending page data at 4800 and 2400 b/s.
Most devices also support a V.29 modem for transmitting page data
at 7200 and 9600 b/s.
Newer devices also support a V.17 modem which is a half-duplex trellis-coded
modem that runs at 14400 b/s with fallback speeds of 12000, 9600, and 7200 b/s.
Note that the techniques by which page data are transmitted by
a facsimile modem are different from those of a normal data modem.
The basic flow of the Group 3 protocol is illustrated by
the single page transfer shown in Figure 1.
There are five phases of communication:
Phase A. Call establishment.
Phase B. Pre-message procedure.
Phase C. Message transmission.
Phase D. Post-message procedure.
Phase E. Call release.
In the following sections it is assumed that the sender and receiver
interact using binary coded signaling rather than tonal procedures.
The tonal mechanisms are required for backward compatibility with older
facsimile equipment.
Phase A. Call establishment.
A facsimile call can be placed manually or automatically.
We will not discuss manual call placement.
With regard to automatic call establishment, the only thing
to note is that the caller is required
to transmit a
CNG
tone and the receiver is required to
advertise its presence by transmitting a
CED
tone.
The presence of a
CNG
tone is used by many combination facsimile
and data modems to automatically determine if a caller is a facsimile
modem, a data modem, or a voice caller.
This automatic identification scheme is termed adaptive-answer
in the Class 2 specification.
Phase B. Pre-message procedure.
In the pre-message stage of transmission the receiver identifies its
capabilities by sending a Digital Identification Signal (DIS)
at 300 b/s, after which the sender responds
with a Digital Command Signal (DCS) specifying the set of
capabilities to use for the
message data that will follow in Phase C.
The transmitter then sends a high speed training signal that is used
to synchronize the two modems.
A burst of data (1.5 seconds of zero data) is also transmitted in
this procedure and the receiver is expected to analyze the received
data and decide if the line quality is acceptable for transmission.
If the receiver is satisfied with the signal quality it responds to
the training with a Confirmation (CFR) message at 300 b/s and Phase C
is entered.
Otherwise, if the training failed, a Failure To Train (FTT) message
is returned to the sender who has the option to continue or abort the call.
The exact procedure to use when training fails is not part of the T.30
specification.
Class 2 modems include this procedure as part of the firmware in the
modem.
Class 1 modems do not include this procedure; instead the host-based
software is required to decide how to proceed.
FlexFAX's Class 1 driver tries to do training twice
at a given signaling rate
before stepping down to a slower signaling rate.
This corresponds to what many facsimile modems and machines do.
Beware, however, that some Class 2 modems have firmware bugs that cause
them to infinitely loop doing training under certain conditions.
Phase C. Message transmission.
The sender transmits a (usually short) training signal followed
by a page of T.4-encoded image data.
Data is sent on the high speed message carrier.
The sender and receiver apply various timeouts to this process to
insure that sender and receiver remain in step.
For example, each scanline of data in a T.4-encoded image is punctuated
by an End-Of-Line (EOL) code.
Class 2 modems require that at least one
EOL
code be presented by the
host every 5 seconds; otherwise the page transmission is aborted.
The receiver places similar constraints on reception of page data.
There are two aspects to the image data transfer that are noteworthy.
First, as part of the capabilities negotiation done in Phase B a
minimum scanline time is selected.
This value specifies the minimum amount of time that the receiver requires
to process a row of image data.
Receiving devices that immediately print a facsimile may need time to
mark black on a page and advance the paper to the next row on the page.
Senders must honor the requested minimum scanline time by zero-padding
each scanline that so that it takes at least the requested time.
Class 2 modems do this scanline padding automatically in the modem and
only the raw scanline data must be transferred from the host.
Class 1 modems typically do the zero-padding on the host and transfer
the padded scanline data to the modem.
A second issue is the duration of the training signal sent prior to the
page data.
V.27 and V.29 modems use a longer training signal than V.17 modems.
This also can affect the time spent in Phase C during a facsimile
transmission.
Phase D. Post-message procedure.
The sender terminates a page of data transmitted on the high speed carrier
with a Return To Control (RTC) code--six consecutive
EOL
codes.
After
RTC
is transmitted the sender switches back to the low speed carrier
and sends a Post-Page Message (PPM) to indicate whether this page is:
followed by another page in the same document (MPS),
followed by more pages in a new document (EOM), or
the last page in the last document (EOP).
When the receiver receives
RTC
it listens on the low speed carrier for
the post-page message and then responds with a message that acknowledges
reception of the page or that indicates the page was received in error.
In addition the Post-Page Response (PPR) returned by the receiver can
request that the sender redo training if it believes that signal quality
has degraded.
Note that the sender is free to retrain at the same signaling rate or at
a lower rate in response to a request to retrain.
The post-page handshake involves critical timing to insure the
sender and receiver remain synchronized.
If the sender does not receive a response to its
post-page message within approximately 3 seconds
it will retransmit the message.
The sender will transmit the post-page message
at most three times before aborting Phase D.
Also, the delay between switching from the high speed message carrier to
the low speed carrier is tricky.
If this delay is made too short the receiver may miss the start of the
post-page message and discard the data.
If the delay is made too long the receiver may timeout waiting for a
post-page message
before the sender has had an opportunity to send its full complement of
three messages.
The timing characteristics of different facsimile devices varies widely
in Phase D.
Some machines are notorious for being slow to switch from the high speed
message carrier to the low speed message carrier resulting in their only
seeing the second or third transmit of the post-page message.
Phase E. Call release.
The caller sends a Disconnect (DCN)
message and then both sides are free
to hangup the line.
In this section the test results are presented and
analyzed from the perspective of the sender and the receiver.
From the sender perspective, the modems are evaluated
in terms of how well they communicate with the various receivers.
Our examination of the receivers is done to look for trends in the
performance of the receiving devices that may be used to
understand anomalous results.
Results are summarized in a number of tables that have the following
column headings:
The Tries column gives the number of times that FlexFAX attempted
to send a facsimile to the destination, while
the Calls column indicates the number of times the modem reported that
the receiver picked up the phone.
These numbers are different when calls failed because the line was busy,
because the receiver did not answer,
or because the sending and receiving modems were unable to establish carrier.
ConnTime is the total amount of time spent on the phone processing
facsimile jobs--this number should be identical to the connect time
charged by the phone company.
Note that connect time does not include the time spent on the phone waiting
for the phone to be answered.
[
The software was configured to wait up to 60 seconds for carrier.
Busy signals were recognized in significantly less than this time.
Modems that reported ambiguous results for dialing were assumed to
reach the receiver but be unsuccessful establishing carrier.
]
The Pages column indicates the total number of pages that were
successfully transmitted.
There were initially 37 receivers but four were dropped from the experiment
because of repeated failures.
This left 33 receivers, so the total number of pages that a sender
transmits successfully should be 3 x 33 = 99.
(And each receiver should receive 3 pages from 10 different modems for
3 x 10 = 30 pages.)
The final column marked TypRate is the most frequent signaling
rate used to transmit page data.
Tables 5a and 5b show the results of the transmission tests
from the perspective of the receiving devices.
Destination Tries Calls Pages TypRate ConnTime Device
----------------------------------------------------------------------
+121522136xx 26 15 24 14400 53:41 SupraFAX v.32bis
+130597073xx 34 2 6 9600 25:55 Hayes SmartFax
+140328387xx 10 10 30 9600 51:18 Everex 24/96D
+150866368xx 17 17 24 9600 53:09 SupraFAX v.32bis
+151084934xx 16 9 27 14400 40:11 ZyXEL 1496
+151974615xx 21 13 18 9600 1:00:16 Telebit WorldBlazer
+160278929xx 41 7 3 9600 31:28 Telebit WorldBlazer
+161723336xx 12 12 27 9600 56:13 Telebit WorldBlazer
+170825003xx 10 10 27 14400 38:51 PPI PM14400FX
+170836718xx 23 14 24 14400 47:07 ZyXEL 1496
+170889876xx 23 14 27 14400 47:30 USR Sportster
+181830557xx 17 15 30 4800 2:06:17 Intel SatisFaxion 100
+358045529xx 2 0 0 9600 0:45 Multi-Tech 1432
+41142227xx 26 21 9 9600 59:03 Multi-Tech 1432
+446127561xx 23 14 18 9600 49:12 Zoom 14.4X
+495117178xx 18 13 24 14400 49:26 ZyXEL 1496
+4991318587xx 18 13 24 9600 1:03:48 Multi-Tech 1432
Table 5a. Summary of transmission tests by receiver modem.
The Ricoh Fax65 at +160278928yy and the Multi-Tech 1432 at +358045529xx
were not called for all test runs
for various reasons and their results have been discounted.
The Telebit modem at +160278929xx also was dropped because it appeared
to either be misconfigured or unavailable during most of the testing.
As noted previously, the Hayes SmartFax system at +130597073xx crashed
early on in the testing and did not respond to later calls.
Otherwise, all devices were assumed to be functional and operating normally.
Destination Tries Calls Pages TypRate ConnTime Device
----------------------------------------------------------------------
+121282556xx 11 11 30 9600 58:53 Panasonic Panafax PD-4200
+140874538xx 13 10 30 9600 57:20 Fujitsu DEX 170
+141596576xx 16 16 27 9600 1:08:47 NEC BitFax
+141623971xx 12 12 29 9600 1:06:17 OmniFax G66
+141636206xx 11 11 30 9600 51:43 Xerox Telecopier 7033
+141696989xx 18 17 21 9600 1:07:58 Pitney-Bowes 9300
+151292941xx 11 10 30 9600 1:06:29 XEROX Telecopier 7010
+151974637xx 11 11 30 9600 57:15 Lanier Model 120
+160278928yy 3 1 1 9600 7:18 Ricoh Fax65
+160278928xx 11 11 30 9600 57:41 Lanier FaxWriter 4000
+160496057xx 11 11 30 14400 46:33 Canon FAX-L785
+161945506xx 12 12 28 9600 1:02:51 Ricoh 9200
+161954639xx 13 13 29 9600 1:03:20 HP FAX-200
+171864874xx 13 12 27 9600 53:52 Brother InstaFax 390
+191858837xx 15 10 27 9600 52:35 Ricoh FAX 3000L
+446127560xx 21 13 20 9600 57:21 Canon FAX-L770
+4681025xx 12 11 30 9600 56:36 OKIFAX OF-110N
+49428616xx 16 15 30 9600 1:12:18 GUIS ETFax 7
+495115613xx 10 10 30 9600 58:43 Canon 260E
+4956180442xx 20 12 25 9600 1:02:50 Olivetti OFX 430
Table 5b. Summary of transmission tests by receiver machine.
Some observations can be made regarding the performance of the
receiving devices:
The facsimile machines functioned significantly better than the
standalone facsimile modems.
Of course it is difficult to fully assess this as the quality of the modems
is questionable (especially when the results of this experiment are
taken into account) and their operation is open to confusion--i.e. a
facsimile machine is significantly easier to operate correctly than a modem.
Nonetheless one should recognize that since a facsimile machine is
a self-contained system that can be fully tested for correct operation,
it is possible to achieve a higher degree of reliability than a
standalone modem unit that functions with a variety of hardware
and software systems.
International phone calls were significantly more prone to failure.
In 58% of the international calls, carrier was successfully
established; compared to 71% of the continental calls
(83% if we discount +130597073xx and +160278929xx).
Tables 5c and 5d show the average time spent in each phase
of the Group 3 facsimile protocol, collated by receiver.
Phases C and D are broken down by page: ``C[2]'' is the time spent in Phase
C delivering page 2 of the test facsimile.
The numbers were calculated from the timestamps recorded in the
FlexFAX session log files.
Only phases that were completed successfully were
counted in the calculations.
Phase E was not calculated because it is difficult to deduce
the timing for Class 2 modems.
Note that the per-phase timings are based
on the time at which status messages were received by the host from
the modem.
If a modem, in particular a Class 2 modem, delays the delivery
of a status message then these numbers will be skewed.
Per-phase times are averaged over all the tests and
given in ``minutes:seconds''
(beware that the calculation of the
per-phase times effectively average transfer rates for
V.29
and
V.17
modems.)
Appendix A shows the complete set of
per-phase times for each modem and receiver.
Table 6 gives a synopsis of the transmission test results from the
point of view of the sending modem.
As described in section 4.1, only the data for 33 receivers was used.
Modem Tries Calls ConnTime Pages TypRate
-------------------------------------------------------------------
AT&T DataPort 14.4/FAX 36 35 3:10:50 99 9600
Multi-Tech 1432BA 36 34 2:48:55 96 9600
Everex 24/96D 51 36 3:38:33 93 9600
Digicom Scout+/V.17 54 45 3:13:29 90 9600
Zyxel 1496E 53 45 3:14:48 89 9600
SupraFAXModem V.32bis 52 49 3:23:29 87 9600
Twincom 14.4/DF 63 57 4:01:09 83 9600
Telebit T3000 50 39 2:48:02 81 9600
USR Sportster 50 45 3:07:44 81 9600
Digicom Scout+/V.29 72 32 2:48:25 67 9600
Table 6. Summary of transmission tests by sending modem.
There are two entries in Table 6 for the Digicom Scout+ modem.
These entries correspond to the modem operating with two different
versions of firmware: one that supports V.27, V.29, and V.17;
and one that supports only V.27 and V.29.
Section 4.3.2 explains the reason for the additional tests.
The following sections present the individual results for each modem
and explore some of the problems that were encountered.
Each section contains a table similar to Table 6 that indicates
only the receivers that required some unusual effort to complete
the transmission.
All receivers that are not listed in these tables received their test
facsimile at the maximum signaling rate the sender
and receiver were capable of, and without anomalous events.
4.3.1 AT&T DataPort 14.4/FAX
Table 7 shows the anomalous jobs for the AT&T DataPort.
The first call to the Supra modem at +121522136xx failed to establish
carrier; a subsequent call succeeded without a problem.
The Telebit modem at +151974615xx failed on the first call to respond to
the PPM sent after the first page; a subsequent call succeeded without
a problem.
Both Ricoh facsimile machines required page retransmissions at a lower
signaling rate.
The GUIS ETFax machine at +49428616xx required a second call to transfer
the third page; the first call failed when the receiver did
not acknowledge the
EOP
post-page message.
The Multi-Tech modem at +4991318587xx took excessive time to transfer
the 3-page document because each of the first two pages required
retransmission (though at the same signaling rate).
Destination Tries Calls ConnTime Pages TypRate Device
--------------------------------------------------------------
+121522136xx 2 2 6:13 3 9600 SupraFAX v.32bis
+151974615xx 2 2 7:27 3 9600 Telebit WorldBlazer
+161945506xx 1 1 9:07 3 7200 Ricoh 9200
+191858837xx 1 1 8:16 3 7200 Ricoh FAX 3000L
+49428616xx 2 2 8:02 3 9600 GUIS ETFax 7
+4991318587xx 1 1 8:08 3 9600 Multi-Tech 1432
Table 7. Transmission anomalies for DataPort 14.4/FAX.
4.3.2 Digicom Systems Scout+
Table 8 shows the anomalous jobs for the Digicom Scout+.
The Scout+ had a decided problem establishing carrier with both
modems and machines.
Frequently the modem would return a
+FCERROR
status when
calling, indicating that an incorrect carrier was encountered.
For example,
Oct 04 03:45:05.62: [ 1504]: DIAL 130597073xx
Oct 04 03:45:05.62: [ 1504]: <-- [21:ATS24=2DT130597073xx@]
Oct 04 03:45:40.53: [ 1504]: --> [8:+FCERROR]
Oct 04 03:45:55.55: [ 1504]: --> [10:NO CARRIER]
Oct 04 03:45:55.55: [ 1504]: SEND FAILED: No carrier detected
This is the reason for many of the unsuccessful calls to the:
Fujitsu DEX 170 at +140874538xx,
Pitney-Bowes 9300 at +141696989xx,
and
XEROX Telecopier 7010 at +151292941xx.
Destination Tries Calls ConnTime Pages TypRate Device
--------------------------------------------------------------
+140874538xx 4 1 8:02 3 9600 Fujitsu DEX 170
+141696989xx 4 3 9:08 0 2400 Pitney-Bowes 9300
+150866368xx 2 2 6:14 3 14400 SupraFAX v.32bis
+151292941xx 2 1 6:12 3 9600 XEROX Telecopier 7010
+170836718xx 2 2 4:39 3 14400 ZyXEL 1496
+170889876xx 5 2 6:46 3 14400 USR Sportster
+181830557xx 3 2 13:16 3 4800 Intel SatisFaxion 100
+41142227xx 3 3 5:29 0 14400 Multi-Tech 1432
+446127561xx 3 3 6:49 0 2400 Zoom 14.4X
+4956180442xx 2 2 12:42 3 4800 Olivetti OFX 430
+4991318587xx 2 2 7:25 3 9600 Multi-Tech 1432
Table 8. Transmission anomalies for Scout+ with V.17 support.
The calls to the Pitney-Bowes 9300 at +141696989xx and the
Multi-Tech 1432 at +41142227xx failed because no response was
received to the PPM sent after the first page.
After three successive attempts to send a page FlexFAX will abort
the job and return the untransmitted contents to the sender.
The call to the Zoom 14.4X at +446127561xx failed because the
two modems were unable to succeed in training.
The first call to the Supra modem at +150866368xx,
the ZyXEL modem at 170836718xx, and the Sportster at +170889876xx,
failed because of training; with the subsequent call going perfectly.
Other calls failed either because of failure to train the modems
or because of failure to receive a response to the post-page message
for the first page.
The call to the Olivetti OFX 430 at +4956180442xx took a long time
because of retraining and having to make an extra call to retransmit
the last page.
In general it appeared that the Scout+ performed poorly in the face
of line noise and/or low signal quality.
Discussions with informed parties confirmed this observation: firmware
revision 3507/3506 has problems with line noise and poor signal quality.
For this reason the tests were rerun with a previous version
of the firmware that did not include V.17 support but which was
thought to perform well otherwise.
The results for firmware revision 2A19/2931 are shown in Table 8a.
Problems with misrecognized carrier (+FCERROR) were encountered for
many of the same receivers.
Otherwise the older firmware did a better job of communicating with
facsimile machines (as opposed to modems);
but performed worse than the newer firmware that includes support for V.17.
The Everex modem is the oldest modem that was tested; in fact it is no
longer sold.
The older hardware design was seen in the lower transmission performance.
Table 9 shows the anomalous jobs for the EverFax 24/96D.
Destination Tries Calls ConnTime Pages TypRate Device
--------------------------------------------------------------
+121522136xx 4 3 8:49 3 9600 SupraFAX v.32bis
+141696989xx 2 2 9:57 3 9600 Pitney-Bowes 9300
+170836718xx 3 1 5:51 3 9600 ZyXEL 1496
+181830557xx 2 2 14:42 3 4800 Intel SatisFaxion 100
+191858837xx 5 0 0:00 0 0 Ricoh FAX 3000L
+49428616xx 2 2 11:17 3 9600 GUIS ETFax 7
+4956180442xx 5 0 0:00 0 0 Olivetti OFX 430
Table 9. Transmission anomalies for EverFax 24/96D.
Typically when the modem was able to establish a connection
communication was very reliable.
Calls to the GUIS ETFax 7 at +49428616xx,
the Pitney-Bowes 9300 at +141696989xx, and
the Intel SatisFaxion 100 at +181830557xx
required an additional call because the receiver
failed to respond to a post-page message.
4.3.4 Multi Tech Systems MT1432BA
Table 10 shows the anomalous jobs for the MT1432BA.
Calls to the Telebit modem at +151974615xx
failed because no response was received to the post-page message;
subsequent calls were not answered and the jobs were prematurely
removed from the queue.
The call to the Ricoh 9200 at +161945506xx required excessive time
because the third page had to be retransmitted.
The GUIS ETFax at +49428616xx required a second call to retransmit
the third page; the first try failed when no acknowledgement was received
to the post-page message.
The Multi-Tech modem at +4991318587xx took extra time because the first
page had to be retransmitted.
Destination Tries Calls ConnTime Pages TypRate Device
--------------------------------------------------------------
+151974615xx 2 1 3:22 0 9600 Telebit WorldBlazer
+161945506xx 1 1 7:05 3 9600 Ricoh 9200
+49428616xx 2 2 8:11 3 9600 GUIS ETFax 7
+4991318587xx 1 1 6:32 3 9600 Multi-Tech 1432
Table 10. Transmission anomalies for MT1432BA.
4.3.5 Supra SupraFAXModem V.32bis
Table 11 shows the anomalous jobs for the Supra modem.
There are some interesting results here.
The OmniFax G66 at +141623971xx required extra time because the
modem retrained down to 7200 baud from the requested 9600 baud.
The XEROX Telecopier 7010 at +151292941xx took extra time because
the third page had to be retransmitted.
The Canon 260E at +495115613xx also required extra time because
of retraining; though this time no page retransmission was requested.
The remaining problems were reported as
+FHNG: 51 or +FHNG: 52;
the latter indicates ``No response to
MPS
repeated 3 times'' (i.e. no response to the
MPS
post-page message), while the former means
``RSPREC error/got DCN'',
or disconnect was received when waiting
for the post-page response.
These sort of failures are indicative of a modem that does not
do a good job handling line noise and poor signal quality.
The Supra modem is based on the Rockwell
RC144DP,
a part that is considered to be susceptible to poor line conditions.
Destination Tries Calls ConnTime Pages TypRate Device
--------------------------------------------------------------
+141596576xx 2 2 8:17 3 9600 NEC BitFax
+141623971xx 1 1 7:07 3 7200 OmniFax G66
+141636206xx 2 2 6:41 3 14400 Xerox Telecopier 7033
+150866368xx 2 2 4:48 3 14400 SupraFAX v.32bis
+151292941xx 1 1 7:26 3 9600 XEROX Telecopier 7010
+160278928xx 2 2 8:41 3 9600 Lanier FaxWriter 4000
+160496057xx 2 2 6:11 3 14400 Canon FAX-L785
+170836718xx 3 3 5:28 0 14400 ZyXEL 1496
+170889876xx 5 3 2:39 0 14400 USR Sportster
+181830557xx 2 2 14:31 3 4800 Intel SatisFaxion 100
+191858837xx 2 2 7:59 3 9600 Ricoh FAX 3000L
+41142227xx 3 3 6:37 0 14400 Multi-Tech 1432
+446127561xx 2 2 6:26 3 14400 Zoom 14.4X
+495115613xx 1 1 7:05 3 9600 Canon 260E
+495117178xx 4 3 7:46 0 14400 ZyXEL 1496
Table 11. Transmission anomalies for SupraFAXModem V.32bis.
4.3.6 Telebit Systems T3000
Table 12 shows the anomalous jobs for the Telebit T3000.
Several jobs were prematurely terminated which accounts for their
having only three or four tries instead of the expected five.
Calls to two different ZyXEL modems failed the first time to
establish carrier but
were successful on the second try--this may be related to the
ZyXEL's adaptive-answer scheme (it is not known if all the ZyXEL
modems involved were configured to use adaptive-answer).
The Pitney-Bowes 9300 at +141696989xx failed because of a failure
to receive a response to the
MPS
post-page message.
Destination Tries Calls ConnTime Pages TypRate Device
--------------------------------------------------------------
+121522136xx 4 0 3:36 0 9600 SupraFAX v.32bis
+141696989xx 3 3 8:28 0 9600 Pitney-Bowes 9300
+171864874xx 4 3 2:28 0 9600 Brother InstaFax 390
+41142227xx 3 3 2:48 0 9600 Multi-Tech 1432
+446127560xx 4 0 2:36 0 9600 Canon FAX-L770
+4991318587xx 3 3 2:43 0 9600 Multi-Tech 1432
Table 12. Transmission anomalies for Telebit T3000.
The Pitney-Bowes 9300 at +141696989xx,
the Multi-Tech 1432 at +41142227xx, and
the Canon FAX-L770 at +446127560xx
failed with what looks to be an oversight in the modem firmware.
The modem sent a ``NO CARRIER'' result string without first
sending a Class 2 ``+FHNG:'' result:
Oct 03 21:16:33.34: [19850]: <-- [6:AT+FDT]
Oct 03 21:16:34.50: [19850]: --> [74:+FHT:FF C0 C2 04 04 04 04 ...
Oct 03 21:16:34.50: [19850]: --> [26:+FHT:FF C8 C1 00 62 1F 00 ]
Oct 03 21:16:39.93: [19850]: --> [74:+FHR:FF C0 02 0C CC EC 4C ...
Oct 03 21:16:40.29: [19850]: --> [26:+FHR:FF C8 01 00 76 5F 00 ]
Oct 03 21:16:41.31: [19850]: --> [74:+FHT:FF C0 C2 04 04 04 04 ...
Oct 03 21:16:41.31: [19850]: --> [26:+FHT:FF C8 C1 00 62 1F 00 ]
Oct 03 21:16:45.47: [19850]: --> [10:NO CARRIER]
The Class 2 specification is not clear on whether or not this
is permissible; about all that can be said is that the Telebit
modem is unique in its operation in this respect.
The calls to the Brother InstaFax 390 at +171864874xx failed with
another problem that is definitely a firmware bug:
Oct 03 23:32:40.14: [25845]: DIAL 171864874xx
Oct 03 23:32:40.15: [25845]: <-- [15:ATDT171864874xx]
Oct 03 23:33:11.98: [25845]: --> [74:+FHR:FF C0 02 9C 2C 2C EC ...
Oct 03 23:33:12.27: [25845]: --> [26:+FHR:FF C8 01 00 72 1D 00 ]
Oct 03 23:33:12.27: [25845]: --> [28:+FCSI:" 718 648 74XX"]
The modem did not send the host the expected
+FCON
message that indicates
a facsimile connection was established.
4.3.7 Twincom 14.4/DF
The Twincom modem was running old firmware.
The vendor was contacted several weeks
before these tests were run, but newer firmware
was not available for testing.
Table 13 shows the anomalous jobs for the Twincom modem.
Like the Supra modem the Twincom is based on the Rockwell
RC144DP;
a component that is considered to be susceptible to poor line conditions.
Destination Tries Calls ConnTime Pages TypRate Device
--------------------------------------------------------------
+121522136xx 5 5 11:21 3 14400 SupraFAX v.32bis
+141596576xx 2 2 8:55 3 9600 NEC BitFax
+141623971xx 1 1 9:53 3 9600 OmniFax G66
+141696989xx 3 3 8:24 0 9600 Pitney-Bowes 9300
+150866368xx 3 3 4:09 0 14400 SupraFAX v.32bis
+151292941xx 1 1 14:10 3 9600 XEROX Telecopier 7010
+151974615xx 3 3 10:27 0 9600 Telebit WorldBlazer
+151974637xx 2 2 7:44 3 9600 Lanier Model 120
+161723336xx 3 3 10:22 0 9600 Telebit WorldBlazer
+170825003xx 2 2 4:37 3 14400 PPI PM14400FX
+170836718xx 3 3 5:34 3 14400 ZyXEL 1496
+170889876xx 3 1 3:37 3 14400 USR Sportster
+181830557xx 1 1 18:11 3 7200 Intel SatisFaxion 100
+41142227xx 3 3 12:12 0 14400 Multi-Tech 1432
+446127560xx 5 1 5:29 2 9600 Canon FAX-L770
+4681025xx 2 2 7:48 3 9600 OKIFAX OF-110N
+49428616xx 3 3 10:31 3 9600 GUIS ETFax 7
+495117178xx 2 2 4:57 3 14400 ZyXEL 1496
+4991318587xx 2 2 11:23 3 9600 Multi-Tech 1432
Table 13. Transmission anomalies for Twincom 14.4/DF.
A significant test was to the Multi-Tech 1432 at +41142227xx.
Two different calls to this receiver left the modem in a state where
it held carrier but did not respond to
DTR.
Manual intervention was required to hangup the line.
This is a serious problem that alone makes the modem unacceptable for
unmonitored operation.
The Pitney-Bowes 9300 at +141696989xx and
the Supra at +150866368xx failed because there was no response to the
post-page message.
The Telebit modems at +151974615xx and +161723336xx, and the Multi-Tech
1432 at +41142227xx failed because the modem locked up during Phase B/C;
for example:
Oct 03 18:58:09.40: [17733]: <-- [23:AT+FDIS=1,3,0,2,0,0,0,0]
Oct 03 18:58:09.68: [17733]: --> [2:OK]
Oct 03 18:58:09.68: [17733]: <-- [6:AT+FDT]
Oct 03 19:01:09.68: [17733]: TIMEOUT: reading line from modem
FlexFAX times out the
AT+FDT
command after three minutes to insure the
sender does not leave the phone off hook due to a modem problem.
This timeout is configurable and scaled according to the signaling
rate used for the high speed message carrier.
In practice three minutes is significantly
longer than could possibly be needed
to carry out the Phase B handshaking.
4.3.8 Zero One Networking ZyXEL 1496E
Table 14 shows the anomalous jobs for the ZyXEL 1496E.
The ZyXEL firmware has a bad problem of not reporting useful hangup codes
when problems occur.
Virtually any problem is reported to the host as ``+FHNG: 50'' which
means ``Unspecified Transmit Phase D error, including
+FPHCTO
timeout between data and
+FET
command.''
Consequently it is difficult to interpret many of the results.
Destination Tries Calls ConnTime Pages TypRate Device
--------------------------------------------------------------
+121282556xx 2 2 8:00 3 9600 Pan. Panafax PD-4200
+141596576xx 3 3 10:29 3 9600 NEC BitFax
+141623971xx 3 3 10:24 2 9600 OmniFax G66
+150866368xx 2 2 5:12 3 14400 SupraFAX v.32bis
+161945506xx 3 3 8:14 1 9600 Ricoh 9200
+161954639xx 4 4 12:20 2 9600 HP FAX-200
+41142227xx 3 3 6:56 0 14400 Multi-Tech 1432
+446127561xx 5 0 0:00 0 0 Zoom 14.4X
Table 14. Transmission anomalies for ZyXEL 1496E.
The Panasonic Panafax PD-4200 at +121282556xx required an extra call because
the response to the
EOP
was apparently not received.
Unfortunately the modem does not indicate this directly
but instead reports:
Oct 02 10:18:49.33: [ 579]: <-- [8:AT+FET=2]
Oct 02 10:18:55.38: [ 579]: --> [10:NO CARRIER]
without a hangup code.
This is a firmware bug since the modem
should present a
+FHNG:
status message.
Three tries were required to transmit the third page to the NEC BitFax
at +141596576xx.
The first try failed with the above problem, while the second try returned:
Oct 02 12:43:42.00: [ 579]: <-- [8:AT+FET=2]
Oct 02 12:43:58.24: [ 579]: --> [8:+FHNG:50]
The OmniFax G66 at +141623971xx failed with problems similar to the above
two jobs.
The first two pages went through without a problem, but the third page
was unsuccessful with either no hangup code or with hangup code 50.
The Supra modem at +150866368xx required two calls because the first failed
with a ``Unspecified Transmit Phase B error'' (hangup code 20).
Once again a better hangup code would help to understand what actually
took place.
The Ricoh 9200 at +161945506xx failed after three attempts to send
the second page.
As above, a generic error code (50) was returned to the host.
The results are indicative of a modem that has problems with line noise.
This is surprising since the modem is reputed to do well as data modem.
4.3.9 US Robotics Sportster
Table 15 shows the anomalous jobs for the Sportster modem.
A significant test was to the Zoom modem at +446127561xx.
On one occasion the modem entered a state where
it held carrier but did not respond to
DTR.
Manual intervention was required to hangup the line (after over an
hour of connect time!)
This is a serious problem that alone makes the modem unacceptable for
unmonitored operation.
Destination Tries Calls ConnTime Pages TypRate Device
--------------------------------------------------------------
+121522136xx 2 1 4:54 3 14400 SupraFAX v.32bis
+141596576xx 3 3 7:49 0 9600 NEC BitFax
+150866368xx 3 3 7:37 0 0 SupraFAX v.32bis
+151974615xx 4 1 5:59 0 9600 Telebit WorldBlazer
+181830557xx 4 3 17:52 3 4800 Intel SatisFaxion 100
+41142227xx 3 3 6:12 0 14400 Multi-Tech 1432
+446127560xx 3 3 8:22 0 9600 Canon FAX-L770
+446127561xx 3 3 7:41 0 14400 Zoom 14.4X
+4991318587xx 1 1 6:06 3 9600 Multi-Tech 1432
Table 15. Transmission anomalies for USR Sportster.
The Supra modem at +121522136xx took a little extra time because of
some handshaking messages had to be retransmitted.
The NEC BitFax at +141596576xx failed when three consecutive attempts
to send the first page were unsuccessful because no response was
received to the post-page message.
Communication with the Supra modem at +150866368xx failed because
the modems were unable to establish training--specifically the sender
never received a
CFR/FTT
response to training.
The Telebit modem at +151974615xx failed when no response was received
to the first post-page message; subsequent calls failed when no
carrier was established.
The Intel modem at +181830557xx required three tries to successfully
transfer the first page, after which the remainder of the document
went through without a problem.
The Multi-Tech modem at +4991318587xx required extra time because the
second page had to be retransmitted.
The data presented in Section 4 and Appendix A
is too difficult for most people
to understand and use in comparing facsimile modems.
For better or worse, people want a single number that they can use
to guide them in comparing the modems.
Creating a single number is however problematic.
The results of Section 4 indicate there are at least two distinct
characteristics
that are important in rating a modem: reliability and performance.
Reliability refers to whether or not a modem accomplishes a requested
task--in this case delivering the test facsimile document to a
receiving device.
Performance is a measure of how well the modem carries out the set of
tasks it has been assigned.
Some modems support
V.27,
V.29, and
V.17,
but do not handle poor line conditions well or have difficulties
establishing carrier under certain conditions.
These modems may perform well in limited use but not be as reliable as
other modems.
Other modems may excel under poor line conditions but not support the
higher signaling rate of
V.17.
These modems should score high for reliability but may not rate as the
highest performing modems available.
Our measure for comparing modems rates reliability and performance separately,
combining them afterwards for the purpose of defining a single
quality rating.
The
reliability
of a modem, Rm, is given by Equation 5.1:
Rm = # pages delivered / total # pages (5.1)
Remember that a page is not considered delivered unless an acknowledgement
is received from the receiving device.
Note also that the reliability of a Class 1 modem is heavily
dependent on the host software that implements the Group 3 facsimile
protocol.
This is much less true for Class 2 modems.
In our tests the same software was used for all tests.
We assume that it functions correctly and efficiently.
The results tend to support this premise.
When all pages are delivered, Rm reduces to:
Rm = 1
The
performance
of a modem, Pm, is given Equation 5.2:
.EQ I (5.2)
Pm = Sum {r:Receivers} Sum {p:Phase A-Phase D} T(r,p) (5.2)
where T(r,p) is the average time spent in phase p of the
Group 3 protocol communicating with receiver r.
If a modem does not successfully complete a phase then the average time
for a modem of its capabilities is substituted.
That is, if a modem that supports only
V.27
and
V.29
is unable to successfully transmit page 3 to a particular receiver,
then the average time for similar modems
for each unexecuted phase of the transfer is used
when calculating Pm.
The above equations measure how well a modem does
under error-free conditions.
When a modem does not transmit a facsimile
on the first call it should be penalized.
Also, when a modem
sends multiple copies of a page it should be penalized.
Thus we define an
error metric
for a modem as Em:
Em = Wep (# extra pages transmitted / maximum # extra pages) x (5.3)
Wec (# extra calls / (.67 x maximum # extra calls))
where Wep and Wec are weights that reflect the
relative importance of each term in Em.
The .67 term in Equation 5.3 is present
to balance the two terms because the maximum number of extra pages is
2 x 3 x 33 = 198 and the maximum number of
extra calls is 4 x 33 = 132.
Given the above definitions, we define
a quality rating Qm by:
Qm = Wr x R m + Wp x (Tbase / Pm) + Em (5.4)
.EN
where Wr and Wp are weights that are used to reflect
the relative importance of the Rm and Pm terms and
Tbase is a
baseline time
that is combined with Wp to normalize the performance numbers
to a range that is more consistent with the other terms.
For a modem that transmits all the test data without extraneous
phone calls or page retransmissions the metric reduces to:
Qm = Wr x 1 + Wp x (Tbase / Pm) - 0
= Wr + Wp x (Tbase / Pm)
So long as a modem is able to transmit all the test pages its rating is
controlled by its performance metric Pm and any error costs.
A failed call incurs cost through an extra phone call and possibly
an extra page transmission.
This cost appears in the error term Em.
Modems are usually double-penalized for pages that are never successfully
transmitted because the failed page appears both in Rm and indirectly
in Em as extra calls and (probably) extraneous transmitted pages.
Table 16 shows the parameters for Qm derived from the test data.
The performance metric Pm was calculated as described above.
Because there were relatively few modems that supported
V.29
but not
V.17,
the results for the Everex 24/96D were not used in calculating ``fill in''
entries for failed phases (since the Everex times were significantly
skewed from the norm).
To complete the calculations for the Qm we need
to select a baseline time and values for the different weights.
Choosing weights requires selecting ranges for the different factors
that reflect desired characteristics of a ``good modem''.
Our criteria weights reliability significantly higher than performance,
with extra calls more important than extraneous pages.
In addition we want quality metrics for a good modem to be around 100.
A possible set of weights that meets our criteria are:
Wr of 80, Tbase of 2:23:11,
Wp of 16, Wep of 8, and Wec of 20.
[The value of Tbase is based on an analysis of the
T.4-encoded
data, the average phase times, and the receiver capabilities.
It represents the best expected time that a sender can do.]
Using these weights we define a FaxStone to be:
This set of parameters has many good intuitive properties.
The reliability term is in the range [0,80] while the performance term
is in the approximate range [0,16].
This weights reliability over performance by a factor of five-as desired.
The error metric is in the range [-28,0] with the majority of the
error metrics in the range [-5,0].
The overall range for the quality rating
is large enough to distinguish between modems
but not so wide as to make comparisons difficult.
Differences in modem reliability are distinguishable and
it is possible to identify raw performance by looking solely at the
Pm term.
Clearly other weights may be selected in order to emphasize other
performance characteristics of the modems.
Table 17 shows the FaxStones calculated from our test data.
The columns marked R, P, and E are the weighted reliability, performance,
and error terms, respectively.
Two modems were clearly the best performers:
the AT&T DataPort 14.4/FAX and the Multi-Tech MT1432BA.
The modems were extremely reliable.
The DataPort lagged noticeably behind the Multi-Tech modem
in performance because it did not support
V.17,
however this was not especially
significant in the results because the majority
of the receiving devices also did not support
V.17.
The Everex 24/96D proved to be very reliable also, though it
was unable to connect to several of the more difficult receivers.
Its performance lagged significantly behind all the other modems.
This is due to it being based on old hardware technology.
The remaining modems all rated fairly close together:
Digicom Scout+, ZyXEL 1496E, USR Sportster, SupraFAXModem v.32bis, Telebit T3000, and Twincom 14.4/DF.
The Scout+ had a significant problem establishing carrier
with a variety of receivers.
The ZyXEL 1496E was particularly disappointing because of the relatively
low cost of the modem and its excellent reputation as a data modem.
The test results however indicate that the firmware has problems
and the modem may be susceptible to line noise during facsimile operation.
The USR Sportster modem had the highest performance rating of any modem
but rated very unreliable in testing.
Also, as reported earlier, the Sportster had a
serious firmware problem that makes unattended use uncertain.
The Telebit modem had at least one firmware problem that when corrected
will significantly raise the modem's FaxStone rating.
This modem also appears to be one of the quickest modems and also
appears to handle poor line conditions well.
The Twincom 14.4/DF rated last in the tests;
it also exhibited serious firmware problems.
Table 17 includes price estimates in US dollars for the modems that
were tested.
Given these prices, the best buy is the AT&T DataPort 14.4/FAX.
The one complaint that can be made about the modem is that it does not
support
V.17.
Since Telebit offers fax upgrades to existing T3000 and WorldBlazer modems
for less than US$200 this may also be an excellent route to take for
people that already own a Telebit modem, once
the Telebit firmware problems are resolved.
The Multi-Tech modem looks to be an excellent modem for people that
require a Class 2 modem.
The only complaint one can make about the MT1432BA
is that it does not support 2D-encoded data (though this should be
trivial to add to the firmware).
This experiment used a very pragmatic approach to evaluating each
modem's performance.
Another valid approach is to test each modem using equipment that
can simulate varying line conditions and protocol errors.
This would be useful in understanding exactly how well the modems
do under poor line conditions rather than guessing at such behaviour
based on the way the modem operates.
Another worthwhile area of work is to get a better understanding of
the value of the optional
T.30
Error Correction Mode (ECM).
ECM
could only be used to communicate with the facsimile machines in
this experiment, but it is potentially a worthwhile mechanism for
improving the reliability of facsimile communication when poor
line conditions are encountered.
Finally, the set of receivers used in this experiment were all
located in highly industrialized areas of the world.
It is important to recognize that other parts of the world may
not have such high quality phone systems.
In such areas devices such as echo-suppressors may have
more of an effect on communication.
The test results reported here are likely
not to reflect how well the modems work under such conditions.
John Banning provided a lot of useful feedback and corrected my
wayward thinking when constructing the various metrics.
Ed McCreight helped understand many of the issues involved in
comparing the modems.
The following people volunteered their time and equipment; their help
is greatly appreciated.
Adrian Collins Bill Morrow Brian Katzung C. Harald Koch
Carsten Koch Charlie Slater Dan Busarow David Vrona
Dirk Husemann Heiner Meuser Ian Darwin James Stansell
Jim Patterson Jochen Ruhland John R. Koehring John Schwegler
John Sellens John T. Kohl Joseph E. Sacco L. Todd Masco
Larry Williamson Lyndon Nerenberg Martin Lichtin Paul Lupa
Peter Shipley Rickard Schoultz Robert Weiner Vance Shipley
Certain vendors were kind enough to provide me with modems for use
in testing; their assistance was critical:
Digicom Systems (Wolfgang Henke), Multi-Tech (Pete Hanlon),
Twincom (John O'Connell). and
US Robotics (Pete Frankiewicz).
Thanks also to Nick Doshi at Zero One Networking and Eric Ziegast
of UUNET for providing firmware updates for existing modems.
FlexFAX version 2.2.1 is available by public ftp from the host
sgi.com in the directory ~ftp/sgi/fax.
The PostScript\(tm source files for the test document are
available also by public ftp from sgi.com in the directory
~ftp/sgi/fax/bakeoff.
A PostScript version of Appendix A is also available in this directory.
The session logs are available to modem vendors by request.