Re: flexfax: Large number of errors with Hylafax and ZyXEL Omni modem

[Steve Williams <steve@genie96.com>]

Hi Shuvam,

I have seen quite a few postings from you.  While I must welcome 
the efforts to better the software, YOU are the only one in a position
to really determine your environment, and what the problem is.

I have been involved with HylaFax for several years.  I have been on the 
beta test list for most of that time, and put together the binary 
distribution for AIX.

I run the TPC.INT Cell for Calgary, Canada using a 1990 vintage IBM RS6000
computer, with 7 MultiTech ZDX Modems.  I have also used ( in the past ) 
Practical Periphal Modems, with a fairly high sucess rate.

Though I have briefly visited India, I am not familiar with the telephone 
infrastructure there, or even what area you are providing coverate for, 
so forgive me if I make any assumptions that are not valid in your
environment.

I initially set up the fax software on our system at work to benefit 
us at work ( funny that <g> ).  We have 7 phone lines that I wanted to 
be able to dial in on, as well as use for outgoing data, and outgoing
faxes.  I do not use any of them for incoming faxes except for 
massive testing of the system ( sending from 3 modems to receiving from
three modems ... ).

When I saw how reliable it worked for our purposes at work, even using
our old ancient RS6000 320's ( 1990 vintage ), I got interested in 
offering the service to TPC.INT.

When I say reliable for our purposes at work, I REGULARILY send out 250 
faxes to our clients and only have 1 or two failures due to flakey 
serial ports, in which a resend invariably works.  So, I can send 
250 faxes out ( typically ) three modems and have a marginal failure
rate.  Please understand, some of these receiving fax machines are old 
manual faxes that need the "Receive" button pressed, others are relayed 
through the phone systems, where sometimes someone will answer "line 5", 
when line 5 is actually a fax, so they hang up real fast, all those 
types of situations.

The classic one is a client that has to be at work to receive the fax
because the fax line is line 9 on the phone system, and they have to
manually forward the phone call to the fax machine at a certain extension
in order to receive the fax.  Sending faxes at night, with at 30 % discount
doesn't work with them :-(

When I joined TPC.INT, I was dismayed at the failure rate of the faxes.

I started worrying that the modems were no good, that the software was 
no good, UNTIL I started monitoring the failed faxes.

For a period of several days, I dialed manually on a handset the phone
number of all faxes that failed.  While I did not keep records, I seem to
recall that out of about 20 failed faxes that I checked using a 
telephone handset, only 5 or so  was actually a FAX MACHINE!

So, 15/20 failures that I experienced was due to a WRONG PHONE NUMBER
being entered.

The other ones..., who knows, full moon?

I will include an awk script that you can use to parse the xferlog 
of HylaFax.  This will give a standard output that some of us on the
mailing list can compare to.

I am sorry that I can't give credit to the person who created this script,
but I am pretty sure I got it from the mailing list, so the records
should be there somewhere.

Here is a list of the errors that I have gotten recently.

I have "tweaked" the log files to show the stats as if they are for one
modem to shorten this email.

I have included two portions.

One is generated by our internal staff, hopefully to numbers that they
pretty much know are valid..., though typo errors do happen.

The second one is TPC.INT traffic.  I have broken this out so that a 
comparison of the errors can be made.

-------------------------------------------------------------
Genie Computer Systems (1996) Inc. Internally generated faxes
-------------------------------------------------------------
Modem: tty5
Total faxes sent:                                                       414

Error                                                           No  Rate/1000
------------------------------------------------------------------------------
Unspecified Transmit Phase B error; too many attempts to sen     5     12.1
Unspecified Transmit Phase B error                              11     26.6
DIS/DTC received 3 times; DCS not recognized; Giving up afte     2      4.8
Can not open document file                                       2      4.8
                                                                 1      2.4
RSPREC error/got DCN                                             1      2.4
No answer (T.30 T1 timeout); too many attempts to send           1      2.4
Unknown problem (check modem power)                              3      7.2
Job aborted by user                                              5     12.1
No response to EOP repeated 3 times                              2      4.8
DIS/DTC received 3 times; DCS not recognized                     3      7.2
No carrier detected; too many attempts to dial                   3      7.2
No answer (T.30 T1 timeout)                                     10     24.2
Unable to transmit page (giving up after 3 attempts)             1      2.4
No carrier detected                                             64    154.6
------------------------------------------------------------------------------
Total                                                          114    275.4


-------------------------
TPC.INT generated Traffic
-------------------------
Modem: tty5
Total faxes sent:                                                       394

Error                                                           No  Rate/1000
------------------------------------------------------------------------------
Unspecified Transmit Phase B error; too many attempts to sen     7     17.8
Unspecified Transmit Phase B error                              16     40.6
Unspecified Transmit Phase B error; too many attempts to dia     1      2.5
No answer (T.30 T1 timeout); too many attempts to send           1      2.5
Unknown problem (check modem power)                              7     17.8
No response to MPS repeated 3 times                              1      2.5
No response to EOP repeated 3 times                              1      2.5
No carrier detected; too many attempts to dial                   4     10.2
No answer (T.30 T1 timeout)                                     41    104.1
No answer (T.30 T1 timeout); too many attempts to dial           4     10.2
No carrier detected                                            113    286.8
------------------------------------------------------------------------------
Total                                                          196    497.5



As you can see, the "No Answer"/"No carrier" messages are radically 
different between the two situations.  180/1000 vs 400/1000.

What I would be interested in seeing is YOUR output from the below 
awk script to see where exactly the errors you are having.

Please remember that these error statistics are being incremented ON THE
FLY.  If a fax fails twice, but works on the third time, TWO ERRORS ARE
STILL RECORDED.

I enclose an example from ONE OF OUR CLIENTS that demonstrates this...
Typically, this is where they have to plug the phone line from the
computer modem to the fax machine, then not answer line 3.  Maybe it
works on the third time...

( an exact excerpt from the log file )

Attempt One...
--------------
03/27/98 10:46	SEND	00000585	tty5	433	"OHIPFEE"	\
	faxuser@geniers.genie	"XXXXXXXXXXX"	""	65535	0	\
	1:16	0:00	"No carrier detected"

Attempt Two...
--------------
03/27/98 10:51	SEND	00000594	tty5	433	"OHIPFEE"	\
	faxuser@geniers.genie	"XXXXXXXXXXX"	""	65535	0	\
	1:32	0:03	"No answer (T.30 T1 timeout)"

Attempt Three...
----------------
03/27/98 10:57	SEND	00000603	tty5	433	"OHIPFEE"	\
	faxuser@geniers.genie	"XXXXXXXXXXX"	"XXXXXXXXXX"	33030	\
	2	1:56	1:11	""

YEAH!!!, it worked.

So, I am not sure how this is recorded in the AWK file.  Maybe someone
that is awake can take a boo at the script and say if these errors are
included.

Regardless, it would be worth seeing the output from the awk script
on your data files.


modem.errors 
-------------
SNIP HERE
---------
#!/bin/sh
#
# errorstats - Script to parse error data out of xferlog
#
if [ $# -gt 0 ]; then
	XFERLOG=$1
else
	XFERLOG=/var/spool/fax/etc/xferlog
fi
gawk -F "\t" '
	function doline(a, b, c) { printf "%-60.60s %5i %8.1f\n", a, b, c }
	{
		if ( $4 != "" ) total[$4] += 1
		if ( $11 == "0") {
			errname[$14] = 0
			errcount[$4,$14] += 1
		}
	}
	END {
		for (m in total) {
			printf "Modem: %s\n",m
			printf "%-68.68s %6i\n\n","Total faxes sent:",total[m]
			printf "%-60.60s %5s %10s\n","Error", "No","Rate/1000"
			printf "---------------------------------------"
			printf "---------------------------------------\n"
			tot = 0
			for (n in errname) {
				if ( errcount[m, n] > 0 ) {
					name = n
					gsub("\"","",name)
					rate = errcount[m, n] * 1000 / total[m]
					tot += errcount[m, n]
					doline(name, errcount[m, n], rate)
				}
			}
			printf "---------------------------------------"
			printf "---------------------------------------\n"
			doline("Total", tot, tot * 1000 / total[m])
			printf "\n"
		}
	}' $XFERLOG 
---------
END

So, let's see WHY you are having the problems.  Maybe consider dialing
some of the failed fax phone numbers by hand and seing if they are 
actually a fax machine or not.  After 10 or so random calls, you should
have a feeling if they are fax machines or humans answering the phones.
If it's the latter, not much HylaFax can do about that <g>..

Let's see your stats....

Take care,

> 
> This is in continuation to earlier mails from myself and others on this
> topic. I'm running a TPC cell using Linux, Hylafax 4.0pl2, and a ZyXEL
> Omni O288S, as mentioned before. This is the statistical count of the
> Hylafax transfer figures for the last day:
> 
>      52	0
>      39	2
>       6	3
>      13	4
>       1	5
>       1	6
>       2	7
>       1	9
>       1	10
>       1	11
>       1	12
>       1	13
>       1	14
> 
> The first field is count or frequency, and the second is the number of
> people who sent a total number of pages as shown. So "52 0" means that
> 52 people queued one or more jobs and could send only 0 pages. The
> second line is "39 2" which means that 39 people sent a total of 2 pages
> each. (This is the minimum length of most TPC jobs, because they have
> a cover page, and a short message page after that. This means that 39
> people sent one job each.) And so on.
> 
> The total is 121 people, who have tried sending faxes out. Out of them
> 52 people couldn't send out anything at all, as we can see. This is
> unacceptable by any standards, I would think. Harald Pollack has said that
> any public fax broadcaster will encounter about 40% failures. Another
> gentleman has been using Multitech modems and has reported 5 to 10%
> failures only. I am reporting about 45% faxes failed, which means perhaps
> 60 to 80% of alls calls made, failing. (This is because each fax fails
> only after it has made several attempts to go through.)
> 
> Another figure shows that 242 pages were transmitted successfully,
> but 313 error-calls were recorded. These errors exclude the ones where
> carrier couldn't be established. This means that each of these 313 calls
> were completed at the Telco level, and so will be billed to me. Is this
> typical? I am sure all TPC operators who use Hylafax will be getting
> their daily report from "faxcron". Can they send me a five line email
> just cutting out the day's "Total" line from yesterday's report, together
> with a line giving which area they operate in?
> 
> Most of us run TPC as a voluntary service. We need to cut wasteful
> expenditure.  We use Hylafax, and we need the Hylafax community to
> come together to figure out whether there is indeed any problem in
> the software which might be increasing errors, as Harald's and others'
> mails have hinted. I have not seen much discussion on the Hylafax
> mailing lists about these fundamental aspects of its behaviour; most of
> the discussion revolves around new setup problems, cover pages, and the
> like, which are important but are not of use in this specific case.
> 
> Is Sam Leffler still active on this list or with this software? Is Matt
> Apitz or David Woolley or anyone else in a position to take up these
> issues and bring down the errors, specially with ZyXEL modems? One set
> of ZyXEL users who don't use Hylafax, swear by its fax implementation.
> Should all Hylafax users then switch out of ZyXEL to Multitech? Or
> should all TPC operators switch out of Hylafax?
> 
> Hoping for more responses on this list,
> 
> regards,
> Shuvam
> 


-- 
	Steve Williams, Calgary, Alberta, Canada
	Genie Computer Systems Inc.
	steve@genie96.com

"A man doesn't begin to attain wisdom until he recognizes that he is 
 no longer indespensable."
- Admiral Richard E. Byrd ( 1888-1957 )