AW: AW: flexfax: Problems submitting very large Number of jobs sequen tial ly

[Christoph Stotz <pif@logo.de>]

Hello Nico!

Sorry, my email was send to the mailing list too, it goes "BCC" - my
error.

Writing a loop is a nice Idea, but I�m generating this batch-Script from
a Windows-PC connected to a SQL-Server where all the Data is stored.
Every
fax is diffrent (Name, Address, even logos and firms). Fax-Attributes
may
change (Priority, Modempool, Send-Time etc.). I thought about
trasmitting
every fax by connecting directly to the daemon (ftp isn�t really
difficult),
but I have to resend the failures 2 or 3 times and using my batch-script
I don�t have to regenerate every time the tiff�s...

Don�t worry about your time - You already helped me and maybe some other
flexfax-User has some more idea what could by the problem. My first
step will be to buy another 64/128MB of Memory...

Thanx!

Christoph Stotz
logo: GmbH
-----Urspr�ngliche Nachricht-----
Von: Nico Kadel-Garcia [mailto:raoul@cirl.meei.harvard.edu]
Gesendet am: Freitag, 24. Juli 1998 17:04
An: Christoph Stotz
Betreff: Re: AW: flexfax: Problems submitting very large Number of jobs
sequen tial ly

-----BEGIN PGP SIGNED MESSAGE-----

Chris, I'd much rather look at this kind of stuff on the mailing list:
it gets more people involved.

> The answers to you questions are:
> 
> Operating System is Linux 2.0.30.
> Hylafax is 4.0pl2.
> 
> The batch script looks like that:
> 
> <---cut---><---cut---><---cut---><---cut--->
> faxconfig ModemClass
> "exgroup:ttyD2|ttyD3|ttyD4|ttyD5|ttyD6|ttyD7|ttyX00|ttyX04"
> sendfax -n -s 'a4' -t 3 -k 'now + 1 month' -m -B 14400 -b 2400 -i
> 'ex0001' -h exgroup@ -d '<number_to_dial1>' <filename1>.tif
> sendfax -n -s 'a4' -t 3 -k 'now + 1 month' -m -B 14400 -b 2400 -i
> 'ex0002' -h exgroup@ -d '<number_to_dial2>' <filename2>.tif
> sendfax -n -s 'a4' -t 3 -k 'now + 1 month' -m -B 14400 -b 2400 -i
> 'ex0003' -h exgroup@ -d '<number_to_dial3>' <filename3>.tif
> sendfax -n -s 'a4' -t 3 -k 'now + 1 month' -m -B 14400 -b 2400 -i
> 'ex0004' -h exgroup@ -d '<number_to_dial4>' <filename4>.tif
> sendfax -n -s 'a4' -t 3 -k 'now + 1 month' -m -B 14400 -b 2400 -i
> 'ex0005' -h exgroup@ -d '<number_to_dial5>' <filename5>.tif
> (5000 lines like these following)
> <---cut---><---cut---><---cut---><---cut--->

AAAARRRGGGHHH! You are cranking up *HOW MANY* sendfaxes via shell
script?
Wouldn't it make more sense to write a small loop for this? And to use
the
same sendfax command and file to send to multiple recipients?

> Ovviously the "<..>" strings in the real batch-file are replaced by
real
> data.
> 
> The Spool partition is 3.5GB. With 5000 faxes on it and (!) in the
queue
> it is 93% full.

GACK!

> My Swap-Partition is about 130MB (I know, I�m Crazy ;-)
> Total (real) Memory is 64MB. During the Submitting-Phase (after 2000
> faxes), cat /proc/meminfo gives:

Swap looks reasonable for your system size, but oh, my ghod, that is a
lot of faxes. 

> <---cut---><---cut---><---cut---><---cut--->
>         total:    used:    free:  shared: buffers:  cached:
> Mem:  65560576 64159744  1400832 15183872 19591168 30703616
> Swap: 133885952  8286208 125599744
> MemTotal:     64024 kB
> MemFree:       1368 kB
> MemShared:    14828 kB
> Buffers:      19132 kB
> Cached:       29984 kB
> SwapTotal:   130748 kB
> SwapFree:    122656 kB
> <---cut---><---cut---><---cut---><---cut--->

Looks like you're running out of memory, doesn't it? When you start
running
out of real memory in a lot of active processes, expect a lot of
swapping.
Do you hear the disk grinding a lot when this happens?

> hfaxd is running as standalone and is started at bootup by
> /sbin/init.d/hylafax start.

Good.

> Why do you believe I�m overloading hfaxd with too many TCP-Connections
?
> I believe that
> the tcp-Connection is made only in the moment of submitting the job.
If
> it is in the
> Queue, waiting for a line to become free, it should not use any
> resource...

As I understand it, the TCP connection transmits the fax data from
sendfax, through the hfaxd, to the faxq, in an FTP like fashion.  Thus
each sendfax opens up its own TCP connection for a significant amount
of time, and operating systems can only handle so many TCP
requests. Then they tend to crash.

I could be wrong: the HylaFAX could be using hfaxd in this case merely
to notify the faxq of a file stashed in its queue by sendfax, but I
doubt it....

> By the way: may I submit a job without using "sendfax", writing
directly
> the Queue-File ?

Maybe? I never explored that. You'd have to make sure the files were
pre-processed into TIFF correctly, but I think you're already doing
that.

It's not clear to me whether the batch script you use needs genuinely
different TIFF files, or whether you can bundle up your faxes in the
same sendfax script and save a *lot* of disk and processing time, which
might help.

If you need a more thorough consultation on this, maybe we should
arrange a teleconsulting setup and professional rates: I'm reluctant
to get too deeply into your setup on my own limited free time....

			Nico Garcia
			Senior Engineer, CIRL 
			Mass. Eye and Ear Infirmary
			raoul@cirl.meei.harvard.edu

-----BEGIN PGP SIGNATURE-----
Version: 2.6.2

iQCVAwUBNbiiYD/+ItycgIJRAQG/HgP+IueS8RFgWcJ3Rytya6+Qz6M78mzogfdZ
XTH5ElPacjf5q3moC8+KJVG0Ok+piEsPVqEEwohfnl2Jzjf1sZAyBpUOY8Hn0jf6
ukgowhu02t/Q/zOKyzzSqK+lzeepx/OJ7U238Vh/Tmk285ofwIBIptpQOvjtL9Wv
cAXNKpNm594=
=ry8t
-----END PGP SIGNATURE-----