AW: AW: flexfax: Problems submitting very large Number of jobs sequen tial ly

[Christoph Stotz <pif@logo.de>]

Thank you for your posting. I have a few Questions about it:

How can I use "btrees" for my big directories under Linux ?

May I delete the info/* files after every faxmailing and will I improve
speed in this way ?

Why should I programm a Queue-Controller for the Queue-Controller ?
Writing for
"nr_fax_in_queue<modems+x" seems to me the job of faxq...


Regards,

Christoph Stotz
logo: GmbH


-----Ursprüngliche Nachricht-----
Von: Mike Bristow [mailto:michaelb@bessie.mojo.org]
Gesendet am: Dienstag, 28. Juli 1998 11:12
An: flexfax@sgi.com
Betreff: Re: AW: flexfax: Problems submitting very large Number of jobs
sequen tial ly

Hi,

On Mon, Jul 27, 1998 at 02:42:19AM +0000, Didier Gautheron wrote:
> Hi
> 
> Christoph Stotz wrote:
> > The theory of too many files don´t seem to me to be the solution.
> Yes but a 1500-5000 files in a directory is a problem on all OS. and

[only those that use a linked list for directories; those that use
btrees
 are much happier with big directories.]

> here you have 4 to 5 dirs :
> info
> sendq
> doneq
> docq

I've got a patch that puts the info files in subdirectories under info,
although I don't think that will help here as (AFAIU) as the numbers 
being called are in a small subset of all possible numbers.   (as
opposed to
the tpc cell I see which reached a totally stupid info directory size
with
~30,000 entries)

> > Is it possible to tell hylafax to call a batch-file if the number
> > of jobs in queue is smaller than the number of free modemlines
> > avaible ?
> I don't think you could do this out of the box.
> But you could use some trick on xferlog and or faxwatch to get a tick
> each time a fax is sent.

This is probably the approach I'd take; prep the queue with the number 
of modems + 1 or two; run sendfax everytime a line gets written which 
matches /""$/ or something.

> The text below is over simplified but I hope there's no big mistake.
> 
> The delay is coming from the faxq's scheduler : it's a multi threads
prg
> using fork and exec to achieve its goals.
> The main thread (faxq) is waiting in a loop for events:
> events are coming from:
> fifo : 
> - hfaxq submiting job
> - faxgetty
> - faxsend (not sendfax , faxsend is the prg which actually sends the
> fax)
> child's dead:
> - mainly faxsend
> timeout
> - mainly from jobs waiting queue
> 
> Of course when faxq is not in its waiting loop it doesn't see other
> events coming, they are waiting in system's buffers.
> 
> faxq is shooting itself in the foot and it's pushing your computer in
> the red zone with big queue :
> runScheduler in faxd/faxQueueApp.c++ is processing all the jobs in the
> run gueue even if no modems are in READY state. So in your case the
main
> thread is reading 1500 and up files (jobs status file) for nothing -
if
> the main goal of hylafax is sending faxes, not trashing out of
> delay,tries or whatever faxes, of course - And runSheduler is called
> many times: each time you add a job, each time a job finished, and so
> on.
> 
> It's easy to change this ie : change assignModem behavior so
> runScheduler will return when there's no more modems available . I
tried
> and yes now with a 10000 run queue, sendfax takes the same time. 
> But side effects are unknow, after all it's just faxq's main routines
> :).

Perhaps it would be better to change:

while (getnextentryinqueue() != NULL) {
.

to:

while (getnextentryinqueue() != NULL && ++counter < QUEUE_TOO_BIG) {
.

[or whatever]

That will help cut faxq down to size when it's deciding which fax to
send
as well; when the queue has 9,000,000 entries it's probably less
important 
to weight each entry and ponder on the relative merits of each one.  
QUEUE_TOO_BIG 	should probably be at least a compile time, probably
better
as run time, tweakable.  It should be different on a 386 with a shonky
modem
compared to a dual P-II 400 with a bank of 256 modems. :-)

-- 
"Virgin Trains: It's quicker by balloon."
	-- Peter Corlett