fetchmail-friends
[Top] [All Lists]

[fetchmail]lost UIDs

2005-03-05 07:15:09
I have a problem with fetchmail, which occurs every once in a while (and
also happened with older versions of fetchmail). I use fetchmail in
daemon mode (started from my KDE startup script when I log in) to poll
two mail servers.
Sometimes when I log in the .fetchids file is very short, it contains
only very few uids. And therefore fetchmail loads down very many old
mails. And since I get very much spam, this is very much, often several
thousand mails. My first thought was, that probably fetchmail is killed
in a bad moment during writing the .fetchids by system shutdown. This is
one possibility, but I had a look into the code and found a bug, which
also could cause this problem.

In the function uid_swap_lists (in uid.c) there is this code:
    if (ctl->newsaved)
    {
        /* old state of mailbox may now be irrelevant */
        if (outlevel >= O_DEBUG)
            report(stdout, GT_("swapping UID lists\n"));
        free_str_list(&ctl->oldsaved);
        ctl->oldsaved = ctl->newsaved;
        ctl->newsaved = (struct idlist *) NULL;
    }

So first the old oldsaved list is freed, and then it is overwritten with
the newsaved list. If the TERM signal arrives during the free_str_list,
an incomplete oldsaved list will be saved to .fetchids. And since
free_str_list is implemented by a recursion, it really shortens the list
beginning at the end (BTW I do not know, why this is implemented by a
recursion rather then by a loop. If there are many messages on the
server this could lead to a really deep recursion with the possibility
of a stack overflow).
IMHO the correct solution would be:
    if (ctl->newsaved)
    {
        struct idlist *temp;
        /* old state of mailbox may now be irrelevant */
        if (outlevel >= O_DEBUG)
            report(stdout, GT_("swapping UID lists\n"));
        temp = ctrl->oldsaved;
        ctl->oldsaved = ctl->newsaved;
        ctl->newsaved = (struct idlist *) NULL;
        free_str_list(&ctl->oldsaved);
    }


An additional good idea would be not to overwrite the old idfile, but write to new idfile (e.g. ~/.fetchids-new), and after completion rename the file (e.g. "rename("~/.fetchids-new", "~/.fetchids");" or similar). Since there is a comment in the code "FIXME: do not overwrite the old idfile" I assume that you already planned such change and I wonder, why you did not implement it. I think, it is a very easy change, it is probably just about 3 or 4 additional lines; if you want, I can provide you a patch.


Here is the information you require (according to the FAQ) in a bug report:
1. Linux 2.6.10 (but the problem occurred also with older versions)
2. fetchmail_6.2.5-12_i386.deb
3. I do not see a greeting when logging into the pop3 server (not
relevant for the problem anyway).
4. Exim 4.34
5. -d 600
5.
This is fetchmail release 6.2.5+NTLM+SDPS+SSL+NLS
Fallback MDA: (none)
Linux pc235 2.6.10 #1 Mon Dec 27 10:43:40 CET 2004 i686 GNU/Linux
Taking options from command line and /home/weihs/.fetchmailrc
Idfile is /home/weihs/.fetchids
Fetchmail will show progress dots even in logfiles.
Fetchmail will forward misaddressed multidrop messages to weihs.
Options for retrieving from XXXX(_at_)XXX(_dot_)XXX(_dot_)XXX(_dot_)XX:
  True name of server is mail.ict.tuwien.ac.at.
  Protocol is POP3 (forcing UIDL use).
  All available authentication methods will be tried.
  SSL encrypted sessions enabled.
  Server nonresponse timeout is 300 seconds (default).
  Default mailbox selected.
  Only new messages will be retrieved (--all off).
  Fetched messages will be kept on the server (--keep on).
  Old messages will not be flushed before message retrieval (--flush off).
  Rewrite of server-local addresses is enabled (--norewrite off).
  Carriage-return stripping is disabled (stripcr off).
  Carriage-return forcing is disabled (forcecr off).
  Interpretation of Content-Transfer-Encoding is enabled (pass8bits off).
  MIME decoding is disabled (mimedecode off).
  Idle after poll is disabled (idle off).
  Nonempty Status lines will be kept (dropstatus off)
  Delivered-To lines will be kept (dropdelivered off)
  Fetch message size limit is 100 (--fetchsizelimit 100).
  Do binary search of UIDs during 9 out of 10 polls (--fastuidl 10).
  Messages will be SMTP-forwarded to: localhost (default)
  Single-drop mode: 1 local name(s) recognized.
  4098 UIDs saved.


Manfred Weihs


PS: It seems that the modarator of this list is no more active. I already submitted this bug report twice in February and only got a mail that the modarator has to review it for approval (because of post by non-member to a members-only list). But I neither got a notification of the moderator's decision nor was my message posted to the list. So obviously the current moderator does not review these posts. Therefore you should either allow non-member posts or get a new (active) moderator, because now it is necessary to become a list member just for submitting a bug report. This is very inconvenient.


<Prev in Thread] Current Thread [Next in Thread>