Hello,
I've subscribed myself to the list now but if the admin releases the
email I sent earlier
you'll get a message out of order.
I've taken Clint's UIDL patch and applied it to my Debian fetchmail
archive because
I needed the functionality. The UIDL matching in IMAP is useful for
keeping a
backup of an IMAP mailbox without deleting all of the mail off the server or
interfering with the SEEN status. I can see it being a problem with
lots (1000+)
of emails on the remote system because of the necessary polling, but I
figure
that's something the user needs to be aware of.
At any rate, I've attached a new patch based on Clint's that fixes an
issue I
was having with multiple folder retrieval. It wouldn't mark the right
hashes as "seen" from the second and onward mailboxes if fetchmail was
told to
pull mail from multiple mailboxes.
Since each folder starts over from 1 for the 'um' the id_find function was
getting confused. To fix it I've added a "seen message" 'um' that is
(unsigned int)-1.
I then check to see if some item in the list has the existing 'um' then
mark that item
with the seen 'um'. The hashes marked seen are no longer needed for
message retrieval,
only hash verification so the ums for "mark_seen" work properly over
multiple mailboxes.
Here's the rundown:
Benefits:
*) Allows synchronizing of remote IMAP mailbox like POP.
This is useful if there is no POP access to the server.
*) Leaves the READ flag alone on the remote server.
Useful if there's an interactive IMAP client also attached
to the mailbox.
Drawbacks:
*) Has to download every header and compile the hash.
This could be an issue in a mailbox with a large
number of messages.
Depends on how you look at it:
*) Doesn't check for marked-deleted state before computing hash and
downloading. This can be bad because the backup mailbox will have
messages that are marked but not yet expunged. This can be good
because the backup mailbox will have messages that are marked but
not yet expunged. ;-)
*) If multiple mailboxes are downloaded it will only download messages
that have matching hashes once. I've found this to be a benefit because
a Mozilla "move" filter will mark a message for deletion but not expunge
the mailbox. My test runs have downloaded the "marked for deletion"
message
then skipped the copy in the other mailbox. Of course I'm running a
procmail
filter on the side running fetchmail so it sorts the email into the
proper
local mailbox.
There may be a way to make Mozilla move then expunge but I haven't
played with
the filter enough to be certain what effects adding a "delete"
modifier on the
filter will have (if any).
Room for improvement:
*) Implement a faster way to look up hashes.
*) Find a way to expunge the mailbox before reading any messages without
breaking any existing functionality.
Thanks,
Paul
Only in fetchmail-6.2.5.orig/contrib: RCS
diff -r fetchmail-6.2.5.orig/imap.c fetchmail-6.2.5/imap.c
41a42,46
/* this is needed for multiple mailboxes
* with uidl support
*/
static unsigned int um_seen = (unsigned int)-1;
708c713,843
< if (!ctl->fetchall && count > 0)
---
/* Adding support for using Unique IDs (although we have to generate
* them for ourselves) to determine which messages we should fetch.
* We will rely on the uidl parameter already in the config file
* to specify this parameter for us
*/
if (ctl->server.uidl && ctl->keep)
{
if (unseen_messages)
free(unseen_messages);
unseen_messages = xmalloc(count * sizeof(unsigned int));
memset(unseen_messages, 0, count * sizeof(unsigned int));
unseen = 0;
gen_send(sock, "FETCH 1:* (BODY.PEEK[HEADER.FIELDS (MESSAGE-ID
RECEIVED)])");
do
{
char *ep;
ok = gen_recv(sock, buf, sizeof(buf));
if (ok != 0)
{
report(stderr, GT_("retreiving headers for IMAP UIDL hash
failed\n"));
return(PS_PROTOCOL);
}
else
{
unsigned int um, size;
if (sscanf(buf, "* %u FETCH %*s %*s %*s {%u}", &um, &size) == 2)
{
char *header = NULL;
/* Read all subsequent lines until the line begins with
). */
ok = 0;
do
{
ok = gen_recv(sock, buf, sizeof(buf));
if (ok != 0)
{
report(stderr, "reading headers for IMAP UIDL
hash failed\n");
return(PS_PROTOCOL);
}
if (header == NULL)
{
header = xmalloc(strlen(buf)+3);
header[0] = '\0';
}
else
{
header = xrealloc(header,
strlen(header)+strlen(buf)+3);
}
if (strlen(buf)>0 && buf[0] != ')')
{
int hl;
strcpy(header+strlen(header), buf);
hl = strlen(header);
/* useless - optmization: elminate
header[strlen(header)] = '\r';
*/
header[hl] = '\n';
header[hl+1] = '\0';
}
} while (buf[0] != ')' && ok == 0);
/* We now have the header so lets make a hash and
determine if we've seen it before */
char headdigest[33];
strcpy(headdigest, MD5Digest((unsigned char *)header));
if (outlevel >= O_DEBUG)
report(stderr, "header hash: %s\n", headdigest);
if (header != NULL)
free(header);
/* The following code was ripped almost verbatim from
pop3.c in pop3_getrange */
struct idlist *old, *new;
/* look to see if this id is already taken, if so
* change it to the already_seen mark.
*/
new = id_find(&ctl->newsaved, um );
if ( 0 != new )
{
new->val.status.num = um_seen;
}
new = save_str(&ctl->newsaved, headdigest, UID_UNSEEN);
new->val.status.num = um;
if ((old = str_in_list(&ctl->oldsaved, headdigest,
FALSE)))
{
flag mark = old->val.status.mark;
if (mark == UID_DELETED || mark == UID_EXPUNGED)
{
if (outlevel >= O_VERBOSE)
report(stderr, GT_("id=%s (num=%d) was
deleted but is still present!\n"), headdigest, um);
/* mark it as seen now */
old->val.status.mark = mark = UID_SEEN;
}
new->val.status.mark = mark;
if (mark == UID_UNSEEN && um <= count)
{
unseen_messages[unseen++] = um;
if (outlevel >= O_DEBUG)
report(stderr, GT_("%u is unseen.
(marked)\n"), um);
if (startcount > um)
startcount = um;
}
}
else if (um <= count)
{
unseen_messages[unseen++] = um;
if (outlevel >= O_DEBUG)
report(stderr, GT_("%u is unseen. (saving)\n"),
um);
if (startcount > um)
startcount = um;
/* add it to oldsaved also! In case, we do not
* swap the lists (say, due to socket error),
* the same mail will not be downloaded again.
*/
old = save_str(&ctl->oldsaved, headdigest,
UID_UNSEEN);
old->val.status.num = um;
}
}
}
} while
(tag[0] != '\0' && strncmp(buf, tag, strlen(tag)));
}
else if (!ctl->fetchall && count > 0)
1077a1213,1244
if (ctl->server.uidl && ctl->keep)
{
struct idlist *sdp;
if (outlevel >= O_DEBUG)
report(stderr, "finding number %d\n", number);
/* Code again ripped from pop.c */
if ((sdp = id_find(&ctl->newsaved, number)))
{
sdp->val.status.mark = UID_SEEN;
if (outlevel >= O_DEBUG)
report(stderr, "marked %d (%s) new seen\n", number, sdp->id);
}
else
return(PS_ERROR);
/* mark it as seen in oldsaved also! In case, we do not swap the lists
* (say, due to socket error), the same mail will not be downloaded
* again.
*/
if ((sdp = id_find(&ctl->oldsaved, number)))
{
sdp->val.status.mark = UID_SEEN;
if (outlevel >= O_DEBUG)
report(stderr, "marked %d (%s) old seen\n", number, sdp->id);
}
else
return(PS_ERROR);
return(PS_SUCCESS);
}
else
{
1082a1250
}