fetchmail-friends
[Top] [All Lists]

Re: [fetchmail]Fixed UIDL + IMAP patch

2004-05-20 14:43:38
Brian Candler wrote:

On Thu, May 20, 2004 at 10:00:18AM -0700, Clint Sharp wrote:
Also the Unique IDs are not consistent between sessions (they are usually just the message number in the mailbox which will change with an expunge).

That's also untrue. From RFC 3501:

2.3.1.1.        Unique Identifier (UID) Message Attribute

  A 32-bit value assigned to each message, which when used with the
  unique identifier validity value (see below) forms a 64-bit value
  that MUST NOT refer to any other message in the mailbox or any
  subsequent mailbox with the same name forever.  Unique identifiers
  are assigned in a strictly ascending fashion in the mailbox; as each
  message is added to the mailbox it is assigned a higher UID than the
  message(s) which were added previously.  Unlike message sequence
  numbers, unique identifiers are not necessarily contiguous.

  The unique identifier of a message MUST NOT change during the
  session, and SHOULD NOT change between sessions.  Any change of
  unique identifiers between sessions MUST be detectable using the
  UIDVALIDITY mechanism discussed below.  Persistent unique identifiers
  are required for a client to resynchronize its state from a previous
  session with the server (e.g., disconnected or offline access
  clients); this is discussed further in [IMAP-DISC].

Translated: the UID never changes, unless the server for some reason loses
its state and has to renumber the messages from scratch. In that case, the
UIDVALIDITY changes to allow the client to detect that this renumbering has
taken place.

Brian.

_______________________________________________
Fetchmail-friends mailing list
Fetchmail-friends(_at_)lists(_dot_)ccil(_dot_)org
http://lists.ccil.org/cgi-bin/mailman/listinfo/fetchmail-friends

It is true on the reference IMAP implementation I used (UW IMAP). I read the RFC, as I attempted to implement this patch first with IMAP UID listings (UID SEARCH), as this would have been much bandwidth effecient than polling the entire set of headers for the mailbox every time fetchmail logs into check email. I'm aware of the downsides of the patch, however, I can't think of a better way to do it that will work properly across various IMAP server implementations.

The RFC States: "The unique identifier of a message MUST NOT change during the session, and SHOULD NOT change between sessions." SHOULD NOT change between the sessions is the key word. However, it CAN change, which means that this is useless in my implementation. First of all, it's only guaranteed consistent with the same UIDVALIDITY value. However, UW IMAP implementation doesn't even keep the same UIDVALIDITY between sessions, even without an expunge or any sort of internal reordering of the email, as seen by the example at the end of the email. Exchange keeps consistent UID values even after deleting messages and expunging, but we need to work on the lowest common denominator, which in my testing was UW. I don't have access to Cyrus or any of the other popular IMAP servers to test. UIDs are only unique combined with the UIDVALIDITY value, and UW as my reference implementation for designing this patch did not maintain any sort of consistent state, and thusly this method was useless for me (I suppose UW's implementation would probably be fairly useless for anyone trying to do offline IMAP as well). This would wreak havoc on fetchmail if the UID numbers are reallocated, causing us to redownload and re-deliver email which was already seen and possibly missing mail which was never seen. Also, what would we do when the UIDVALIDITY value changed? Redeliver all mail? We have no cached copies of the mail which was delivered, so there would be no way to compare what has been delivered, we'd be forced to redeliver all mail.

I don't mean to contradictory, but I don't see any way IMAP's UID facilities could possibly work for fetchmail, given the inconsitencies of the implementation and even the fact that the implementation says that the unique key (Mailbox+UID+UIDVALIDITY) SHOULD NOT, but can change. In this case, I think the method I used to implement the patch (MD5Sum of the Message-ID and Received headers, which should be much closer to globally unique) is the only way to guarantee not missing mail and guarantee that we're not going to re-deliver mail.

Clint

Example of UW's UIDVALIDITY issue:

-bash-2.05b$ telnet localhost 143
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS AUTH=LOGIN] localhost.localdomain IMAP4rev1 2001.315rh at Thu, 20 May 2004 15:56:19 -0500 (CDT)
A1 LOGIN * *
A1 OK [CAPABILITY IMAP4REV1 IDLE NAMESPACE MAILBOX-REFERRALS SCAN SORT THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND] User clint authenticated
A2 SELECT "INBOX"
* 934 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1085086598] UID validity status
* OK [UIDNEXT 935] Predicted next UID
* NO [UIDNOTSTICKY] Non-permanent unique identifiers: /home/typhoon/homes/clint/mail.txt
* FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Draft \Seen)] Permanent flags* OK [UNSEEN 31] first unseen message in /home/typhoon/homes/clint/mail.txt
A2 OK [READ-WRITE] SELECT completed
A3 LOGOUT
* BYE stage IMAP4rev1 server terminating connection
A3 OK LOGOUT completed
Connection closed by foreign host.
-bash-2.05b$ telnet localhost 143
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS AUTH=LOGIN] localhost.localdomain IMAP4rev1 2001.315rh at Thu, 20 May 2004 15:56:44 -0500 (CDT)
A1 LOGIN * *
A1 OK [CAPABILITY IMAP4REV1 IDLE NAMESPACE MAILBOX-REFERRALS SCAN SORT THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND] User clint authenticated
A2 SELECT "INBOX"
* 934 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1085086614] UID validity status
* OK [UIDNEXT 935] Predicted next UID
* NO [UIDNOTSTICKY] Non-permanent unique identifiers: /home/typhoon/homes/clint/mail.txt
* FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Draft \Seen)] Permanent flags* OK [UNSEEN 31] first unseen message in /home/typhoon/homes/clint/mail.txt
A2 OK [READ-WRITE] SELECT completed
A3 LOGOUT
* BYE stage IMAP4rev1 server terminating connection
A3 OK LOGOUT completed
Connection closed by foreign host.