Brian Candler wrote:
On Thu, May 20, 2004 at 10:00:18AM -0700, Clint Sharp wrote:
Also the
Unique IDs are not consistent between sessions (they are usually just
the message number in the mailbox which will change with an expunge).
That's also untrue. From RFC 3501:
2.3.1.1. Unique Identifier (UID) Message Attribute
A 32-bit value assigned to each message, which when used with the
unique identifier validity value (see below) forms a 64-bit value
that MUST NOT refer to any other message in the mailbox or any
subsequent mailbox with the same name forever. Unique identifiers
are assigned in a strictly ascending fashion in the mailbox; as each
message is added to the mailbox it is assigned a higher UID than the
message(s) which were added previously. Unlike message sequence
numbers, unique identifiers are not necessarily contiguous.
The unique identifier of a message MUST NOT change during the
session, and SHOULD NOT change between sessions. Any change of
unique identifiers between sessions MUST be detectable using the
UIDVALIDITY mechanism discussed below. Persistent unique identifiers
are required for a client to resynchronize its state from a previous
session with the server (e.g., disconnected or offline access
clients); this is discussed further in [IMAP-DISC].
Translated: the UID never changes, unless the server for some reason loses
its state and has to renumber the messages from scratch. In that case, the
UIDVALIDITY changes to allow the client to detect that this renumbering has
taken place.
Brian.
_______________________________________________
Fetchmail-friends mailing list
Fetchmail-friends(_at_)lists(_dot_)ccil(_dot_)org
http://lists.ccil.org/cgi-bin/mailman/listinfo/fetchmail-friends
It is true on the reference IMAP implementation I used (UW IMAP). I
read the RFC, as I attempted to implement this patch first with IMAP UID
listings (UID SEARCH), as this would have been much bandwidth effecient
than polling the entire set of headers for the mailbox every time
fetchmail logs into check email. I'm aware of the downsides of the
patch, however, I can't think of a better way to do it that will work
properly across various IMAP server implementations.
The RFC States: "The unique identifier of a message MUST NOT change
during the session, and SHOULD NOT change between sessions." SHOULD NOT
change between the sessions is the key word. However, it CAN change,
which means that this is useless in my implementation. First of all,
it's only guaranteed consistent with the same UIDVALIDITY value.
However, UW IMAP implementation doesn't even keep the same UIDVALIDITY
between sessions, even without an expunge or any sort of internal
reordering of the email, as seen by the example at the end of the
email. Exchange keeps consistent UID values even after deleting
messages and expunging, but we need to work on the lowest common
denominator, which in my testing was UW. I don't have access to Cyrus
or any of the other popular IMAP servers to test. UIDs are only unique
combined with the UIDVALIDITY value, and UW as my reference
implementation for designing this patch did not maintain any sort of
consistent state, and thusly this method was useless for me (I suppose
UW's implementation would probably be fairly useless for anyone trying
to do offline IMAP as well). This would wreak havoc on fetchmail if the
UID numbers are reallocated, causing us to redownload and re-deliver
email which was already seen and possibly missing mail which was never
seen. Also, what would we do when the UIDVALIDITY value changed?
Redeliver all mail? We have no cached copies of the mail which was
delivered, so there would be no way to compare what has been delivered,
we'd be forced to redeliver all mail.
I don't mean to contradictory, but I don't see any way IMAP's UID
facilities could possibly work for fetchmail, given the inconsitencies
of the implementation and even the fact that the implementation says
that the unique key (Mailbox+UID+UIDVALIDITY) SHOULD NOT, but can
change. In this case, I think the method I used to implement the patch
(MD5Sum of the Message-ID and Received headers, which should be much
closer to globally unique) is the only way to guarantee not missing mail
and guarantee that we're not going to re-deliver mail.
Clint
Example of UW's UIDVALIDITY issue:
-bash-2.05b$ telnet localhost 143
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS AUTH=LOGIN]
localhost.localdomain IMAP4rev1 2001.315rh at Thu, 20 May 2004 15:56:19
-0500 (CDT)
A1 LOGIN * *
A1 OK [CAPABILITY IMAP4REV1 IDLE NAMESPACE MAILBOX-REFERRALS SCAN SORT
THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND] User clint
authenticated
A2 SELECT "INBOX"
* 934 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1085086598] UID validity status
* OK [UIDNEXT 935] Predicted next UID
* NO [UIDNOTSTICKY] Non-permanent unique identifiers:
/home/typhoon/homes/clint/mail.txt
* FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Draft \Seen)]
Permanent flags* OK [UNSEEN 31] first unseen message in
/home/typhoon/homes/clint/mail.txt
A2 OK [READ-WRITE] SELECT completed
A3 LOGOUT
* BYE stage IMAP4rev1 server terminating connection
A3 OK LOGOUT completed
Connection closed by foreign host.
-bash-2.05b$ telnet localhost 143
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4REV1 LOGIN-REFERRALS STARTTLS AUTH=LOGIN]
localhost.localdomain IMAP4rev1 2001.315rh at Thu, 20 May 2004 15:56:44
-0500 (CDT)
A1 LOGIN * *
A1 OK [CAPABILITY IMAP4REV1 IDLE NAMESPACE MAILBOX-REFERRALS SCAN SORT
THREAD=REFERENCES THREAD=ORDEREDSUBJECT MULTIAPPEND] User clint
authenticated
A2 SELECT "INBOX"
* 934 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1085086614] UID validity status
* OK [UIDNEXT 935] Predicted next UID
* NO [UIDNOTSTICKY] Non-permanent unique identifiers:
/home/typhoon/homes/clint/mail.txt
* FLAGS (\Answered \Flagged \Deleted \Draft \Seen)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Draft \Seen)]
Permanent flags* OK [UNSEEN 31] first unseen message in
/home/typhoon/homes/clint/mail.txt
A2 OK [READ-WRITE] SELECT completed
A3 LOGOUT
* BYE stage IMAP4rev1 server terminating connection
A3 OK LOGOUT completed
Connection closed by foreign host.