Martin,
I agree that a message type is missing from the proposed
protocol. While your motivation is perfectly valid, there is an even
more general reasoning that leads to the same conclusion:
The protocol relies on timeouts to resolve deadlocks from
broken implementations and other situations where processing may get
stuck. A timeout trigger would be reset if there is any activity
related to a given transaction. If data stops flowing (because of a
valid delay like loading a virus archive), there must be a way to
create a data-less activity: "everything is normal, I am just slow,
hang on".
I suggest adding a still-here message:
<still-here>
<still-here xid>
<still-here xid amid>
which can be used by both OPES processor and callout server, as
needed. The message does not affect the state machine except it may
reset the corresponding timeout trigger at the recipient side. If the
sender uses no IDs, the message can be used to keep idle OPES
connection alive and for similar keep-alive purposes unrelated to any
particular transaction. If xid is used, all OPES-trouble-detection
timeouts for that transaction should be reset. If amid is used, all
trouble-detection timeouts for that application message should be
reset (but other application messages may still timeout, depending on
the state and protocol).
Would keep-alive be a better name? I prefer still-here because it
does not imply a specific action. It is up to the other side what to
do when a still-here message is received. This blends well with OPES
protocol design by keeping processor and callout server states as
independent as possible. We probably do not care about specific names
at this point though.
I hesitate adding "change-probability" and "progress" fields. I am not
convinced that they are the best way to implement what you want.
change-probability field: if the primary purpose is to fight latency,
we should address that directly. Latency does not have to be related
to pending _changes_:
- Download progress indication: Always a good idea if
application protocol supports it. Callout server is
a poor judge of the progress though because it is just
a hop in the pipeline. There may be more callout servers,
for example. OPES processor is in a better position
to estimate progress (based on past performance). User
Agent is in an even better position (and most support
this feature!).
- Data trickling: same as download progress indication;
if application protocol supports sending empty messages,
they can be sent. Knowing change-probability does not
help with that. If callout server can trickle real
data, it should do it anyway.
- LateClearance: seems like this should ALWAYS be used
(if possible) for content that MAY have a virus because
change-probability for such content is always positive at
the beginning and because LateClearance overhead can be made
very low.
progress field: the semantics seems to be covered by the still-here
message itself. "I am still here (doing something, naturally),
hang on if you can!"
Am I downplaying the importance of change-probability and progress
fields? Do you expect implementations hack them in as OPES extensions
if we do not define them? Are there corresponding ICAP extensions
already?
On a related note, would it be of any benefit to add a flag to
data-have message from the callout server indicating that the
corresponding data has been modified by the callout server? Or should
we address that when we start talking about
authentication/privacy/etc. support?
Thank you,
Alex.
On Fri, 21 Feb 2003, Martin Stecher wrote:
There is one common scenario with virus filtering in a callout server
that I wrote about some time ago.
Do you think an additional message type could help and is worth here?
Scenario:
A virus filter often cannot send chunks of data back to the OPES
processor before it has the complete data. Only then it can start
the scanning process.
On the other hand it modifies data only in rare situations.
Most results are "no virus found, use original message unmodified".
Only if virus was found, it modifies (repair) the data or replaces
it in parts or completly by an error message.
So a possible message from callout server to OPES processor could be
<data-not-yet xid amid change-probability progress>
It informs the OEPS processor that the callout server is working on
this although it does not reply data now. If "change-probability" is
low, the OPES processor could use techniques to fight the raising
latency time,
for example:
- some kind of download progress indication,
- "data trickling" or
- the proposed LateClearance content encoding
(see draft-stecher-lclr-encoding-00.txt)
The "progress" parameter can inform that the callout processor needs
even more time although all data was already received (e.g. an archive
needs to be extracted first which is a currently ongoing lengthy
process)
Thanks
Martin