Re: OPES protocol, pre-draft

Oskar,

> Let me propose a simplification of the protocol: remove 
> [copied] flag and make this behavior compulsory.
> Reasons:
>
> 1. The main goal of chunks is to reduce network traffic. OPES server
You probably mean "OPES processor". Let's try to stick to the 
terminology we've in the existing documents, might help avoiding 
misunderstandings.
> implementation that discards message chunk by chunk as soon as one 
> is send to callout server does not look a good idea. Even if call 
> starts before the complete message is received by the OPES processor 
> it should be able to assemble the complete message anyway.
This might require the OPES processor to possibly buffer huge amounts 
of data, which might impact scalability of the approach.
Imagine a large number of users simultaneously downloading large files 
via HTTP, and the OPES processor does a callout for virus scanning for 
each of these files (in parallel). Since virus scanning involves long 
delays in processing, quite some data would accumulate at the OPES 
processor for temporary buffering...
> 2.Reliability. If the original message is discarded by OPES processor 
> before transaction with the callout processor is completed callout 
> processor failure (including connection outage) forces OPES processor 
> to fail on this message, while preserved copy provides multiple 
> ways for recovery - default action on callout server unavailable, 
> repeat request to a different callout server. And the message 
> was already there, savings from premature discarding are very 
> small.
Absolutely right, and that's why our architecture and protocol should 
allow an OPES processor to buffer the data. Then it's up to the OPES 
processor to decide whether to do that or not.
> 3. catch 22: if message processing is simple and anticipated 
> processing time is small - so are the savings from message discarding.
> If anticipated processing time is significant - savings may be 
> bigger but so are the risks related to failure and needs for 
> reliable recovery. 
Yup, see my comments above.

Also, for some transactions it might be OK to fail when the callout 
server fails. Example: If a user requests mandatory virus scanning for 
file download, it might be perfectly fine to reply with an error 
messge when the callout server running the virus scanning fails, 
rather then sending back a non-scanned version from the buffer of the 
OPES processor.
Thanks,
  Markus