On Fri, 21 Mar 2003, Markus Hofmann wrote:
However, what happens if there is a partial match? Should the
dispatcher keep the leftovers until the end of the message? Until
copied data with higher offset is reused? Until wont-need message is
received? A combination of these?
Hm... I would assume until either data with higher offset is reused
or the end of the message is reached.
In the first case, it's probably safe to assume that data with a
lower offset has already been handled properly (since we require
ordered and reliable delivery of callout protocol messages).
It is not safe in general because there are examples where callout
server might want to reorder message content (e.g., change the parts
order of a multipart e-mail message or move an ad banner to the end of
an HTML page). To make it safe, we would have to explicitly document
this assumption (and, hence, prevent copying optimization in those
rare(?) cases where callout server needs to change the order of
message parts).
In the later case, the callout transaction is finished and the
associated buffers can be release (after possibly forwarding copied
data).
Yes.
Even if the callout server sends a <wont-need> message, the OPES
processor might prefer to keep a copy, for example to recover from a
potential failure of the callout server.
Sure. This kind of copying is out of OCP scope.
There is also a potential deadlock situation here. If the amount
of dispatcher buffer is limited, the dispatcher may stop sending
more data to the callout server, and the callout server may need
more data to respond (and let OPES dispatcher clear some of the
copied chunks). We may want to address this explicitly instead of
relying on timeouts.
Now, this doesn't solve the problem, but I'm just wondering why would
the OPES processor stop sending more data when its buffer is limited
or filled up? The OPES processor would still receive data from the
direction of the content produce/consumer, right? In this case, if the
OPES processor runs out of buffer, the data would be lost anyway so
the OPES processor can also keep forwarding it to the callout server...
The OPES processor will stop reading incoming data when its buffers
are full. That's how most proxies (and servers) I know work today. One
can always control its input, relying on transport protocol to slow
down the producer (TCP) or to drop packets (UDP).
When deadlock occurs, the OPES processor is not reading incoming data
because its buffers are full, and the callout server is not sending
any data back because it waits for more incoming data to make a
decision.
Note that we can make the callout server aware of the deadlock by
sending a <data-pause> message to it (possibly two messages, the last
one in response to <data-need> message from the callout server).
However, "being aware" does not solve the problem. We need either a
mechanism to break the deadlock or protocol changes to avoid it.
Relying on timeouts is bad in this case because it is not an abnormal
situation.
Suggestions?
Alex.