Alex Rousskov wrote:
If the callout server uses "data-have-as-is" message that matches the
entire data block sent by the dispatcher, then the dispatcher can
probably free the data (we do not have to support duplication of data,
I guess; if a byte range has been sent, it is unlikely to be sent
again).
Yup, I would think so. When the callout server replies with a
<data-as-is>, the OPES processor starts forwarding the refered to,
copied data segments and frees the respective buffer as the data is
forwarded.
However, what happens if there is a partial match? Should the
dispatcher keep the leftovers until the end of the message? Until
copied data with higher offset is reused? Until wont-need message is
received? A combination of these?
Hm... I would assume until either data with higher offset is reused or
the end of the message is reached.
In the first case, it's probably safe to assume that data with a lower
offset has already been handled properly (since we require ordered and
reliable delivery of callout protocol messages). In the later case,
the callout transaction is finished and the associated buffers can be
release (after possibly forwarding copied data).
Even if the callout server sends a <wont-need> message, the OPES
processor might prefer to keep a copy, for example to recover from a
potential failure of the callout server.
There is also a potential deadlock situation here. If the amount of
dispatcher buffer is limited, the dispatcher may stop sending more
data to the callout server, and the callout server may need more data
to respond (and let OPES dispatcher clear some of the copied chunks).
We may want to address this explicitly instead of relying on
timeouts.
Now, this doesn't solve the problem, but I'm just wondering why would
the OPES processor stop sending more data when its buffer is limited
or filled up? The OPES processor would still receive data from the
direction of the content produce/consumer, right? In this case, if the
OPES processor runs out of buffer, the data would be lost anyway so
the OPES processor can also keep forwarding it to the callout server...
Thanks,
Markus