Re: [openpgp] [Cfrg] streamable AEAD construct for stored data?

On Nov 7, 2015, at 10:54 AM, Andy Lutomirski <luto(_at_)amacapital(_dot_)net> 
wrote:
> On Fri, Nov 6, 2015 at 5:46 PM, Bryan Ford <brynosaurus(_at_)gmail(_dot_)com 
> <mailto:brynosaurus(_at_)gmail(_dot_)com>> wrote:
> > 1. How important is the ability to achieve goal #1 above in the OpenPGP
> > format (streaming-mode integrity-checking)?
> Are you willing to accept a format that allows streaming decryption
> but not streaming encryption?  If so, then you'd only need one
> signature if you organize your Merkle tree correctly.  In fact:
That’s certainly one reasonable use-case to consider, but I don’t think it 
fully satisfies the “filter-mode encrypted backup” scenario that DKG originally 
brought up.  If I’m encrypting and streaming my huge backups to a huge network 
connection, then I don’t want the encryption side to have to store the whole 
huge backup tarball locally either.  My intuition is that if the "filter-mode 
streaming backup” scenario is important on either the encryption or decryption 
side, then it’s probably important to be able to handle on both sides.

On the other hand, it might be quite reasonable to define an encrypted message 
format that allows either mode of operation, and have the encryptor choose the 
mode of operation automatically depending on the situation.  If the encryptor 
sees that both its input and its output are regular files, i.e., it knows the 
total length of its input and can seek in its output, then the encryptor could 
(as you suggest) leave room for a signed Merkle tree at the beginning, process 
everything, then go back to the beginning and fill in the Merkle tree and the 
signature on its root.  Then a streaming-mode decryptor would get the signed 
Merkle tree first and subsequently be able to integrity-check all subsequent 
chunks independently.  However, if the encryptor finds that either it doesn’t 
know the length of its input in advance, or has been told to send output to a 
non-seekable stream (e.g., over an SSH-forwarded network pipe), then the 
encryptor might have to fall back to a more incremental approach, with 
signatures (or signature-plus-incremental-Merkle-trees) after each chunk.

> > 2. How important is the ability to achieve goal #2 above in the OpenPGP
> > format (random-access integrity-checking)?
> It's fairly easy to imagine a format that allows both streaming
> verification and random-access verification with minimal size
> overhead.  You could even create the thing in a semi-streamy manner,
> where you'd stream out the bulk portion with blanks where the internal
> nodes go and then write the internal nodes after the fact.
>
> The best of all worlds might be to treat the Merkle data and the
> signature as a detached file.  I bet that one could streamily encrypt
> and sign a big file and produce *two* output streams: the bulk data
> and a detached serialization of intermediate nodes, where there's a
> single signature at the end.  A reader with access to both files could
> random-access it or seek the detached signature a bit and then stream
> the bulk file.
I think detached files sound convenient to implementers but prove troublesome 
to users who need to know how to juggle and keep track of two files where 
before they only need to keep track of one. :)

B

> --Andy

smime.p7s
Description: S/MIME cryptographic signature

_______________________________________________
openpgp mailing list
openpgp(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/openpgp