xsl-list
[Top] [All Lists]

Re: [xsl] Approach to transform 250GB xml data

2014-09-10 09:43:12
Well, it's unfortunate that this is a bit of a challenge, but it seems to
me there are several XML databases (several are open source or have free
trial periods) that would be able to store the elements in such a way that
you could query in "chunks" and as a worse case, possibly concatenate the
chunks.

XQuery allows you to query sequences, and you can enclose those in some
other element when you concatenate.

Let us know how that goes, Happy to suggest some DB's , but off list,
because I don't want to show some preference and likely would miss some
that are completely capable of this.

Anyway, this get's me thinking about whether a completely different kind of
algorithm could be applied, such as, ahem, pointers to the start of
elements in the text file. An index to the key elements. I suppose that's
not too different from what the XML databases do.

Good luck and let us know how you solved this.

--Hank



On Wednesday, September 10, 2014, Imsieke, Gerrit, le-tex
gerrit(_dot_)imsieke(_at_)le-tex(_dot_)de 
<xsl-list-service(_at_)lists(_dot_)mulberrytech(_dot_)com> wrote:

Out of curiosity: how do you intend to access/process the 250 GB once they
are transformed?

If it is a huge DB dump, maybe you can dump it in slices or, if it is an
XML database with XSLT 2 capabilities, transform it in place.

Gerrit


[snip]




-- 
Hank Ratzesberger
XMLWerks.com
--~----------------------------------------------------------------
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list
EasyUnsubscribe: http://lists.mulberrytech.com/unsub/xsl-list/1167547
or by email: xsl-list-unsub(_at_)lists(_dot_)mulberrytech(_dot_)com
--~--
<Prev in Thread] Current Thread [Next in Thread>