Hi Marc,
Just for reference---it may not be possible or optimal in your
implementation---the way I was thinking of making this streamable is
the following:
1. First do a for-each-group using group-adjacent to break the set
into large sub-sets (large enough where multi-threading is still
useful but small enough that they fit inside your memory capacity.)
Since this uses group-adjacent it is streamable.
2. Within the for-each-group, create a map whose fibers are the
individual groups you want to process using multiple threads.
3. Process those groups using a for-each with the saxon:threads attribute.
On Tue, Jan 27, 2015 at 9:22 PM, Marc Kupietz kupietz(_at_)ids-mannheim(_dot_)de
<xsl-list-service(_at_)lists(_dot_)mulberrytech(_dot_)com> wrote:
Thanks, Michael and David for the quick replies! I already had the
suspicion that if it was be so easily transferrable from xsl:for-each,
in a proper way it would probably be done, already.
David's map+stream approach brought be to an even easier solution for
my problem: As it mainly concerns the 2nd stage of a 2-stage
conversion I can easily produce several "grouped" result-documents
(maybe named pipes) in the first stage and process them in parallel
and using streaming in the second stage. This way I can use adjustable
package sizes and all cores without running out of memory and
hopefully end-up with i/o as bottleneck :)
Marc
--
"A false conclusion, once arrived at and widely accepted is not
dislodged easily, and the less it is understood, the more tenaciously
it is held." - Cantor's Law of Preservation of Ignorance.
--~----------------------------------------------------------------
XSL-List info and archive: http://www.mulberrytech.com/xsl/xsl-list
EasyUnsubscribe: http://lists.mulberrytech.com/unsub/xsl-list/1167547
or by email: xsl-list-unsub(_at_)lists(_dot_)mulberrytech(_dot_)com
--~--