xsl-list
[Top] [All Lists]

RE: [xsl] Regular expression /s whitespace : Which whitespace?

2006-04-21 03:15:09
I am using 
count(tokenize(lower-case(.),'(\s|[,.!:;])+')[string(.)]) -a 
technique I retrieved from the list for counting words. I 
have been questioned about the regular expression that is 
being used to find white spaces. The content can contain many 
kinds of whitespaces and i am being asked to defend using 
this expression to find words. Does the saxon 8b 
interpretation of this regular expression covers as whitespaces

Splitting text into words is potentially much more complicated than this,
depending on what you are trying to achieve and what natural languages you
are trying to process. (For example, in English, how should hypenated words
be handled? What if the hyphen appears just before a newline?) Only you can
judge whether this crude approach is good enough to meet your particular
requirements.

The definition of \s is found in http://www.w3.org/TR/xmlschema-2/#regexs
(which XPath 2.0 references normatively): it matches any one of the four
characters x09, x0A, x0D, x20.

You could consider replacing the second argument of tokenize with 

'\W+' 

which matches any sequence of characters classified in Unicode as
separators, punctuation, or "other" - loosely, non-word characters.

Michael Kay
http://www.saxonica.com/



--~------------------------------------------------------------------
XSL-List info and archive:  http://www.mulberrytech.com/xsl/xsl-list
To unsubscribe, go to: http://lists.mulberrytech.com/xsl-list/
or e-mail: <mailto:xsl-list-unsubscribe(_at_)lists(_dot_)mulberrytech(_dot_)com>
--~--

<Prev in Thread] Current Thread [Next in Thread>