XMLTOOLS(7)

NAME
xmltools - XML manipulation tools

DESCRIPTION
xmltools is a collection of command-line utilities for XML stream manipulation. It provides fast and memory-efficient commands for editing tasks similar to what grep(1) and sed(1) can do on text files.

The remainder of this document is outlined as follows:

1.  CONCEPTS

2.  PATTERN LANGUAGE OVERVIEW
    Types and conversions
    Matching

3.  EXPRESSIONS
    Constants
    Functions
    Usual operators
    Regex operators
    Navigational operators
    Sequence operators
    Union operator

4.  ABBREVIATED SYNTAX

5.  IMPLEMENTATION NOTES

6.  EXAMPLES

7.  COMPATIBILITY

8.  SEE ALSO

9.  STANDARDS

10. AUTHORS

CONCEPTS
All utilities understand a common selection and command language similar to the XML Path Language “(XPath”).

The primary use of the pattern language is to select (match) parts of the XML tree, to which further actions can be bound. The language operates on XML intervals, an intuitive notion with good properties for editing tasks. Given two sibling elements A and B (i.e. A and B have the same parent), the XML interval from A to B is simply defined as all nodes from A to B inclusive, as well as their descendants. For example, in ‘<z><a><x/></a><b/><c/><d/></z>’ the interval from ‘a’ to ‘c’ is ‘<a><x/></a><b/><c/>’.

Matching an XML interval operates, predictably, in two steps, by first matching the lower bound, and then finding the upper bound, given the lower bound.

Matching is carried out according to a navigational metaphor, borrowed from XPath. This approach can be seen as follows, in the context of xmltools. The program maintains a virtual cursor on one element of the XML tree (the current or context element), and can move this cursor in one of two directions (axes): to one of the child elements or to the next sibling of the current element. It makes this choice based on information provided by the current element, as well as its past choices and operations (hence, it is some kind of push-down automaton navigating the XML tree).

Let's call the set of elements needed to determine whether the current element satisfies some pattern the look-ahead context. Minimizing the size of the look-ahead context is the key to writing fast and memory-friendly patterns. The pattern language description below gives indications pertaining to the impact on the look-ahead context size of each pattern construct.

PATTERN LANGUAGE OVERVIEW
This section gives an informal description of the pattern language, as understood by all the utilities. Please refer to external/bsd/xmltools/regxml/pattern.y for the exact grammar.

The pattern language is a simple expression language, comprising unary and binary operators, as well as integer and string literals. You can group subexpressions in parentheses, to alter the precedence of the various operators.

  Types and conversions
Subpatterns can yield one of four types: the null value, an integer, a character string, or an XML element. Moreover, XML elements can take the special value FAIL. Converting between types is done as follows:

•   Conversion from null to integer yields 0. Conversion from null to string yields the empty string.

•   Conversions to null always yield null.

•   Conversion from integer to string yields the decimal string representation of that integer as computed by sprintf(3).

•   Conversion from string to integer yields the decimal integer value of the string as computed by strtol(3).

•   Conversion from null to XML element yields FAIL.

•   Conversion from integer to XML element yields the current element if the integer is non-zero, FAIL otherwise.

•   Conversion from string to XML element yields the current element.

•   Operations that involve converting from XML element to integer or string are not supported at present time.

  Matching
In addition to the traditional expression syntax and semantics, the pattern language includes a matching construct, denoted by a subpattern in braces ‘({}’). This construct marks the beginning of a candidate XML interval. The current element is taken as the candidate lower bound. If it matches, the subpattern enclosed within the braces determines the extent (upper bound) of the interval. If it fails, the candidate interval is discarded.

EXPRESSIONS
The following table lists all available operators, in order of precedence, from most strongly binding to least strongly binding, with their associativity:
Operator Associativity
-------- -------------
* () {} none
/ < >? none
* div mod left to right
+ - left to right
[] left to right
= = left to right
= != ~ !~ left to right
/ // % %% right to left
and or right to left
 | right to left

The following sections describe each construct in more detail.

  Constants
String literals can be written with or without quotes, which may be simple or double. Inside quotes, the backslash character can be used to escape itself or embedded quotes.

String literals written without quotes end at the first special or blank character.

There is no separate syntax for numbers. String literals are converted to integers as needed according to the above conversion rules.

  Functions
Functions are called using the postfix ‘()’ operator, as in C. You cannot define new functions from within the pattern; you can only call predefined functions. The following table lists all built-in functions and their meanings:
Operator Meaning
-------- -------
name() Return name of the current element.
value() Return value of the current element.
relative-position() Return the position of the current element
relative to its siblings; the first child of its parent has a relative position of 1.
node() Return the name of the current element if it is a node,
or null.
text() Return the value of the current element
if it is a text element, or null.
comment() Return the value of the current element if it is a comment,
or null.
processing-instruction(t) Return the value of the current element
if it is a processing instruction for target t, or null.
attribute(a) Return the value of the attribute named a
of current element if any, or null.

As per the conversion rules described above, a null value is converted to FAIL, and a string is converted to the current node. Hence, simply specifying a function call such as ‘text()’ will indeed select all elements matching the correct element type, regardless of their names or values.

Please note that function names are parsed as keywords. Thus, they cannot be used as unquoted strings.

  Any predicate
In addition to type selection functions just described, the any predicate is available in the form of the zero-operand operator ‘*’. As its name suggests, it matches any element and is functionnally equivalent to:
    node()|text()|comment()|processing-instruction(...)
where the ellipsis stands for all possible processing instruction targets contained in the input.

  Usual operators
Arithmetic and relational operators behave as they do in C (except the division operator is spelled ‘div’ and the remainder is written ‘mod’), first converting their operands to integers, and producing an integer result.

Equality operators attempt a string comparison if both their arguments are strings or null; otherwise, both arguments are converted to integers and a numerical comparison is performed.

  Regex operators
Regular expression matching is performed by the ‘~’ and ‘!~’ operators, as in awk(1). Only extended regular expressions (ERE) are accepted. See re_format(7) for more information.

  Navigational operators
These are pattern-specific operators. Each binary navigational operator denotes a conditional change of the current element. If the left operand matches against the current node, then the right operand will be evaluated against the next node following the axis implied by the navigational operator. The following table lists all navigational operators and their axes:
Operator Axis
-------- ----
/ any child
// any descendant
% next sibling
%% any following sibling

For example, consider the following pattern:
    name()=foo / text()=bar

The program will attempt to evaluate the predicate ‘name()=foo’ first, against the current node. If it matches (that is, the name of the context element is ‘foo’), then the right part will be evaluated against all children of the current element, and if any of them matches (that is, they are text nodes containing ‘bar’), the result of the whole expression will be the matched child element.

  Anchor operators
The unary prefix ‘/, <’ and postfix ‘>’ operators are anchor operators. An anchor operator adds a structural condition to its operand. The operand is evaluated only if the structural predicate is met. The following table lists these conditions:
Operator Condition
-------- ---------
/ root element
< first child of a given parent
> last child of a given parent

  Sequence operators
The ‘and’ and ‘or’ operators carry a sequential semantic, as in Lisp or in Perl. The ‘and’ operator yields its first FAIL operand, or its last operand if none failed. The ‘or’ operator yields its first non-FAIL operand, or FAIL if all failed.

The ‘[]’ operator is another notation for ‘and’. Specifically, ‘a[b]’ is equivalent to ‘b and a’.

These operators have short-circuit evaluation. Non-evaluated operands do not qualify for further evaluation. This is especially important during the first matching step, when searching for the lower bound of an XML interval (see the union operator below).

  Option operator
The ‘’? postfix unary operator indicates that its operand may evaluate to the empty interval without failing. In itself, this operator has no effect. Its action shows in combination with parent operators.

If a navigational operator has a nullable operand, it accepts intervals that would be matched by the same expression with or without the optional part.

Other operators treat nullable operands as an interval of minimal length. Yielding such an interval is considered a success. For example, the following assertions are trivially true:

•   ‘a? and b’ is equivalent to ‘b’.

•   ‘a? or b’ is equivalent to ‘a?’.

Patterns containing nullable parts may be implicitly nullable themselves.

  Union operator
The ‘|’ operator evaluates both of its operands.

If the matcher is looking for the lower bound of an XML interval, then both branches are considered. This is in contrast with the ‘or’ operator described above.

If the matcher is looking for the upper bound of an XML interval, then the result of the union operator will be the element which makes for the longest interval.

ABBREVIATED SYNTAX
The pattern language includes a couple of abbreviated constructs, for oft-used bits of the syntax:

•   A terminal predicate that is not part of any matching construct ‘({}’) is taken as applying to both the lower and upper bounds of the interval, in effect matching subtrees. For the purpose of this clause, a terminal predicate is defined to be any subexpression that is at the end of a navigational chain made of union and navigational operators.

    This is more easily understood through an example. The following pattern matches all elements named ‘foo’ or ‘bar’ and their associated subtrees:
        name()=foo|name()=bar
    It is equivalent to the following explicit pattern:
        {name()=foo}|{name()=bar}

    In the same way, the following two patterns are equivalent:
        node()=foo/text()=bar
        node()=foo/{text()=bar}

•   Navigational operators accept constants as their operands. In that case, the constant do not stand for themselves, but instead act as equality predicates. If the constant is a non-zero integer, then it is matched against the relative position of the current element. Otherwise, the constant string is matched against the name of the current element, if it is a node.

    For example, the pattern ‘foo’ is equivalent to ‘node()=foo’ and the following two patterns are also equivalent:
        foo/bar|baz
        node()=foo/node()=bar|node()=baz

•   The ‘attribute’ function can be written ‘@’ without parentheses. For example, you can write ‘@foo’ instead of ‘attribute(foo)’.

IMPLEMENTATION NOTES
The tools are built on top of libregxml, which implements the actual pattern matching on XML streams. The library has been designed with broader goals in mind, but is not yet ready for uses outside of the xmltools project.

Among the issues to be fixed before libregxml can be opened to a more general public is its reliance on yacc(1) to parse the pattern grammar. At the moment, BSD yacc does not provide reentrant C bindings, hence multiple copies of libregxml cannot run in different threads at the same time.

EXAMPLES
Please refer to individual man pages of each tool.

COMPATIBILITY
The tools are designed to be independent of the parser library used as a backend. At the moment, only Expat is supported, but it is intended that in the future multiple backends will be available.

Depending on the XML parser library, some features of XML may or may not be supported, and some extensions to the language may be allowed. This section describes recommendations aiming at making an XML document compatible with all future combinations of tools and backends.

  Compatibility rules

1.  “not use doctype declarations for any purpose other than validation. In particular, do not rely on doctype declarations to provide default attributes and do not use internal entities for arbitrary substitutions; if you wanted to save typing, you wouldn't be using XML. Use of numeric and predefined entities is OK. Use of directly-encoded characters is preferred. Also, do not use CDATA sections.

2.  “not presuppose the XML processor has access to any resource beside your file: do not use external entities.

3.  If possible, use UTF-8. In any case, do not assume the XML processor is able to handle multiple character sets: all the documents and other data supplied to the program should use the same character set.

4.  “ assume that all whitespace-only text elements are ignored by the parser by default. This behavior may be switched off in a tool-specific manner, but beware of the implications of adding nodes to the input tree. Please note that this rule works well for most purposes, however, including processing of HTML ‘pre’ elements which only contain verbatim text.

The excluded features are not strictly needed for a tree representation, and in most cases can be worked around if really needed. For instance, an external utility could be used to process entities.

  Multiroot extension
The XML standard forbids multiple root nodes, but some tools may generate such forests (e.g. through extraction of individual nodes). It is intended that in the future, all tools will also accept this representation on input.

As an added compatibility rules, if a doctype or XML declaration is provided, then the document should only contain one root node. The rationale is that such declarations are meant to indicate conformance to the XML standard or some form of validation, and hence, in order to get anything out of those, you have to have only one root node anyway.

  Entities and CDATA
Use of CDATA and entities is discouraged even if you rely on a true XML parser such as Expat. There are quite a few issues with entities and CDATA sections, which all stem from the fact that they are alternative ways of writing something else (plain text).

This means that we have to convert them first to plain text, so as to be able to operate on them (e.g. comparing an entity with its translation to plain text should yield true). But then it is impossible to guess whether we should output the entity or the plain text equivalent. We could build complex heuristics to handle this, or we could (and this is what has been done) just provide minimal support.

Minimal support means that some parser backends might read and understand these constructs, but the output always uses the plain text versions.

SEE ALSO
re_format(7), regxml(3), xmlgrep(1)

STANDARDS
The Extensible Markup Language 1.0 Fifth Edition “(XML 1.0”) is defined by http://www.w3.org/TR/2008/REC-xml-20081126.

The pattern language borrows heavily from the XML Path Language 1.0 “(XPath 1.0”) as defined by http://www.w3.org/TR/1999/REC-xpath-19991116.

AUTHORS
Nhat Minh Lê <minh@NetBSD.org>