DDC Query Language Documentation

Top-Level Rule(s)

Context Query Rules

Query Filter Rules

Count-Query Rules

List-like Constituents

Preterminals and Aliases

Common Definitions

The underlying scanner makes use of several regular expression 'macros' using the notation {MACRO}. The following macros are defined for the underlying scanner:

 /* whitespace */          
ws              [ \t\n\r\f\v]

 /* integer boundary characters */
int_boundary     [ \t\n\r\f\v\0&|!?^%,:;#*=~(){}<>\[\]\\/\'\".$@_+-]

 /* bareword symbols */
symbol_cfirst   [^ \t\n\r\f\v\0&|!?^%,:;#*=~(){}<>\[\]\\/\'\".$@]
symbol_crest    [^ \t\n\r\f\v\0&|!?^%,:;#*=~(){}<>\[\]\\/\'\"]
symbol_cescape  (\\.)
symbol_text     ({symbol_cescape}|{symbol_cfirst})({symbol_cescape}|{symbol_crest})*

 /* subcorpus symbols (allowing bare '/' and '*') */
corpus_cfirst	[^ \t\n\r\f\v\0&|!?^%,:;#=~(){}<>\[\]\\\'\"$@]
corpus_crest	[^ \t\n\r\f\v\0&|!?^%,:;#=~(){}<>\[\]\\\'\"]
corpus_text	({symbol_cescape}|{corpus_cfirst})({symbol_cescape}|{corpus_crest})*

 /* bareword index names (underscore and digits are ok, but no '.', '-', or '+') */
index_char      [^ \t\n\r\f\v\0&|!?^%,:;#*=~(){}<>\[\]\\/\'\".$@+-]
index_name      ({index_char}|{symbol_cescape})+

 /* single-quoted symbols */
sq_text         ([^\']|{symbol_cescape})*

 /* regular expressions */
regex_text      ((\\.)|[^\\/])*
regex_modifier  [dgimsx]

 /* lexer comments */
comment_text    (\\.|[^\]])*

Comments

 "#:"([^\n]*)                       /* line comment, ignored by scanner */               
 "#["{comment_text}"]"              /* block comment, ignored by scanner */

The scanner supports line comments introduced by "#:" and extending to the end of the current line, as well as block comments of the form "#[...]" from non-exclusive start-states (i.e. pretty much everywhere except within regular expressions, date-strings, or expansion chains). Such comments will be ignored by the scanner and thus are not recoverable from a parsed query object. If you need to pass comments through the query parser, use the #COMMENT keyword instead. Since v2.1.16.

Query Keywords

(?i:near)
(?i:"!="|"&!="|"&="{ws}*!|!with|with(out|{ws}*!))
(?i:"|="|withor|orwith|wor)
(?i:"&="|with)
(?i:count)
(?i:(file|doc)?id)
(?i:(file|doc)_?(name)?)
(?i:date)
(?i:#(comment|cmt))
(?i:#(co?n?te?xt?|n))
(?i:#(with)?in)
(?i:#(sep(arate)?|nojoin)(_hits)?)
(?i:#(nosep(arate)?|join)(_hits)?)
(?i:#has(_field)?)
(?i:#file(_?)names)
(?i:#debug_rank)
(?i:#(greater|de?sc)(_by)?_rank)
(?i:#(less|asc)(_by)?_rank)     
(?i:#(greater|de?sc)(_by)?_date)
(?i:#(less|asc)(_by)?_date)     
(?i:#(is_|has_)?date)           
(?i:#(greater|de?sc)(_by)?_size)
(?i:#(less|asc)(_by)?_size)     
(?i:#(is_|has_)?size)           
(?i:#((less|asc)(_by)?_)?left)
(?i:#((greater|de?sc)(_by)?_)left)
(?i:#((less|asc)(_by)?_)?right)
(?i:#((greater|de?sc)(_by)?_)right)
(?i:#((less|asc)(_by)?_)?mid(dle)?)
(?i:#((greater|de?sc)(_by)?_)mid(dle)?)
(?i:#(less|asc)(_by)?)
(?i:#(greater|de?sc)(_by)?)
(?i:#rand(om)?)
(?i:#by)
(?i:#samp(le)?)
(?i:#clim(it)?)
(?i:#((less|asc)(_by)?_key))
(?i:#((greater|de?sc)(_by)?_key))
(?i:#((less|asc)(_by)?_(count|val(ue)?)))
(?i:#((greater|de?sc)(_by)?_(count|val(ue)?)))
(?i:#prune(_less|_asc)?(_by)?)
(?i:#prune(_greater|_de?sc)(_by)?)

The scanner supports the various reserved words used in the query grammar. Query keywords are matched case-insensitively; e.g. "NEAR", "near", "Near", and "nEaR" are all valid instantiations of the query keyword NEAR. Query keywords must be escaped if they occur anywhere in a literal symbol, e.g. by enclosing the symbol in single quotes.

Match-IDs

"="[0-9]+                           /* user-specified match-id (valid range: 1-255) */

The parser supports user-specified match-IDs for use e.g. by context-dependent sort operators. Match-IDs are decimal integers between 1 and 255, introduced by a leading '=' (equals sign). If no match-IDs were specified for a query, every matched token of that query is treated as if it had been assigned a match-ID of 1. Otherwise, if at least one match-ID was explicitly assigned, any matched tokens without explicit match-IDs are assigned a match-ID of 255.

Regular Expressions

 "/"{regex_text}"/"{regex_modifier}*                    /* slash-quoted regex, with option modifiers */
 "!/"{regex_text}"/"{regex_modifier}*                   /* negated regex (complement) */
 "s/"{regex_text}"/"{regex_text}"/"{regex_modifier}*    /* search-and-replace regex (for count-by) */

The scanner supports regular expressions enclosed by forward slashes, with an optional suffix modifer string. If the expression is immediately preceded by an exclamation point ("!"), it is treated as a negated condition, and evaluates to the complement of the set of all index items matching the (non-negated) regular expression. DDC uses PCRE as its underlying regular expression engine, which should support the full Perl regular expression syntax. In particular, C-style escapes ("\t", "\n", "\0", etc.) and Unicode character escapes ("\x{17f}", "\x{364}") may occur within regular expressions. Literal slashes as part of a regular expression must be escaped with a backslash ("\/").

The regular expression modifier "/g" (mnemonic: 'match Globally') has a special meaning when used in DDC regex match expressions: if this modifier is set, only those index items are retrieved which match the regular expression in the entirety; otherwise all index items for which any substring matches the regular expression will be retrieved. Internally this behavior is implemented by implicitly inserting beginning- and end-of-string anchors into the regular expression, thus /REGEX/g is a equivalent to /^(?:REGEX)$/. When used in search-and-replace operations s/SEARCH/REPLACE/g, the "/g" modifier causes all occurrences of the SEARCH pattern to be replaced by the REPLACE string.

As of DDC version 2.0.47, the non-standard regular expression modifier "/d" (mnemonic: 'ignore Diacritics') causes both the regular expression itself and the index items against which it is matched to be converted from the source encoding (currently only UTF-8 is supported) to 7-bit ASCII before matching. Conversion is performed internally using iconv with the target encoding "ASCII//TRANSLIT"; which should map e.g. the Unicode codepoint U+00F6 ("ö", LATIN SMALL LETTER O WITH DIAERESIS) to a bare ASCII "o" (U+006F, LATIN SMALL LETTER O). Results vary depending on the locale settings (LC_CTYPE) for the running DDC server process.

Punctuation Operators and Special Characters

"&&"                                /* clause-level conjunction operator */
"||"                                /* clause-level disjunction operator */
"#="                                /* exact-distance hint in phrase queries */
"#<"                                /* max-distance hint for phrase queries */
"#>"                                /* min-distance hint for phrase queries */
"$."                                /* anchor pseudo-index (position-index in "break collection" segment) */
[!.,:;@%^#=/~]                      /* single-character punctuation operators */
[\[\]{}()<>]                        /* parentheses */
[\"]                                /* double quotes (for pharse queries) */
":{"                                /* thesaurus-query opener */
"@{"                                /* literal-set opener */
"*{"                                /* suffix- or infix-set opener */
"}*"                                /* prefix- or infix-set closer */
"*"                                 /* universal match wildcard */

The scanner supports the various punctuation operators used in the query grammar. Punctuation operators should be escaped if they occur anywhere in a literal symbol, e.g. by enclosing the symbol in single quotes.

Truncated Symbols

"*"\'{sq_text}\'"*"                 /* dual-truncated quoted string (infix symbol) */
   \'{sq_text}\'"*"                 /* right-truncated quoted string (prefix symbol) */
"*"\'{sq_text}\'                    /* left-truncated quoted string (suffix symbol) */

"*"{symbol_text}"*"                 /* dual-truncated bare string (infix symbol) */
   {symbol_text}"*"                 /* right-truncated bare string (prefix symbol) */
"*"{symbol_text}                    /* left-truncated bare string (suffix symbol) */

Truncated symbols (for prefix-, suffix-, and infix-queries) are supported directly by the scanner, in order to distinguish these from literal symbols containing an asterisk character ("*"). Literal asterisks must be escaped if they occur anywhere in a literal symbol, e.g. by enclosing the symbol in single quotes. Note that despite this restriction, the asterisk does NOT function as a general "wildcard character" for DDC queries. If you need to match multiple substrings of a single index item, use a regular expression instead, e.g. "/^foo.*bar.*baz$/" for a wildcard condition "foo*bar*baz".

Integers

\'[\+\-]?[0-9]+\'                   /* single-quoted decimal integer */
  [\+\-]?[0-9]+/{int_boundary}      /* bareword decimal integer */

The scanner supports integers in decimal notation, optionally enclosed by single quotes and/or prefixed with a sign character. It may be necessary to introduce whitespace or quotes in order to separate an integer token from surrounding symbols.

Dates

\'[0-9\-]+\'                        /* single-quoted date-like string */
  [0-9]{4,}[0-9\-]+/{int_boundary}  /* bareword date-like string */

The scanner supports date-like tokens in a superset of the ISO 8601 format (YYYY-MM-DD), optionally enclosed by single quotes.

Index Names

"$"{index_name}                     /* dollar-prefixed index name bareword */
"$'"{sq_text}"'"                    /* quoted index name */
"$"                                 /* bare dollar symbol (default index) */

As of v2.1.1, the query scanner treats index names specially. No whitespace is allowed between the '$' prefix and the index name itself, and valid bareword index names are those by the {index_name} definition. For maximum compatibility, index names should containing only ASCII letters, decimal digits, and/or underscores. Index names not fulfilling these criteria can nonetheless be specified using backslash-escapes, or single quotes.

Symbols and Barewords

\'{sq_text}\'                       /* single-quoted symbol */
{symbol_text}                       /* bareword symbol */

Symbols (strings) are the basic building blocks of the DDC query language. The scanner supports "bareword" symbols (e.g. my_symbol) consisting of only alphanumeric and non-ASCII characters, underscores, and backslash-escapes, as well as single-quoted strings (e.g. 'symbol with whitespace'), in which only embedded quotes must be escaped.

Term Expander Pipelines

"|"{ws}*"-"                         /* alias for default term expander pipeline (index-dependent) */
"|"{ws}*\'{sq_text}\'               /* quoted term expander label */
"|"{ws}*{symbol_text}               /* bare term expander label */

The scanner supports special rules for user specification of term expansion pipelines used by the qw_bareword and qw_set_infl query types. Each link in a term expansion pipeline is prefixed by the special symbol "|" (vertical bar or 'pipe' symbol), followed by optional whitespace and the label of a named expander to apply as a symbol. In order for the query to be evaluated, each expander label in a term expansion pipeline must be defined for the corpus to be queried. The expanders "id", "null", "case", and "infl" should always be defined. The default expanders may be overridden and/or additional expanders defined in a corpus .opt file. The special label "-" (hyphen) may be used as an alias for the default expander for the index to be queried, whose label is identical to the long index name as declared in the corpus .opt file. By default, only the "Token" index has a pre-defined expansion pipeline, defined as 'infl' (built-in language-dependent lemmatization and re-inflection) by default. If the CaseInsensitive flag is set in the corpus .opt file, the default Token pipeline is 'infl|case' (inflection followed by built-in letter-case twiddling). The default expansion pipeline for any other other index is empty, i.e. equivalent to the identity expander "id" a.k.a. "null". Literal pipe symbols ("|") must be escaped if they occur anywhere in a literal symbol, e.g. by enclosing the symbol in single quotes.

Subcorpus Paths

{corpus_text}                       /* bareword subcorpus path (including '/', '*', and '**') */
\'{sq_text}\'                       /* single-quoted subcorpus path */

A branch-server query (see ddc_cfg(5)) may be restricted to a subset of available sub*corpora by means of the : directive (:PATH1,PATH2,...,PATHn). Each subcorpus PATH in the list is parsed syntactically as an atomic symbol (allowing un-escaped "/", "*", and "**" for convenience), but is otherwise ignored by physical leaf servers when queried directly.

Prior to v2.2.0, each PATH in a branch-server query had to be the literal name of an immediate daughter node (see ddc_cfg(5)), and only those daughters explicitly specified were queried. As of v2.2.0, branch servers allow slash-separated sub*corpus paths in PATHs as well as (limited) wildcard strings * indicating all immediate daughters of the corresonding node. As of v2.2.8, branch and leaf servers also allow path wildcard strings ** indicating any direct or indirect descendant of the current node:

Paths	Description
:	all descendants (empty list: default)
:*	all descendants (explicit wildcard)
:foo,bar	immediate daughters foo and bar only
:foo/bar	granddaughter bar, which is a child of daughter foo
:*/bar	all granddaughters named bar
:*/bar,*/*/bar	all granddaughters or great-granddaughters named bar
:**/baz	any node or descendant named baz
:**/baz/**/bonk	any descendant named bonk which is descended from a node named baz

For best results, individual server node labels themselves should not contain any literal slashes (/), commas (,), or colons (:).

Other Terminals

{ws}+                               /* whitespace: skip */
\0                                  /* null byte: skip */
.                                   /* any other character (byte) is its own token */ }

Whitespace and NUL bytes are ignored. Any other input byte is returned as a symbol of length 1.

Compilation Errors

Not all queries which are grammatical according to the query grammar given above are in fact admissible DDC queries. In particular, pure negative conditions such as !foo, !foo && !bar, or !(foo || bar) will parse successfully but fail to compile. In such cases, DDC should return an error code of errParseError and print an informative diagnostic message to stderr and/or the current log file.

Wildcard Queries

The wildcard query operator (*) matches every token in the corpus, and its use in most cases can be expected to substantially degrade query evaluation performance. There are some exceptions to this rule, however (for ddc v2.0.36 and above), which are listed below. Please note that in some earlier versions of ddc, the optimizations described here were not implemented, so it is always a good rule of thumb to avoid wildcard queries if you can.

count(*)

count(*) queries are optimized and can be expected to run quite fast, provided that only constant or document-level count-keys are used (i.e. no $INDEX items appear in the query's #by[...] clause.

near(...,*,...)

near() queries are optimized to take advantage of universal wildcard sub-queries, provided that at least one non-wildcard sub-query is used as well, i.e queries of the form near(A,*,N), near(*,B,N), near(A,B,*,N), near(A,*,C,N), near(*,B,C,N), near(A,*,*,N), near(*,B,*,N), and near(*,*,C,N) can be expected to run efficiently.

"... * ..."

Like near() queries, phrase queries using the double-quote ("...") operator are optimized to take advantage of universal wildcard sub-queries if at least one non-wildcard sequence item is included in the phrase, e.g. phrases of the forms "A *", "* B", "A * C", "* B * D", etc. can be expected to run efficiently.

with *

with queries containing a universal wildcard are implicitly simplified to the non-wildcard sub-query, thus A with * and * with A are both evaluated as A, without evaluating the wildcard itself.

DTA Index Fields

The DTA corpus .opt file defines the following index fields:

Token w

unicruft Latin-1 approximation of original token text (actually encoded in the Latin-1 subset of UTF-8). This is the default field searched e.g. by bareword SYMBOL queries. As of August, 2011, words containing non-latin-extended characters are not transliterated by default, and should appear (and be searchable) by their original UTF-8 forms without resorting to the "Utf8" index field (with regard to which see below).

Utf8 u

Original UTF-8 text of the source token. Formerly indexed as Token2 w2.

CanonicalToken v

DTA::CAB-normalized wordform for the source token.

Pos p

Part-of-speech assigned to the source token by moot via DTA::CAB. Uses the STTS tag-set.

Lemma l

Lemma assigned to the source token by DTA::CAB. Uses the TAGH morphology via DTA::CAB if available.

Coord coord

Bounding box(es) for the source token. Not really meaningful for querying, but useful for image highlighting.

Page page

Page (scan) number. Formerly used the long index name "age", which appears to have been nothing more than a self-perpetuating typographical error.

Rend r

Typographical ("rendition") attributes for the source token as as '|'-sparated set e.g. "-" (nothing of note), "|aq|" (antiqua), "|b|" (bold?), "|g|" (upper-case?), "|b|g|" (bold? + upper-case?), ...

Context con

Text-structurual features of the source token as a lexicographically sorted '|'-separated list of unique structural properties, or "-" if the token has no special text-structural properties at all. Supported properties inherited from XML ancestor elements are: text, front, body, back, head, left, foot, end, argument, hi, cit, fw, lg, stage, speaker, formula, and table. Additionally, the properties note_foot, note_left, note_right and note_other are used for tokens with an XML note ancestor.

XPath xpath

XPath to the deepest element node containing (the first character of) the current token, or "-" if no such XPath was available during indexing. Note that any prefix "/TEI/text" or "/TEI" is implicitly trimmed from the indexed XPath strings, since all ddc-indexed tokens must occur as descendants of a /TEI/text element.

Line lb

Line number of the current token, relative to the current page as indexed by the page field. Defined as the number of opening (and possibly closing) "lb" preceding the first character of the current token on the current page; e.g. 0 (zero) if no "lb" elements precede the first character of the current token on the current page. Precedence and page-association are computed in terms of TEI document order; e.g. if chapter headings contain lb elements, these will count towards the line number assigned.

DTA Term Expanders

The DTA corpus .opt file defines the following term expanders:

id

Identity expander (no-op)

null

Identity expander (no-op)

case

Case-twiddling expander using current locale settings of the running server process, which should have an LC_CTYPE character set of "UTF-8" or equivalent for correct results. Not used by default.

tolower

lc

Case-twiddling expander which forces input to all lower-case. Same caveats as for the case expander. Not used by default.

toupper

uc

Case-twiddling expander which forces input to all upper-case. Same caveats as for the case expander. Not used by default.

morphy

Built-in DDC lemmatization and re-inflection (morphy-based)

infl

tagh

TAGH-based lemmatization and re-inflection via dedicated DTA::CAB HTTP server at http://194.95.188.28:9098 .

cab

Expansion by an external DTA::CAB HTTP server at http://194.95.188.36:9099 using the "expand" analyzer method (phonetic and rewrite equivalents).

germanet, gn-*

Expansion family using the GermaNet thesaurus to return a set of lemmata related to its argument lemma(ta). Supported relations:

gn-syn  /* synonyms (closure) */
gn-syn1 /* synonyms (depth=1) */
gn-syn2 /* synonyms (depth=2) */

gn-isa  /* hyperonyms/superclasses (closure); alias=gn-sup  */
gn-isa1 /* hyperonyms/superclasses (depth=1); alias=gn-sup1 */
gn-isa2 /* hyperonyms/superclasses (depth=2); alias=gn-sup2 */

gn-asi  /* hyponyms/subclasses (closure); aliases=gn-sub,germanet  */
gn-asi1 /* hyponyms/subclasses (depth=1); alias=gn-sub1 */
gn-asi2 /* hyponyms/subclasses (depth=2); alias=gn-sub2 */

openthesaurus, ot-*

Expansion family using OpenThesaurus to return a set of lemmata related to its argument lemma(ta). Supported relations:

ot-syn  /* synonyms (closure) */
ot-syn1 /* synonyms (depth=1) */
ot-syn2 /* synonyms (depth=2) */

ot-isa  /* hyperonyms/superclasses (closure); alias=ot-sup  */
ot-isa1 /* hyperonyms/superclasses (depth=1); alias=ot-sup1 */
ot-isa2 /* hyperonyms/superclasses (depth=2); alias=ot-sup2 */

ot-asi  /* hyponyms/subclasses (closure); aliases=ot-sub,openthesaurus  */
ot-asi1 /* hyponyms/subclasses (depth=1); alias=ot-sub1 */
ot-asi2 /* hyponyms/subclasses (depth=2); alias=ot-sub2 */

pho

Expansion by the external DTA::CAB HTTP server at using only the "eqpho" analyzer method (phonetic equivalents only).

rw

Expansion by the external DTA::CAB HTTP server at using only the "eqrw" analyzer method (rewrite equivalents only). Should be more precise than the cab or pho expanders, but with poorer recall.

eqlemma

TAGH-based best-lemma match using pre-compiled index via via dedicated DTA::CAB HTTP server at http://194.95.188.36:9099.

Token

Default expansion pipeline for queries of the (transliterated) 'Token' index, defined as eqlemma.

Utf8

Default expansion pipeline for queries of the (raw UTF-8) 'Utf8' index, defined as eqlemma. Note that due to limitations and configuration issues with respect to both the underlying CAB server and the built-in DDC morphological expansion module, target strings containing extinct characters ("ſ", "aͤ", etc.) will not be found by this expander.

CanonicalToken

Default expansion pipeline for queries of the CAB-normalized 'CanonicalToken' index, defined as infl.

Pos

Default expansion pipeline for queries of the 'Pos' (part-of-speech tag) index, defined as toupper.

Lemma

Default expansion pipeline for queries of the 'Lemma', defined as case.

Basic Examples

sein

All expanded variants of "sein" ("sein", "ist", "war", ..., "seyn", "seyd", ..., "seine", "ihre", ...)

@sein

Only the literal (latin-1 approximation) string "sein"

$u=@ſein

Only the literal (original UTF-8) string "ſein"

Wach*

Any string with (latin-1 approximation) prefix "Wach"

*stube

Any string with (latin-1 approximation) suffix "stube"

*achs*

Any string with (latin-1 approximation) substring "achs"

$u=/ſtube$/

Any string with (original UTF-8) suffix "ſtube" (via regex)

$u=/ſtube$/i

Any string with (original UTF-8) suffix "ſtube" (via regex, case-insensitve)

Term Expansion Examples

Tür

All sentences containing any variant of the word 'Tür'

$Token=Tür

... same condition, with explicit index

$Token=Tür |Token

... same condition, with explicit index and expansion pipeline

Tür |Token

... same condition, with explicit expansion pipeline only

Tür |-

... same condition, with "-" expansion pipeline alias

Tür |infl|cab

... same condition, with alternative user expansion pipeline

Tür |infl

All (modern) inflectional variants of the word "Tür" (no CAB expansion)

Tür |cab

All graphematic variants of the form "Tür" (no lemmatization or re-inflection)

Tür |case

All letter-case variants of the form "Tür"

Tür |infl|case

All letter-case variants of any modern inflectional variant of the word "Tür"

Tür |pho

All phonetic variants of the form "Tür"

Tür |rw

All rewrite variants of the form "Tür"

Tür |infl|rw

All rewrite variants of any modern inflectional variant of the word "Tür"

Multi-Token Examples

Verein && Club

All sentences containing both 'Verein' and 'Club' (variants)

Verein || Club

All sentences containing either 'Verein' or 'Club' (variants)

{Verein,Club}

... the same condition, in set notation

@Verein || @Club

All sentences containing either 'Verein' or 'Club' (literals)

@{Verein,Club}

... the same condition, in set notation

*{verein,club}

All strings ending in either "verein" or "club"

/(?:\Qverein\E|\Qclub\E)$/

... the same condition, in regex notation

/(?:\Qverein\E|\Qclub\E)$/i

... case-insensitive (includes e.g. "Verein" and "Club")

"ein Verein"

All variants of the substring "ein Verein"

"@ein @Verein"

The exact substring "ein Verein"

"ein #1 Verein"

All variants of the subsequence "ein Verein" with at most 1 intervening word

"ein #1 {Verein,Club}"

All variants of one of the substrings "ein Verein" or "ein Club" with at most 1 intervening word

Query Filter Examples

Max

All sentences containing a variant of "Max", unsorted

Max #has_field[author,/Busch/]

... only those documents by an author matching the regex /Busch/

Max #has[author,/busch/i]

... same condition, using new #has alias for #has_field and case-insensitive regex

Max #less_by[author]

... sorted in ascending order by author string

Max #greater_by_rank

... sorted in descending order by putative relevance

Max #less_by_size

... sorted in ascending order by hit (sentence) length

Max #less_by_size[3,5]

... sorted in ascending order by hit length, where hit length is between 3 and 4 tokens.

Max #less_by_date

... sorted in ascending order by hit date

Max #date[1865]

... only those hits from 1865