7.3.66. tokenizer_list¶
7.3.66.1. Summary¶
tokenizer_list command lists tokenizers in a database.
7.3.66.3. Usage¶
Here is a simple example.
Execution example:
tokenizer_list
# [
#   [
#     0,
#     1337566253.89858,
#     0.000355720520019531
#   ],
#   [
#     {
#       "name": "TokenDelimit"
#     },
#     {
#       "name": "TokenUnigram"
#     },
#     {
#       "name": "TokenBigram"
#     },
#     {
#       "name": "TokenTrigram"
#     },
#     {
#       "name": "TokenBigramSplitSymbol"
#     },
#     {
#       "name": "TokenBigramSplitSymbolAlpha"
#     },
#     {
#       "name": "TokenBigramSplitSymbolAlphaDigit"
#     },
#     {
#       "name": "TokenBigramIgnoreBlank"
#     },
#     {
#       "name": "TokenBigramIgnoreBlankSplitSymbol"
#     },
#     {
#       "name": "TokenBigramIgnoreBlankSplitSymbolAlpha"
#     },
#     {
#       "name": "TokenBigramIgnoreBlankSplitSymbolAlphaDigit"
#     },
#     {
#       "name": "TokenDelimitNull"
#     },
#     {
#       "name": "TokenRegexp"
#     }
#   ]
# ]
It returns tokenizers in a database.
7.3.66.4. Return value¶
tokenizer_list command returns tokenizers. Each tokenizers has an attribute
that contains the name. The attribute will be increased in the feature:
[HEADER, tokenizers]
HEADER
See Output format about
HEADER.
tokenizers
tokenizersis an array of tokenizer. Tokenizer is an object that has the following attributes.
Name
Description
nameTokenizer name.