Version: 3.x

rasa.nlu.tokenizers._tokenizer

Token Objects

class Token()

 | set(prop: Text, info: Any) -> None

Set property value.

 | get(prop: Text, default: Optional[Any] = None) -> Any

Returns token value.

 | fingerprint() -> Text

Returns a stable hash for this Token.

class Tokenizer(Component)

Base class for tokenizers.

 | __init__(component_config: Dict[Text, Any] = None) -> None

Construct a new tokenizer using the WhitespaceTokenizer framework.

 | tokenize(message: Message, attribute: Text) -> List[Token]

Tokenizes the text of the provided attribute of the incoming message.

 | train(training_data: TrainingData, config: Optional[RasaNLUModelConfig] = None, **kwargs: Any, ,) -> None

Tokenize all training data.

 | process(message: Message, **kwargs: Any) -> None

Tokenize the incoming message.