Version: 3.x

rasa.nlu.tokenizers.spacy_tokenizer

SpacyTokenizer Objects

@DefaultV1Recipe.register(
    DefaultV1Recipe.ComponentType.MESSAGE_TOKENIZER, is_trainable=False
)
class SpacyTokenizer(Tokenizer)

Tokenizer that uses SpaCy.

@classmethod
def required_components(cls) -> List[Type]

Components that should be included in the pipeline before this component.

@staticmethod
def get_default_config() -> Dict[Text, Any]

The component's default config (see parent class for full docstring).

@staticmethod
def required_packages() -> List[Text]

Any extra python dependencies required for this component to run.

def tokenize(message: Message, attribute: Text) -> List[Token]

Tokenizes the text of the provided attribute of the incoming message.