ModelTokenizer¶
- class torchtune.modules.tokenizers.ModelTokenizer(*args, **kwargs)[source]¶
Abstract tokenizer that implements model-specific special token logic in the
tokenize_messages
method. SeeLlama3Tokenizer
for an example implementation of this protocol.