llama3_tokenizer¶

torchtune.models.llama3.llama3_tokenizer(path: str, special_tokens_path: Optional[str] = None) → Llama3Tokenizer[source]¶

Tokenizer for Llama3.

Parameters:

path (str) – path to the tokenizer
special_tokens_path (Optional[str]) – Path to tokenizer.json from Hugging Face model files that contains all registered special tokens, or a local json file structured similarly. Default is None to use the canonical Llama3 special tokens.

Returns:

Instantiation of the Llama3 tokenizer

Return type:

Llama3Tokenizer

Docs