Shortcuts

llama3_tokenizer

torchtune.models.llama3.llama3_tokenizer(path: str, special_tokens_path: Optional[str] = None, max_seq_len: Optional[int] = None, prompt_template: Optional[Union[str, Dict[Literal['system', 'user', 'assistant', 'ipython'], Tuple[str, str]]]] = None) Llama3Tokenizer[source]

Tokenizer for Llama3.

Parameters:
  • path (str) – path to the tokenizer

  • special_tokens_path (Optional[str]) – Path to tokenizer.json from Hugging Face model files that contains all registered special tokens, or a local json file structured similarly. Default is None to use the canonical Llama3 special tokens.

  • max_seq_len (Optional[int]) – maximum sequence length for tokenizing a single list of messages, after which the input will be truncated. Default is None.

  • prompt_template (Optional[_TemplateType]) – optional specified prompt template. If a string, it is assumed to be the dotpath of a PromptTemplateInterface class. If a dictionary, it is assumed to be a custom prompt template mapping role to the prepend/append tags.

Returns:

Instantiation of the Llama3 tokenizer

Return type:

Llama3Tokenizer

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources