llama2_tokenizer

torchtune.models.llama2.llama2_tokenizer(path: str, max_seq_len: Optional[int] = None, prompt_template: Optional[Union[str, Dict[Literal['system', 'user', 'assistant', 'ipython'], Tuple[str, str]]]] = 'torchtune.models.llama2.Llama2ChatTemplate') → Llama2Tokenizer

Tokenizer for Llama2.

Parameters:

• path (str) – path to the tokenizer

• max_seq_len (Optional[int]) – maximum sequence length for tokenizing a single list of messages, after which the input will be truncated. Default is None.

• prompt_template (Optional[_TemplateType]) – optional specified prompt template. If a string, it is assumed to be the dotpath of a PromptTemplateInterface class. If a dictionary, it is assumed to be a custom prompt template mapping role to the prepend/append tags. Default is Llama2ChatTemplate.

Returns:

Instantiation of the Llama2 tokenizer

Return type:

Llama2Tokenizer