parse_hf_tokenizer_json

torchtune.modules.tokenizers.parse_hf_tokenizer_json(tokenizer_json_path: str) → Dict[str, int]

Parse the tokenizer.json file from a Hugging Face model to extract the special token str to id mapping.

Parameters:

tokenizer_json_path (str) – Path to the tokenizer.json file.

Returns:

The special token str to id mapping.

Return type:

Dict[str, int]