BaseTokenizer¶
- class torchtune.modules.tokenizers.BaseTokenizer(*args, **kwargs)[source]¶
Abstract token encoding model that implements
encode
anddecode
methods. SeeSentencePieceBaseTokenizer
andTikTokenBaseTokenizer
for example implementations of this protocol.