wikitext_dataset¶
- torchtune.datasets.wikitext_dataset(tokenizer: ModelTokenizer, source: str = 'wikitext', subset: str = 'wikitext-103-raw-v1', max_seq_len: Optional[int] = None, **load_dataset_kwargs: Dict[str, Any]) TextCompletionDataset [source]¶
Support for family of datasets similar to wikitext, an unstructured text corpus consisting of articles from Wikipedia.
- Parameters:
tokenizer (ModelTokenizer) – Tokenizer used by the model that implements the
tokenize_messages
method.source (str) – path string of dataset, anything supported by Hugging Face’s
load_dataset
(https://huggingface.co/docs/datasets/en/package_reference/loading_methods#datasets.load_dataset.path)subset (str) – name of subset of data to use, see the wikitext page for available subsets.
max_seq_len (Optional[int]) – Maximum number of tokens in the returned input and label token id lists. Default is None, disabling truncation. We recommend setting this to the highest you can fit in memory and is supported by the model. For example, llama2-7B supports up to 4096 for sequence length.
**load_dataset_kwargs (Dict[str, Any]) – additional keyword arguments to pass to
load_dataset
.
- Returns:
the configured TextCompletionDataset
- Return type: