alpaca_dataset¶
- torchtune.datasets.alpaca_dataset(tokenizer: ModelTokenizer, *, source: str = 'tatsu-lab/alpaca', train_on_input: bool = True, max_seq_len: int = 512, packed: bool = False) InstructDataset [source]¶
Support for family of Alpaca-style datasets from Hugging Face Datasets using the data input format and prompt template from the original alpaca codebase, where
instruction
,input
, andoutput
are fields from the dataset.Masking of the prompt during training is controlled by the
train_on_input
flag, which is set toTrue
by default - Iftrain_on_input
is True, the prompt is used during training and contributes to the loss. - Iftrain_on_input
is False, the prompt is masked out (tokens replaced with -100)- Parameters:
tokenizer (ModelTokenizer) – Tokenizer used by the model that implements the
tokenize_messages
method.source (str) – path string of dataset, anything supported by Hugging Face’s load_dataset.
train_on_input (bool) – Whether the model is trained on the prompt or not. Default is True.
max_seq_len (int) – Maximum number of tokens in the returned input and label token id lists. Default is 512, but we recommend setting this to the highest you can fit in memory and is supported by the model. For example, llama2-7B supports up to 4096 for sequence length.
packed (bool) – Whether or not to pack the dataset to
max_seq_len
prior to training. Default is False.
- Returns:
dataset configured with source data and template
- Return type:
Example
>>> alpaca_ds = alpaca_dataset(tokenizer=tokenizer) >>> for batch in Dataloader(alpaca_ds, batch_size=8): >>> print(f"Batch size: {len(batch)}") >>> Batch size: 8