alpaca_dataset

torchtune.datasets.alpaca_dataset(tokenizer: Tokenizer, source: str = 'tatsu-lab/alpaca', train_on_input: bool = True, max_seq_len: int = 512) → InstructDataset

Support for family of Alpaca-style datasets from Hugging Face Datasets using the data input format and prompt template from the original alpaca codebase, where instruction, input, and output are fields from the dataset.

Masking of the prompt during training is controlled by the train_on_input flag, which is set to True by default - If train_on_input is True, the prompt is used during training and contributes to the loss. - If train_on_input is False, the prompt is masked out (tokens replaced with -100)

Parameters:

• tokenizer (Tokenizer) – Tokenizer used to encode data. Tokenize must implement an encode and decode method.

• source (str) – path string of dataset, anything supported by Hugging Face’s load_dataset.

• train_on_input (bool) – Whether the model is trained on the prompt or not. Default is True.

• max_seq_len (int) – Maximum number of tokens in the returned input and label token id lists. Default is 512, but we recommend setting this to the highest you can fit in memory and is supported by the model. For example, llama2-7B supports up to 4096 for sequence length.

Returns:

dataset configured with source data and template

Return type:

InstructDataset

Example

>>> alpaca_ds = alpaca_dataset(tokenizer=tokenizer)
>>> for batch in Dataloader(alpaca_ds, batch_size=8):
>>>     print(f"Batch size: {len(batch)}")
>>> Batch size: 8