instruct_dataset

torchtune.datasets.instruct_dataset(tokenizer: ModelTokenizer, *, source: str, column_map: Optional[Dict[str, str]] = None, train_on_input: bool = False, new_system_prompt: Optional[str] = None, packed: bool = False, filter_fn: Optional[Callable] = None, split: str = 'train', **load_dataset_kwargs: Dict[str, Any]) → Union[SFTDataset, PackedDataset]

Configure a custom dataset with user instruction prompts and model responses.

This builder function can be used to configure a custom instruct dataset directly from the yaml config as an alternative to SFTDataset, as it is made to be config friendly.

The dataset should follow this format:

|  input          |  output          |
|-----------------|------------------|
| "user prompt"   | "model response" |

If your column names are different, you can use the column_map parameter to change the expected column names. For example, if your dataset has columns "question" and "answer" you can use:

column_map = {"input": "question", "output": "answer"}

Masking of the prompt during training is controlled by the train_on_input flag, which is set to False by default - If train_on_input is True, the prompt is used during training and contributes to the loss. - If train_on_input is False, the prompt is masked out (tokens replaced with -100)

Parameters:

• tokenizer (ModelTokenizer) – Tokenizer used by the model that implements the tokenize_messages method.

• source (str) – path to dataset repository on Hugging Face. For local datasets, define source as the data file type (e.g. “json”, “csv”, “text”), pass in the filepath in data_files, and set split="train". See Hugging Face’s load_dataset for more details.

• column_map (Optional[Dict[str, str]]) – a mapping to change the expected “input” and “output” column names to the actual column names in the dataset. Keys should be “input” and “output” and values should be the actual column names. Default is None, keeping the default “input” and “output” column names.

• train_on_input (bool) – Whether the model is trained on the user prompt or not. Default is False.

• new_system_prompt (Optional[str]) – if specified, prepend a system message. This can serve as instructions to guide the model response. Default is None.

• packed (bool) – Whether or not to pack the dataset to tokenizer’s max_seq_len prior to training. Default is False.

• filter_fn (Optional[Callable]) – callable used to filter the dataset prior to any pre-processing. See the Hugging Face docs for more details.

• split (str) – split argument for datasets.load_dataset. You can use this argument to load a subset of a given split, e.g. split="train[:10%]". Default is “train”.

• **load_dataset_kwargs (Dict[str, Any]) – additional keyword arguments to pass to load_dataset, such as data_files or split.

Examples:

my_dataset.json
[
    {
        "question": "What time is it in London?",
        "answer": "It is 10:00 AM in London.",
    },
    {
        ...
    },
    ...,
]

>>> from torchtune.datasets import instruct_dataset
>>> dataset = instruct_dataset(
...     tokenizer=tokenizer,
...     source="json",
...     data_files="my_dataset.json",
...     column_map={
...         "input": "question",
...         "output": "answer",
...     },
...     train_on_input=False,
...     packed=False,
...     split="train",
... )
>>> tokens = dataset[0]["tokens"]
>>> tokenizer.decode(tokens)
"What time is it in London?It is 10:00 AM in London."

This can also be accomplished via the yaml config:

dataset:
  _component_: torchtune.datasets.instruct_dataset
  source: json
  data_files: my_dataset.json
  column_map:
    input: question
    output: answer
  train_on_input: False
  packed: False
  split: train

Returns:

the configured SFTDataset

or PackedDataset if packed=True

Return type:

Union[SFTDataset, PackedDataset]

Raises:

ValueError – If packed=True and tokenizer.max_seq_len is not set.