slimorca_dataset

torchtune.datasets.slimorca_dataset(tokenizer: ModelTokenizer, *, source: str = 'Open-Orca/SlimOrca-Dedup', column_map: Optional[Dict[str, str]] = None, train_on_input: bool = False, new_system_prompt: Optional[str] = None, packed: bool = False, filter_fn: Optional[Callable] = None, split: str = 'train', **load_dataset_kwargs: Dict[str, Any]) → Union[SFTDataset, PackedDataset]

Support for SlimOrca-style family of conversational datasets.

Masking of the prompt during training is controlled by the train_on_input flag, which is set to False by default - If train_on_input is True, the prompt is used during training and contributes to the loss. - If train_on_input is False, the prompt is masked out (tokens replaced with -100)

Parameters:

• tokenizer (ModelTokenizer) – Tokenizer used by the model that implements the tokenize_messages method.

• source (str) – path to dataset repository on Hugging Face. For local datasets, define source as the data file type (e.g. “json”, “csv”, “text”), pass in the filepath in data_files, and set split="train". See Hugging Face’s load_dataset for more details. Default is Open-Orca/SlimOrca-Dedup.

• column_map (Optional[Dict[str, str]]) – a mapping from the expected columns in the message transform ShareGPTToMessages to the new column names in the dataset. Key should be “conversations” and value should be the new column name. If None, use the default column name "conversations" in Open-Orca/SlimOrca-Dedup.

• train_on_input (bool) – Whether the model is trained on the prompt or not. Default is False.

• new_system_prompt (Optional[str]) – if specified, prepend a system message to every sample. This can serve as instructions to guide the model response. Setting this will OVERRIDE any system messages already present in the dataset. Default is None.

• packed (bool) – Whether or not to pack the dataset to tokenizer’s max_seq_len prior to training. Default is False.

• filter_fn (Optional[Callable]) – callable used to filter the dataset prior to any pre-processing. See the Hugging Face docs for more details.

• split (str) – split argument for datasets.load_dataset. You can use this argument to load a subset of a given split, e.g. split="train[:10%]". Default is “train”.

• **load_dataset_kwargs (Dict[str, Any]) – additional keyword arguments to pass to load_dataset.

Returns:

dataset configured with SlimOrca source data

Return type:

Union[SFTDataset, PackedDataset]

Raises:

ValueError – If packed=True and tokenizer.max_seq_len is not set.

Example

>>> ds = slimorca_dataset(tokenizer=tokenizer)
>>> for input, label in ds:
>>>     print(input)
>>>     print(label)
>>>
>>> Sample Output:
>>> [1, 351, 82, 391, 221, 220, 193, 12, 471, ..., 2]
>>> [-100, -100, -100, -100, -100, -100, -100, -100, 471, ..., 2]