Attention

June 2024 Status Update: Removing DataPipes and DataLoader V2

We are re-focusing the torchdata repo to be an iterative enhancement of torch.utils.data.DataLoader. We do not plan on continuing development or maintaining the [DataPipes] and [DataLoaderV2] solutions, and they will be removed from the torchdata repo. We’ll also be revisiting the DataPipes references in pytorch/pytorch. In release torchdata==0.8.0 (July 2024) they will be marked as deprecated, and in 0.9.0 (Oct 2024) they will be deleted. Existing users are advised to pin to torchdata==0.8.0 or an older version until they are able to migrate away. Subsequent releases will not include DataPipes or DataLoaderV2. Please reach out if you suggestions or comments (please use this issue for feedback)

UnZipper

class torchdata.datapipes.iter.UnZipper(source_datapipe: IterDataPipe[Sequence[T]], sequence_length: int, buffer_size: int = 1000, columns_to_skip: Optional[Sequence[int]] = None)

Takes in a DataPipe of Sequences, unpacks each Sequence, and return the elements in separate DataPipes based on their position in the Sequence (functional name: unzip). The number of instances produced equals to the sequence length minus the number of columns to skip.

Note

Each sequence within the DataPipe should have the same length, specified by the input argument sequence_length.

Parameters:

• source_datapipe – Iterable DataPipe with sequences of data

• sequence_length – Length of the sequence within the source_datapipe. All elements should have the same length.

• buffer_size – this restricts how far ahead the leading child DataPipe can read relative to the slowest child DataPipe. Use -1 for the unlimited buffer.

• columns_to_skip – optional indices of columns that the DataPipe should skip (each index should be an integer from 0 to sequence_length - 1)

Example

>>> from torchdata.datapipes.iter import IterableWrapper
>>> source_dp = IterableWrapper([(i, i + 10, i + 20) for i in range(3)])
>>> dp1, dp2, dp3 = source_dp.unzip(sequence_length=3)
>>> list(dp1)
[0, 1, 2]
>>> list(dp2)
[10, 11, 12]
>>> list(dp3)
[20, 21, 22]