Shortcuts

UnZipper

class torchdata.datapipes.iter.UnZipper(source_datapipe: IterDataPipe[Sequence[T]], sequence_length: int, buffer_size: int = 1000, columns_to_skip: Optional[Sequence[int]] = None)

Takes in a DataPipe of Sequences, unpacks each Sequence, and return the elements in separate DataPipes based on their position in the Sequence (functional name: unzip). The number of instances produced equals to the sequence length minus the number of columns to skip.

Note

Each sequence within the DataPipe should have the same length, specified by the input argument sequence_length.

Parameters:
  • source_datapipe – Iterable DataPipe with sequences of data

  • sequence_length – Length of the sequence within the source_datapipe. All elements should have the same length.

  • buffer_size – this restricts how far ahead the leading child DataPipe can read relative to the slowest child DataPipe. Use -1 for the unlimited buffer.

  • columns_to_skip – optional indices of columns that the DataPipe should skip (each index should be an integer from 0 to sequence_length - 1)

Example

>>> from torchdata.datapipes.iter import IterableWrapper
>>> source_dp = IterableWrapper([(i, i + 10, i + 20) for i in range(3)])
>>> dp1, dp2, dp3 = source_dp.unzip(sequence_length=3)
>>> list(dp1)
[0, 1, 2]
>>> list(dp2)
[10, 11, 12]
>>> list(dp3)
[20, 21, 22]

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources