Flattener

class torchdata.datapipes.iter.Flattener(datapipe: IterDataPipe, indices: Optional[Union[Hashable, List[Hashable]]] = None)

returns a flattened copy of the input DataPipe at the per sample/element level based on provided indices (functional name: flatten).

Note

no args will flatten each item in the datapipe 1 level

Parameters:

• datapipe – IterDataPipe with iterable elements

• indices –

  a single index/key for the item to flatten from an iterator item or a list of indices/keys to be flattened

  • Integer(s) is/are used for list/tuple.

  • Key(s) is/are used for dict.

Example

>>> from torchdata.datapipes.iter import IterableWrapper
>>> dp = IterableWrapper([(0, 10, (100, 1000)), (1, 11, (111, 1001)), (2, 12, (122, 1002)), (3, 13, (133, 1003)), (4, 14, (144, 1004))])
>>> flatten_dp = dp.flatten(2)
>>> list(flatten_dp)
[(0, 10, 100, 1000), (1, 11, 111, 1001), (2, 12, 122, 1002), (3, 13, 133, 1003), (4, 14, 144, 1004)]
>>>
>>> dp = IterableWrapper([(0, (1, 2)), (3, (4, 5)), (6, (7, 8))])
>>> flatten_dp = dp.flatten()
>>> list(flatten_dp)
[(0, 1, 2), (3, 4, 5), (6, 7, 8)]