FlatMapper

class torchdata.datapipes.iter.FlatMapper(datapipe: IterDataPipe, fn: Optional[Callable] = None, input_col=None)

Applies a function over each item from the source DataPipe, then flattens the outputs to a single, unnested IterDataPipe (functional name: flatmap).

Note

The output from fn must be a Sequence. Otherwise, an error will be raised. If fn is None, source DataPipe will be just flattened vertically, provided that items can be unpacked.

Parameters:

• datapipe – Source IterDataPipe

• fn – the function to be applied to each element in the DataPipe, the output must be a Sequence

• input_col – Index or indices of data which fn is applied, such as: - None as default to apply fn to the data directly. - Integer(s) is/are used for list/tuple. - Key(s) is/are used for dict.

Example

>>> from torchdata.datapipes.iter import IterableWrapper
>>> def fn(e):
>>>     return [e, e * 10]
>>> source_dp = IterableWrapper(list(range(5)))
>>> flatmapped_dp = source_dp.flatmap(fn)
>>> list(flatmapped_dp)
[0, 0, 1, 10, 2, 20, 3, 30, 4, 40]
>>>
>>> source_dp = IterableWrapper([[1, 2, 3], [4, 5, 6]])
>>> flatmapped_dp = source_dp.flatmap()
>>> list(flatmapped_dp)
[1, 2, 3, 4, 5, 6]