ShardingFilter

class torchdata.datapipes.iter.ShardingFilter(source_datapipe: IterDataPipe, sharding_group_filter=None)

Wrapper that allows DataPipe to be sharded (functional name: sharding_filter). After apply_sharding is called, each instance of the DataPipe (on different workers) will have every n-th element of the original DataPipe, where n equals to the number of instances.

Parameters:

source_datapipe – Iterable DataPipe that will be sharded