ShardingFilter¶
- class torchdata.datapipes.iter.ShardingFilter(source_datapipe: IterDataPipe)¶
Wrapper that allows DataPipe to be sharded (functional name:
sharding_filter
). Afterapply_sharding
is called, each instance of the DataPipe (on different workers) will have every n-th element of the original DataPipe, where n equals to the number of instances.- Parameters:
source_datapipe – Iterable DataPipe that will be sharded