Shortcuts

ShardingFilter

class torchdata.datapipes.iter.ShardingFilter(source_datapipe: IterDataPipe)

Wrapper that allows DataPipe to be sharded (functional name: sharding_filter). After apply_sharding is called, each instance of the DataPipe (on different workers) will have every n-th element of the original DataPipe, where n equals to the number of instances.

Parameters:

source_datapipe – Iterable DataPipe that will be sharded

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources