class torchdata.dataloader2.PrototypeMultiProcessingReadingService(num_workers: int = 0, multiprocessing_context=None, prefetch_worker: int = 10, prefetch_mainloop: int = 10)

PrototypeMultiProcessingReadingService that spawns multiple subprocesses to iterate the DataPipe graph. This ReadingService is still under prototype stage and will replace MultiProcessingReadingService eventually.

  • num_workers (int, optional) – How many subprocesses to use for data loading. 0 will be replaced by InProcessReadingService in the future.

  • multiprocessing_context (str, optional) – Multiprocessing starting method. If method is None then the default context is returned. Otherwise method should be ‘fork’, ‘spawn’.

finalize() None

PrototypeMultiProcessingReadingService invalidate states & properly exits all subprocesses.

initialize(datapipe: Union[IterDataPipe, MapDataPipe]) Union[IterDataPipe, MapDataPipe]

PrototypeMultiProcessingReadingService finds information about sharding, separates graph by multiple pieces and reconnects it using queues. creates subprocesses.

initialize_iteration() None

ReadingService spins up service for an epoch. Called at the beginning of every time getting DataLoader2 iterator.


Access comprehensive developer documentation for PyTorch

View Docs


Get in-depth tutorials for beginners and advanced developers

View Tutorials


Find development resources and get your questions answered

View Resources