Shortcuts

IoPathFileOpener

class torchdata.datapipes.iter.IoPathFileOpener(source_datapipe: IterDataPipe[str], mode: str = 'r', pathmgr=None)

Opens files from input datapipe which contains pathnames or URLs, and yields a tuple of pathname and opened file stream (functional name: open_files_by_iopath).

Parameters:
  • source_datapipe – Iterable DataPipe that provides the pathnames or URLs

  • mode – An optional string that specifies the mode in which the file is opened ("r" by default)

  • pathmgr – Custom iopath.PathManager. If not specified, a default PathManager is created.

Note

Default PathManager currently supports local file path, normal HTTP URL and OneDrive URL. S3 URL is supported only with iopath>=0.1.9.

Example

>>> from torchdata.datapipes.iter import IoPathFileLister
>>> datapipe = IoPathFileLister(root=S3URL)
>>> file_dp = datapipe.open_files_by_iopath()

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources