IoPathFileOpener

class torchdata.datapipes.iter.IoPathFileOpener(source_datapipe: IterDataPipe[str], mode: str = 'r', pathmgr=None, handler=None)

Opens files from input datapipe which contains pathnames or URLs, and yields a tuple of pathname and opened file stream (functional name: open_files_by_iopath).

Parameters:

• source_datapipe – Iterable DataPipe that provides the pathnames or URLs

• mode – An optional string that specifies the mode in which the file is opened ("r" by default)

• pathmgr – Custom iopath.PathManager. If not specified, a default PathManager is created.

Note

Default PathManager currently supports local file path, normal HTTP URL and OneDrive URL. S3 URL is supported only with iopath>=0.1.9.

Example:

from torchdata.datapipes.iter import IoPathFileLister

datapipe = IoPathFileLister(root=s3_url)
file_dp = datapipe.open_files_by_iopath()