IoPathFileOpener¶
- class torchdata.datapipes.iter.IoPathFileOpener(source_datapipe: IterDataPipe[str], mode: str = 'r', pathmgr=None)¶
Opens files from input datapipe which contains pathnames or URLs, and yields a tuple of pathname and opened file stream (functional name:
open_files_by_iopath
).- Parameters:
source_datapipe – Iterable DataPipe that provides the pathnames or URLs
mode – An optional string that specifies the mode in which the file is opened (
"r"
by default)pathmgr – Custom
iopath.PathManager
. If not specified, a defaultPathManager
is created.
Note
Default
PathManager
currently supports local file path, normal HTTP URL and OneDrive URL. S3 URL is supported only with iopath>=0.1.9.Example
>>> from torchdata.datapipes.iter import IoPathFileLister >>> datapipe = IoPathFileLister(root=S3URL) >>> file_dp = datapipe.open_files_by_iopath()