FSSpecFileOpener¶
- class torchdata.datapipes.iter.FSSpecFileOpener(source_datapipe: IterDataPipe[str], mode: str = 'r', *, kwargs_for_open: Optional[Dict] = None, **kwargs)¶
Opens files from input datapipe which contains fsspec paths and yields a tuple of pathname and opened file stream (functional name:
open_files_by_fsspec
).- Parameters:
source_datapipe – Iterable DataPipe that provides the pathnames or URLs
mode – An optional string that specifies the mode in which the file is opened (
"r"
by default)kwargs_for_open – Optional Dict to specify kwargs for opening files (
fs.open()
)kwargs – Extra options that are used to establish a particular storage connection, e.g. host, port, username, password, etc.
Example
>>> from torchdata.datapipes.iter import FSSpecFileLister >>> datapipe = FSSpecFileLister(root=dir_path) >>> file_dp = datapipe.open_files_by_fsspec()