FSSpecFileOpener

class torchdata.datapipes.iter.FSSpecFileOpener(source_datapipe: IterDataPipe[str], mode: str = 'r', *, kwargs_for_open: Optional[Dict] = None, **kwargs)

Opens files from input datapipe which contains fsspec paths and yields a tuple of pathname and opened file stream (functional name: open_files_by_fsspec).

Parameters:

• source_datapipe – Iterable DataPipe that provides the pathnames or URLs

• mode – An optional string that specifies the mode in which the file is opened ("r" by default)

• kwargs_for_open – Optional Dict to specify kwargs for opening files (fs.open())

• kwargs – Extra options that are used to establish a particular storage connection, e.g. host, port, username, password, etc.

Example:

from torchdata.datapipes.iter import FSSpecFileLister

datapipe = FSSpecFileLister(root=dir_path)
file_dp = datapipe.open_files_by_fsspec()