FSSpecFileOpener

class torchdata.datapipes.iter.FSSpecFileOpener(source_datapipe: IterDataPipe[str], mode: str = 'r')

Opens files from input datapipe which contains fsspec paths and yields a tuple of pathname and opened file stream (functional name: open_file_by_fsspec).

Parameters

• source_datapipe – Iterable DataPipe that provides the pathnames or URLs

• mode – An optional string that specifies the mode in which the file is opened ("r" by default)

Example

>>> from torchdata.datapipes.iter import FSSpecFileLister
>>> datapipe = FSSpecFileLister(root=dir_path)
>>> file_dp = datapipe.open_file_by_fsspec()