Shortcuts

FSSpecFileOpener

class torchdata.datapipes.iter.FSSpecFileOpener(source_datapipe: IterDataPipe[str], mode: str = 'r', *, kwargs_for_open: Optional[Dict] = None, **kwargs)

Opens files from input datapipe which contains fsspec paths and yields a tuple of pathname and opened file stream (functional name: open_files_by_fsspec).

Parameters:
  • source_datapipe – Iterable DataPipe that provides the pathnames or URLs

  • mode – An optional string that specifies the mode in which the file is opened ("r" by default)

  • kwargs_for_open – Optional Dict to specify kwargs for opening files (fs.open())

  • kwargs – Extra options that are used to establish a particular storage connection, e.g. host, port, username, password, etc.

Example

>>> from torchdata.datapipes.iter import FSSpecFileLister
>>> datapipe = FSSpecFileLister(root=dir_path)
>>> file_dp = datapipe.open_files_by_fsspec()

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources