Shortcuts

FileOpener

class torchdata.datapipes.iter.FileOpener(datapipe: Iterable[str], mode: str = 'r', encoding: Optional[str] = None, length: int = -1)

Given pathnames, opens files and yield pathname and file stream in a tuple (functional name: open_files).

Parameters:
  • datapipe – Iterable datapipe that provides pathnames

  • mode – An optional string that specifies the mode in which the file is opened by open(). It defaults to b which means open for reading in binary mode. Another option is to use t for text mode

  • encoding – An optional string that specifies the encoding of the underlying file. It defaults to None to match the default encoding of open.

  • length – Nominal length of the datapipe

Note

The opened file handles will be closed by Python’s GC periodically. Users can choose to close them explicitly.

Example

>>> from torchdata.datapipes.iter import FileLister, FileOpener, StreamReader
>>> dp = FileLister(root=".").filter(lambda fname: fname.endswith('.txt'))
>>> dp = FileOpener(dp)
>>> dp = StreamReader(dp)
>>> list(dp)
[('./abc.txt', 'abc')]

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources