FileOpener¶
- class torchdata.datapipes.iter.FileOpener(datapipe: Iterable[str], mode: str = 'r', encoding: Optional[str] = None, length: int = -1)¶
Given pathnames, opens files and yield pathname and file stream in a tuple (functional name:
open_files
).- Parameters:
datapipe – Iterable datapipe that provides pathnames
mode – An optional string that specifies the mode in which the file is opened by
open()
. It defaults tor
, other options areb
for reading in binary mode andt
for text mode.encoding – An optional string that specifies the encoding of the underlying file. It defaults to
None
to match the default encoding ofopen
.length – Nominal length of the datapipe
Note
The opened file handles will be closed by Python’s GC periodically. Users can choose to close them explicitly.
Example
>>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import FileLister, FileOpener, StreamReader >>> dp = FileLister(root=".").filter(lambda fname: fname.endswith('.txt')) >>> dp = FileOpener(dp) >>> dp = StreamReader(dp) >>> list(dp) [('./abc.txt', 'abc')]