IoPathFileLister¶
- class torchdata.datapipes.iter.IoPathFileLister(root: Union[str, Sequence[str], IterDataPipe], masks: Union[str, List[str]] = '', *, pathmgr=None)¶
Lists the contents of the directory at the provided
root
pathname or URL, and yields the full pathname or URL for each file within the directory (functional name:list_files_by_iopath
).- Parameters:
root – The root local filepath or URL directory or list of roots to list files from
masks – Unix style filter string or string list for filtering file name(s)
pathmgr – Custom
iopath.PathManager
. If not specified, a defaultPathManager
is created.
Note
Default
PathManager
currently supports local file path, normal HTTP URL and OneDrive URL. S3 URL is supported only withiopath
>=0.1.9.Example
>>> from torchdata.datapipes.iter import IoPathFileLister >>> datapipe = IoPathFileLister(root=S3URL)