FSSpecFileLister¶
- class torchdata.datapipes.iter.FSSpecFileLister(root: Union[str, Sequence[str], IterDataPipe], masks: Union[str, List[str]] = '', **kwargs)¶
Lists the contents of the directory at the provided
root
pathname or URL, and yields the full pathname or URL for each file within the directory (functional name:list_files_by_fsspec
).- Parameters:
root – The root fsspec path directory or list of path directories to list files from
masks – Unix style filter string or string list for filtering file name(s)
kwargs – Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc.
Example
>>> from torchdata.datapipes.iter import FSSpecFileLister >>> datapipe = FSSpecFileLister(root=dir_path)