FSSpecFileLister

class torchdata.datapipes.iter.FSSpecFileLister(root: Union[str, Sequence[str], IterDataPipe], masks: Union[str, List[str]] = '', **kwargs)

Lists the contents of the directory at the provided root pathname or URL, and yields the full pathname or URL for each file within the directory (functional name: list_files_by_fsspec).

Parameters:

• root – The root fsspec path directory or list of path directories to list files from

• masks – Unix style filter string or string list for filtering file name(s)

• kwargs – Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc.

Example:

from torchdata.datapipes.iter import FSSpecFileLister

datapipe = FSSpecFileLister(root=dir_path)