IoPathFileLister

class torchdata.datapipes.iter.IoPathFileLister(root: Union[str, Sequence[str], IterDataPipe], masks: Union[str, List[str]] = '', *, pathmgr=None, handler=None)

Lists the contents of the directory at the provided root pathname or URL, and yields the full pathname or URL for each file within the directory (functional name: list_files_by_iopath).

Parameters:

• root – The root local filepath or URL directory or list of roots to list files from

• masks – Unix style filter string or string list for filtering file name(s)

• pathmgr – Custom iopath.PathManager. If not specified, a default PathManager is created.

Note

Default PathManager currently supports local file path, normal HTTP URL and OneDrive URL. S3 URL is supported only with iopath>=0.1.9.

Example:

from torchdata.datapipes.iter import IoPathFileLister

datapipe = IoPathFileLister(root=s3_url)