Attention

June 2024 Status Update: Removing DataPipes and DataLoader V2

We are re-focusing the torchdata repo to be an iterative enhancement of torch.utils.data.DataLoader. We do not plan on continuing development or maintaining the [DataPipes] and [DataLoaderV2] solutions, and they will be removed from the torchdata repo. We’ll also be revisiting the DataPipes references in pytorch/pytorch. In release torchdata==0.8.0 (July 2024) they will be marked as deprecated, and in 0.9.0 (Oct 2024) they will be deleted. Existing users are advised to pin to torchdata==0.8.0 or an older version until they are able to migrate away. Subsequent releases will not include DataPipes or DataLoaderV2. Please reach out if you suggestions or comments (please use this issue for feedback)

S3FileLister

class torchdata.datapipes.iter.S3FileLister(source_datapipe: IterDataPipe[str], length: int = - 1, request_timeout_ms=- 1, region='', masks: Union[str, List[str]] = '')

[DEPRECATED] Use https://github.com/awslabs/s3-connector-for-pytorch instead.

Iterable DataPipe that lists Amazon S3 file URLs with the given prefixes (functional name: list_files_by_s3). Acceptable prefixes include s3://bucket-name, s3://bucket-name/, s3://bucket-name/folder.

Note

1. source_datapipe must contain a list of valid S3 URLs

2. length is -1 by default, and any call to __len__() is invalid, because the length is unknown until all files are iterated.

3. request_timeout_ms and region will overwrite settings in the configuration file or environment variables.

4. The lack of AWS proper configuration can lead empty response. For more details related to S3 IO DataPipe setup and AWS config, please see the README file.

Parameters:

• source_datapipe – a DataPipe that contains URLs/URL prefixes to s3 files

• length – Nominal length of the datapipe

• request_timeout_ms – timeout setting for each reqeust (3,000ms by default)

• region – region for access files (inferred from credentials by default)

Example:

from torchdata.datapipes.iter import IterableWrapper, S3FileLister

s3_prefixes = IterableWrapper(['s3://bucket-name/folder/', ...])

dp_s3_urls = S3FileLister(s3_prefixes)
for d in dp_s3_urls:
    pass

# Functional API
dp_s3_urls = s3_prefixes.list_files_by_s3(request_timeout_ms=100)
for d in dp_s3_urls:
    pass