Decompressor

class torchdata.datapipes.iter.Decompressor(source_datapipe: IterDataPipe[Tuple[str, IOBase]], file_type: Optional[Union[str, CompressionType]] = None)

Takes tuples of path and compressed stream of data, and returns tuples of path and decompressed stream of data (functional name: decompress). The input compression format can be specified or automatically detected based on the files’ file extensions.

Parameters:

• source_datapipe – IterDataPipe containing tuples of path and compressed stream of data

• file_type – Optional string or CompressionType that represents what compression format of the inputs

Example

>>> from torchdata.datapipes.iter import FileLister, FileOpener
>>> tar_file_dp = FileLister(self.temp_dir.name, "*.tar")
>>> tar_load_dp = FileOpener(tar_file_dp, mode="b")
>>> tar_decompress_dp = Decompressor(tar_load_dp, file_type="tar")
>>> for _, stream in tar_decompress_dp:
>>>     print(stream.read())
b'0123456789abcdef'