Saver¶
- class torchdata.datapipes.iter.Saver(source_datapipe: IterDataPipe[Tuple[Any, Union[bytes, bytearray, str]]], mode: str = 'w', filepath_fn: Optional[Callable] = None)¶
Takes in a DataPipe of tuples of metadata and data, saves the data to the target path generated by the
filepath_fn
and metadata, and yields file path on local file system (functional name:save_to_disk
).- Parameters:
source_datapipe – Iterable DataPipe with tuples of metadata and data
mode – Node in which the file will be opened for write the data (
"w"
by default)filepath_fn – Function that takes in metadata and returns the target path of the new file
Example
>>> from torchdata.datapipes.iter import IterableWrapper >>> import os >>> def filepath_fn(name: str) -> str: >>> return os.path.join(".", os.path.basename(name)) >>> name_to_data = {"1.txt": b"DATA1", "2.txt": b"DATA2", "3.txt": b"DATA3"} >>> source_dp = IterableWrapper(sorted(name_to_data.items())) >>> saver_dp = source_dp.save_to_disk(filepath_fn=filepath_fn, mode="wb") >>> res_file_paths = list(saver_dp) >>> res_file_paths ['./1.txt', './2.txt', './3.txt']