Shortcuts

EndOnDiskCacheHolder

class torchdata.datapipes.iter.EndOnDiskCacheHolder(datapipe, mode='wb', filepath_fn=None, *, same_filepath_fn=False, skip_read=False, timeout=300)

Indicates when the result of prior DataPipe will be saved local files specified by filepath_fn (functional name: end_caching). Moreover, the result of source DataPipe is required to be a tuple of metadata and data, or a tuple of metadata and file handle.

Parameters:
  • datapipe – IterDataPipe with at least one OnDiskCacheHolder in the graph.

  • mode – Mode in which the cached files are opened to write the data on disk. This is needed to be aligned with the type of data or file handle from datapipe. "wb" is used by default.

  • filepath_fn – Optional function to extract filepath from the metadata from datapipe. By default, it would directly use the metadata as file path.

  • same_filepath_fn – Set to True to use same filepath_fn from the OnDiskCacheHolder.

  • skip_read – Boolean value to skip reading the file handle from datapipe. By default, reading is enabled and reading function is created based on the mode.

  • timeout – Integer value of seconds to wait for uncached item to be written to disk

Example

>>> from torchdata.datapipes.iter import IterableWrapper, HttpReader
>>> url = IterableWrapper(["https://path/to/filename", ])
>>> def _filepath_fn(url):
>>>     temp_dir = tempfile.gettempdir()
>>>     return os.path.join(temp_dir, os.path.basename(url))
>>> hash_dict = {"expected_filepath": expected_MD5_hash}
>>> # You must call ``.on_disk_cache`` at some point before ``.end_caching``
>>> cache_dp = url.on_disk_cache(filepath_fn=_filepath_fn, hash_dict=_hash_dict, hash_type="md5")
>>> # You must call ``.end_caching`` at a later point to stop tracing and save the results to local files.
>>> cache_dp = HttpReader(cache_dp).end_caching(mode="wb", filepath_fn=_filepath_fn)

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources