InMemoryCacheHolder¶
- class torchdata.datapipes.map.InMemoryCacheHolder(source_dp: MapDataPipe[T_co])¶
Stores elements from the source DataPipe in memory (functional name:
in_memory_cache
). Once an item is stored, it will remain unchanged and subsequent retrivals will return the same element. Since items fromMapDataPipe
are lazily computed, this can be used to store the results from previousMapDataPipe
and reduce the number of duplicate computations.Note
The default
cache
is aDict
. If another data structure is more suitable as cache for your use- Parameters:
source_dp – source DataPipe from which elements are read and stored in memory
Example
>>> from torchdata.datapipes.map import SequenceWrapper >>> source_dp = SequenceWrapper(range(10)) >>> cache_dp = source_dp.in_memory_cache() >>> list(cache_dp) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]