Shortcuts

MapKeyZipper

class torchdata.datapipes.iter.MapKeyZipper(source_iterdatapipe: IterDataPipe, map_datapipe: MapDataPipe, key_fn: Callable, merge_fn: Optional[Callable] = None, keep_key: bool = False)

Joins the items from the source IterDataPipe with items from a MapDataPipe (functional name: zip_with_map). The matching is done by the provided key_fn, which maps an item from source_iterdatapipe to a key that should exist in the map_datapipe. The return value is created by the merge_fn, which returns a tuple of the two items by default.

Parameters:
  • source_iterdatapipe – IterDataPipe from which items are yield and will be combined with an item from map_datapipe

  • map_datapipe – MapDataPipe that takes a key from key_fn, and returns an item

  • key_fn – Function that maps each item from source_iterdatapipe to a key that exists in map_datapipe

  • keep_key – Option to yield the matching key along with the items in a tuple, resulting in (key, merge_fn(item1, item2)).

  • merge_fn – Function that combines the item from source_iterdatapipe and the matching item from map_datapipe, by default a tuple is created

Example:

from torchdata.datapipes.iter import IterableWrapper
from torchdata.datapipes.map import SequenceWrapper


def merge_fn(tuple_from_iter, value_from_map):
    return tuple_from_iter[0], tuple_from_iter[1] + value_from_map


dp1 = IterableWrapper([('a', 1), ('b', 2), ('c', 3)])
mapdp = SequenceWrapper({'a': 100, 'b': 200, 'c': 300, 'd': 400})
res_dp = dp1.zip_with_map(map_datapipe=mapdp, key_fn=itemgetter(0), merge_fn=merge_fn)

print(list(res_dp))

Output:

[('a', 101), ('b', 202), ('c', 303)]

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources