Attention

June 2024 Status Update: Removing DataPipes and DataLoader V2

We are re-focusing the torchdata repo to be an iterative enhancement of torch.utils.data.DataLoader. We do not plan on continuing development or maintaining the [DataPipes] and [DataLoaderV2] solutions, and they will be removed from the torchdata repo. We’ll also be revisiting the DataPipes references in pytorch/pytorch. In release torchdata==0.8.0 (July 2024) they will be marked as deprecated, and in 0.9.0 (Oct 2024) they will be deleted. Existing users are advised to pin to torchdata==0.8.0 or an older version until they are able to migrate away. Subsequent releases will not include DataPipes or DataLoaderV2. Please reach out if you suggestions or comments (please use this issue for feedback)

to_graph

torchdata.datapipes.utils.to_graph(dp, *, debug: bool = False) → graphviz.Digraph

Visualizes a DataPipe by returning a graphviz.Digraph, which is a graph of the data pipeline. This allows you to visually inspect all the transformation that takes place in your DataPipes.

Note

The package graphviz is required to use this function.

Note

The most common interfaces for the returned graph object are:

• render(): Save the graph to a file.

• view(): Open the graph in a viewer.

Parameters:

• dp – DataPipe that you would like to visualize (generally the last one in a chain of DataPipes).

• debug (bool) – If True, renders internal datapipes that are usually hidden from the user (such as ChildDataPipe of demux and fork). Defaults to False.

Example

>>> from torchdata.datapipes.iter import IterableWrapper
>>> from torchdata.datapipes.utils import to_graph
>>> dp = IterableWrapper(range(10))
>>> dp1, dp2 = dp.demux(num_instances=2, classifier_fn=lambda x: x % 2)
>>> dp1 = dp1.map(lambda x: x + 1)
>>> dp2 = dp2.filter(lambda _: True)
>>> dp3 = dp1.zip(dp2).map(lambda t: t[0] + t[1])
>>> g = to_graph(dp3)
>>> g.view()  # This will open the graph in a viewer