Note
Click here to download the full example code
torch.compile Tutorial¶
Author: William Wen
torch.compile is the latest method to speed up your PyTorch code!
torch.compile makes PyTorch code run faster by
JIT-compiling PyTorch code into optimized kernels,
all while requiring minimal code changes.
In this tutorial, we cover basic torch.compile usage,
and demonstrate the advantages of torch.compile over
previous PyTorch compiler solutions, such as
TorchScript and
FX Tracing.
Contents
Basic Usage
Demonstrating Speedups
Comparison to TorchScript and FX Tracing
TorchDynamo and FX Graphs
Conclusion
Required pip Dependencies
torch >= 2.0torchvisionnumpyscipytabulate
NOTE: a modern NVIDIA GPU (H100, A100, or V100) is recommended for this tutorial in order to reproduce the speedup numbers shown below and documented elsewhere.
import torch
import warnings
gpu_ok = False
if torch.cuda.is_available():
device_cap = torch.cuda.get_device_capability()
if device_cap in ((7, 0), (8, 0), (9, 0)):
gpu_ok = True
if not gpu_ok:
warnings.warn(
"GPU is not NVIDIA V100, A100, or H100. Speedup numbers may be lower "
"than expected."
)
/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py:51: UserWarning:
GPU is not NVIDIA V100, A100, or H100. Speedup numbers may be lower than expected.
Basic Usage¶
torch.compile is included in the latest PyTorch..
Running TorchInductor on GPU requires Triton, which is included with the PyTorch 2.0 nightly
binary. If Triton is still missing, try installing torchtriton via pip
(pip install torchtriton --extra-index-url "https://download.pytorch.org/whl/nightly/cu117"
for CUDA 11.7).
Arbitrary Python functions can be optimized by passing the callable to
torch.compile. We can then call the returned optimized
function in place of the original function.
def foo(x, y):
a = torch.sin(x)
b = torch.cos(y)
return a + b
opt_foo1 = torch.compile(foo)
print(opt_foo1(torch.randn(10, 10), torch.randn(10, 10)))
tensor([[ 1.6850, 1.9924, 1.7090, 0.0034, 1.1414, -0.1822, 0.4861, -0.0536,
-0.2252, 1.9398],
[ 0.3693, -0.0695, 0.1748, 0.3436, 0.1939, 1.5721, 1.9882, -0.2235,
0.3161, 1.2642],
[ 0.2480, 1.8793, 1.7152, 1.6772, 1.8881, 1.4748, 1.3466, 1.7763,
0.7469, 1.0407],
[-0.1121, 1.6015, -0.0188, 0.2128, 0.5218, 1.9838, 0.8185, 0.5093,
-0.3603, 0.1793],
[-1.7890, 1.7532, -0.4040, 0.1222, -0.0029, 1.7975, -0.3877, 0.5123,
0.1673, 0.1330],
[ 1.0627, 0.9609, 0.1019, 1.8814, 0.1142, -0.2338, -0.9621, 0.7631,
0.6506, 0.1853],
[ 0.4584, 1.7648, -0.0444, 1.9610, 1.5884, 0.7353, 1.2190, 1.3662,
1.0938, -0.1587],
[-0.7502, 1.6640, 0.3495, 1.3496, 0.8187, 1.1719, 0.5820, 0.1498,
0.0885, 0.1036],
[ 0.3961, 0.6043, -0.0861, -0.3371, 0.8622, 1.4341, 1.2988, 0.5023,
0.3074, 0.1277],
[ 0.9748, 0.4117, 1.2616, 1.6314, 0.4693, 0.4092, 0.0401, 1.1196,
1.2458, 1.3280]])
Alternatively, we can decorate the function.
@torch.compile
def opt_foo2(x, y):
a = torch.sin(x)
b = torch.cos(y)
return a + b
print(opt_foo2(torch.randn(10, 10), torch.randn(10, 10)))
tensor([[ 0.5360, 0.1697, -0.0561, 0.1890, -0.1310, 1.2276, 1.1739, 0.1944,
-0.1561, 1.6990],
[ 1.0421, 1.9472, 0.2682, 0.2701, 1.3346, 0.7651, 1.0897, 1.1730,
0.6161, 0.9223],
[ 1.5756, 1.5294, 0.0112, -0.1522, -0.7674, 1.8515, -0.2443, 0.3696,
0.2693, 0.8735],
[-0.3701, 1.1190, 1.4164, 1.8648, 1.2080, 0.0732, 1.5274, 0.6868,
1.2440, 1.0715],
[-1.2454, -0.0159, 0.4315, 0.1317, 1.0530, -1.0603, -0.0532, 0.6661,
1.7101, -0.2076],
[-0.7091, 0.7824, 1.7161, 1.2750, 0.6368, 1.2488, 0.4897, 1.2429,
1.3409, 1.3735],
[ 0.8345, 0.0653, 0.3462, 1.2383, -0.4092, 1.6438, -0.0962, 0.4011,
0.2463, -0.5802],
[ 1.6349, 0.7297, 1.2547, -0.3113, 0.9310, 0.1162, 1.7618, 0.4882,
0.7640, 0.2930],
[ 1.1669, -0.7775, 1.2000, 0.6008, -0.2814, 0.5541, 0.5753, 1.4731,
1.6835, 0.7370],
[ 1.5087, 0.6195, 0.1153, 1.2966, 1.8815, 1.1678, 1.5686, 1.6018,
0.2193, 1.3500]])
We can also optimize torch.nn.Module instances.
class MyModule(torch.nn.Module):
def __init__(self):
super().__init__()
self.lin = torch.nn.Linear(100, 10)
def forward(self, x):
return torch.nn.functional.relu(self.lin(x))
mod = MyModule()
opt_mod = torch.compile(mod)
print(opt_mod(torch.randn(10, 100)))
tensor([[0.0000, 0.0000, 0.2419, 0.0446, 0.9011, 0.2674, 0.3633, 0.4984, 0.0000,
0.0988],
[0.6906, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.8490, 0.0000, 0.0000,
0.5475],
[0.0852, 0.2762, 0.7441, 0.0000, 0.0000, 0.1820, 0.0000, 0.0000, 0.0000,
0.0334],
[0.3024, 0.0077, 1.2572, 0.0000, 0.0000, 0.6520, 0.0000, 0.0000, 0.0000,
0.8976],
[0.1998, 0.3333, 0.0000, 0.7803, 0.4202, 0.0915, 0.0000, 1.2543, 0.0000,
0.4615],
[0.2487, 0.4187, 0.0000, 0.0000, 0.5124, 0.0000, 0.2512, 0.0000, 0.5850,
0.0000],
[0.0000, 0.0048, 0.0000, 0.0000, 0.0000, 0.2287, 0.0000, 0.4841, 0.3915,
0.0000],
[0.2017, 0.0000, 0.0896, 1.4135, 0.0593, 0.3788, 0.0000, 0.0000, 0.0000,
0.4972],
[0.0000, 0.0000, 1.6580, 0.6414, 0.0000, 0.0000, 0.0000, 0.0000, 0.6491,
0.7755],
[0.0000, 0.0000, 0.6442, 0.0260, 0.7456, 0.1000, -0.0000, -0.0000, 0.5366,
0.1193]], grad_fn=<CompiledFunctionBackward>)
Demonstrating Speedups¶
Let’s now demonstrate that using torch.compile can speed
up real models. We will compare standard eager mode and
torch.compile by evaluating and training a torchvision model on random data.
Before we start, we need to define some utility functions.
# Returns the result of running `fn()` and the time it took for `fn()` to run,
# in seconds. We use CUDA events and synchronization for the most accurate
# measurements.
def timed(fn):
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
result = fn()
end.record()
torch.cuda.synchronize()
return result, start.elapsed_time(end) / 1000
# Generates random input and targets data for the model, where `b` is
# batch size.
def generate_data(b):
return (
torch.randn(b, 3, 128, 128).to(torch.float32).cuda(),
torch.randint(1000, (b,)).cuda(),
)
N_ITERS = 10
from torchvision.models import densenet121
def init_model():
return densenet121().to(torch.float32).cuda()
First, let’s compare inference.
Note that in the call to torch.compile, we have have the additional
mode argument, which we will discuss below.
def evaluate(mod, inp):
with torch.no_grad():
return mod(inp)
model = init_model()
# Reset since we are using a different mode.
import torch._dynamo
torch._dynamo.reset()
evaluate_opt = torch.compile(evaluate, mode="reduce-overhead")
inp = generate_data(16)[0]
print("eager:", timed(lambda: evaluate(model, inp))[1])
print("compile:", timed(lambda: evaluate_opt(model, inp))[1])
eager: 0.5178863525390625
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py:90: UserWarning:
TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
compile: 31.207962890625
Notice that torch.compile takes a lot longer to complete
compared to eager. This is because torch.compile compiles
the model into optimized kernels as it executes. In our example, the
structure of the model doesn’t change, and so recompilation is not
needed. So if we run our optimized model several more times, we should
see a significant improvement compared to eager.
eager_times = []
for i in range(N_ITERS):
inp = generate_data(16)[0]
_, eager_time = timed(lambda: evaluate(model, inp))
eager_times.append(eager_time)
print(f"eager eval time {i}: {eager_time}")
print("~" * 10)
compile_times = []
for i in range(N_ITERS):
inp = generate_data(16)[0]
_, compile_time = timed(lambda: evaluate_opt(model, inp))
compile_times.append(compile_time)
print(f"compile eval time {i}: {compile_time}")
print("~" * 10)
import numpy as np
eager_med = np.median(eager_times)
compile_med = np.median(compile_times)
speedup = eager_med / compile_med
print(f"(eval) eager median: {eager_med}, compile median: {compile_med}, speedup: {speedup}x")
print("~" * 10)
eager eval time 0: 0.021500255584716795
eager eval time 1: 0.020305248260498048
eager eval time 2: 0.02053536033630371
eager eval time 3: 0.02035980796813965
eager eval time 4: 0.020322208404541017
eager eval time 5: 0.020166015625
eager eval time 6: 0.02028505516052246
eager eval time 7: 0.020547264099121092
eager eval time 8: 0.020112831115722655
eager eval time 9: 0.020261119842529297
~~~~~~~~~~
compile eval time 0: 0.008191519737243653
compile eval time 1: 0.007924511909484863
compile eval time 2: 0.007882368087768555
compile eval time 3: 0.00788431978225708
compile eval time 4: 0.007995327949523926
compile eval time 5: 0.007876575946807862
compile eval time 6: 0.007959424018859863
compile eval time 7: 0.007887775897979737
compile eval time 8: 0.007952159881591796
compile eval time 9: 0.007978975772857666
~~~~~~~~~~
(eval) eager median: 0.020313728332519532, compile median: 0.007938335895538329, speedup: 2.558940387485579x
~~~~~~~~~~
And indeed, we can see that running our model with torch.compile
results in a significant speedup. Speedup mainly comes from reducing Python overhead and
GPU read/writes, and so the observed speedup may vary on factors such as model
architecture and batch size. For example, if a model’s architecture is simple
and the amount of data is large, then the bottleneck would be
GPU compute and the observed speedup may be less significant.
You may also see different speedup results depending on the chosen mode
argument. Since our model and data are small, we want to reduce overhead as
much as possible, and so we chose "reduce-overhead". For your own models,
you may need to experiment with different modes to maximize speedup. You can
read more about modes here.
For general PyTorch benchmarking, you can try using torch.utils.benchmark instead of the timed
function we defined above. We wrote our own timing function in this tutorial to show
torch.compile’s compilation latency.
Now, let’s consider comparing training.
model = init_model()
opt = torch.optim.Adam(model.parameters())
def train(mod, data):
opt.zero_grad(True)
pred = mod(data[0])
loss = torch.nn.CrossEntropyLoss()(pred, data[1])
loss.backward()
opt.step()
eager_times = []
for i in range(N_ITERS):
inp = generate_data(16)
_, eager_time = timed(lambda: train(model, inp))
eager_times.append(eager_time)
print(f"eager train time {i}: {eager_time}")
print("~" * 10)
model = init_model()
opt = torch.optim.Adam(model.parameters())
train_opt = torch.compile(train, mode="reduce-overhead")
compile_times = []
for i in range(N_ITERS):
inp = generate_data(16)
_, compile_time = timed(lambda: train_opt(model, inp))
compile_times.append(compile_time)
print(f"compile train time {i}: {compile_time}")
print("~" * 10)
eager_med = np.median(eager_times)
compile_med = np.median(compile_times)
speedup = eager_med / compile_med
print(f"(train) eager median: {eager_med}, compile median: {compile_med}, speedup: {speedup}x")
print("~" * 10)
eager train time 0: 0.3570961303710937
eager train time 1: 0.06414991760253906
eager train time 2: 0.06317683029174805
eager train time 3: 0.06316553497314453
eager train time 4: 0.06327590560913086
eager train time 5: 0.06376857757568359
eager train time 6: 0.06317385482788086
eager train time 7: 0.06311648178100586
eager train time 8: 0.0632165756225586
eager train time 9: 0.06327478408813476
~~~~~~~~~~
compile train time 0: 63.55679296875
compile train time 1: 0.035544639587402345
compile train time 2: 0.031085216522216796
compile train time 3: 0.030932607650756835
compile train time 4: 0.031107263565063478
compile train time 5: 0.031834016799926756
compile train time 6: 0.03140592002868652
compile train time 7: 0.03136195182800293
compile train time 8: 0.0310728645324707
compile train time 9: 0.03157318305969238
~~~~~~~~~~
(train) eager median: 0.06324567985534668, compile median: 0.03138393592834472, speedup: 2.0152246040696795x
~~~~~~~~~~
Again, we can see that torch.compile takes longer in the first
iteration, as it must compile the model, but in subsequent iterations, we see
significant speedups compared to eager.
Comparison to TorchScript and FX Tracing¶
We have seen that torch.compile can speed up PyTorch code.
Why else should we use torch.compile over existing PyTorch
compiler solutions, such as TorchScript or FX Tracing? Primarily, the
advantage of torch.compile lies in its ability to handle
arbitrary Python code with minimal changes to existing code.
One case that torch.compile can handle that other compiler
solutions struggle with is data-dependent control flow (the
if x.sum() < 0: line below).
def f1(x, y):
if x.sum() < 0:
return -y
return y
# Test that `fn1` and `fn2` return the same result, given
# the same arguments `args`. Typically, `fn1` will be an eager function
# while `fn2` will be a compiled function (torch.compile, TorchScript, or FX graph).
def test_fns(fn1, fn2, args):
out1 = fn1(*args)
out2 = fn2(*args)
return torch.allclose(out1, out2)
inp1 = torch.randn(5, 5)
inp2 = torch.randn(5, 5)
TorchScript tracing f1 results in
silently incorrect results, since only the actual control flow path
is traced.
/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py:268: TracerWarning:
Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
traced 1, 1: True
traced 1, 2: False
FX tracing f1 results in an error due to the presence of
data-dependent control flow.
import traceback as tb
try:
torch.fx.symbolic_trace(f1)
except:
tb.print_exc()
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py", line 298, in <module>
torch.fx.symbolic_trace(f1)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 1109, in symbolic_trace
graph = tracer.trace(root, concrete_args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 778, in trace
(self.create_arg(fn(*args)),),
File "/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py", line 268, in f1
if x.sum() < 0:
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/proxy.py", line 413, in __bool__
return self.tracer.to_bool(self)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/proxy.py", line 276, in to_bool
raise TraceError('symbolically traced variables cannot be used as inputs to control flow')
torch.fx.proxy.TraceError: symbolically traced variables cannot be used as inputs to control flow
If we provide a value for x as we try to FX trace f1, then
we run into the same problem as TorchScript tracing, as the data-dependent
control flow is removed in the traced function.
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py:602: UserWarning:
Was not able to add assertion to guarantee correct input x to specialized function. It is up to the user to make sure that your inputs match the inputs you specialized the function with.
fx 1, 1: True
fx 1, 2: False
Now we can see that torch.compile correctly handles
data-dependent control flow.
# Reset since we are using a different mode.
torch._dynamo.reset()
compile_f1 = torch.compile(f1)
print("compile 1, 1:", test_fns(f1, compile_f1, (inp1, inp2)))
print("compile 1, 2:", test_fns(f1, compile_f1, (-inp1, inp2)))
print("~" * 10)
compile 1, 1: True
compile 1, 2: True
~~~~~~~~~~
TorchScript scripting can handle data-dependent control flow, but this solution comes with its own set of problems. Namely, TorchScript scripting can require major code changes and will raise errors when unsupported Python is used.
In the example below, we forget TorchScript type annotations and we receive
a TorchScript error because the input type for argument y, an int,
does not match with the default argument type, torch.Tensor.
def f2(x, y):
return x + y
inp1 = torch.randn(5, 5)
inp2 = 3
script_f2 = torch.jit.script(f2)
try:
script_f2(inp1, inp2)
except:
tb.print_exc()
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py", line 341, in <module>
script_f2(inp1, inp2)
RuntimeError: f2() Expected a value of type 'Tensor (inferred)' for argument 'y' but instead found type 'int'.
Inferred 'y' to be of type 'Tensor' because it was not annotated with an explicit type.
Position: 1
Value: 3
Declaration: f2(Tensor x, Tensor y) -> Tensor
Cast error details: Unable to cast 3 to Tensor
However, torch.compile is easily able to handle f2.
compile_f2 = torch.compile(f2)
print("compile 2:", test_fns(f2, compile_f2, (inp1, inp2)))
print("~" * 10)
compile 2: True
~~~~~~~~~~
Another case that torch.compile handles well compared to
previous compilers solutions is the usage of non-PyTorch functions.
import scipy
def f3(x):
x = x * 2
x = scipy.fft.dct(x.numpy())
x = torch.from_numpy(x)
x = x * 2
return x
TorchScript tracing treats results from non-PyTorch function calls as constants, and so our results can be silently wrong.
inp1 = torch.randn(5, 5)
inp2 = torch.randn(5, 5)
traced_f3 = torch.jit.trace(f3, (inp1,))
print("traced 3:", test_fns(f3, traced_f3, (inp2,)))
/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py:359: TracerWarning:
Converting a tensor to a NumPy array might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py:360: TracerWarning:
torch.from_numpy results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
traced 3: False
TorchScript scripting and FX tracing disallow non-PyTorch function calls.
try:
torch.jit.script(f3)
except:
tb.print_exc()
try:
torch.fx.symbolic_trace(f3)
except:
tb.print_exc()
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py", line 377, in <module>
torch.jit.script(f3)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/jit/_script.py", line 1341, in script
fn = torch._C._jit_script_compile(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_jit_internal.py", line 1198, in _try_get_dispatched_fn
return boolean_dispatched.get(fn)
File "/opt/conda/envs/py_3.10/lib/python3.10/weakref.py", line 453, in get
return self.data.get(ref(key),default)
TypeError: cannot create weak reference to 'uarray._Function' object
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py", line 382, in <module>
torch.fx.symbolic_trace(f3)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 1109, in symbolic_trace
graph = tracer.trace(root, concrete_args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 778, in trace
(self.create_arg(fn(*args)),),
File "/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py", line 359, in f3
x = scipy.fft.dct(x.numpy())
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/scipy/fft/_backend.py", line 25, in __ua_function__
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/scipy/fft/_pocketfft/realtransforms.py", line 19, in _r2r
tmp = _asfarray(x)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/scipy/fft/_pocketfft/helper.py", line 89, in _asfarray
if x.dtype == np.float16:
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/proxy.py", line 518, in impl
return tracer.create_proxy('call_function', target, args, kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/proxy.py", line 151, in create_proxy
args_ = self.create_arg(args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 373, in create_arg
return super().create_arg(a)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/proxy.py", line 239, in create_arg
return type(a)(self.create_arg(elem) for elem in a)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/proxy.py", line 239, in <genexpr>
return type(a)(self.create_arg(elem) for elem in a)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_symbolic_trace.py", line 373, in create_arg
return super().create_arg(a)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/proxy.py", line 267, in create_arg
raise NotImplementedError(f"argument of type: {type(a)}")
NotImplementedError: argument of type: <class 'type'>
In comparison, torch.compile is easily able to handle
the non-PyTorch function call.
compile_f3 = torch.compile(f3)
print("compile 3:", test_fns(f3, compile_f3, (inp2,)))
compile 3: True
TorchDynamo and FX Graphs¶
One important component of torch.compile is TorchDynamo.
TorchDynamo is responsible for JIT compiling arbitrary Python code into
FX graphs, which can
then be further optimized. TorchDynamo extracts FX graphs by analyzing Python bytecode
during runtime and detecting calls to PyTorch operations.
Normally, TorchInductor, another component of torch.compile,
further compiles the FX graphs into optimized kernels,
but TorchDynamo allows for different backends to be used. In order to inspect
the FX graphs that TorchDynamo outputs, let us create a custom backend that
outputs the FX graph and simply returns the graph’s unoptimized forward method.
from typing import List
def custom_backend(gm: torch.fx.GraphModule, example_inputs: List[torch.Tensor]):
print("custom backend called with FX graph:")
gm.graph.print_tabular()
return gm.forward
# Reset since we are using a different backend.
torch._dynamo.reset()
opt_model = torch.compile(init_model(), backend=custom_backend)
opt_model(generate_data(16)[0])
custom backend called with FX graph:
opcode name target args kwargs
------------- ----------------------------------- ---------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -----------------
placeholder x x () {}
call_module self_features_0 self_features_0 (x,) {}
call_module self_features_1 self_features_1 (self_features_0,) {}
call_module self_features_2 self_features_2 (self_features_1,) {}
call_module self_features_3 self_features_3 (self_features_2,) {}
call_function cat <built-in method cat of type object at 0x7f457ba18540> ([self_features_3], 1) {}
call_module self_features_4_denselayer1_norm1 self_features_4_denselayer1_norm1 (cat,) {}
call_module self_features_4_denselayer1_relu1 self_features_4_denselayer1_relu1 (self_features_4_denselayer1_norm1,) {}
call_module self_features_4_denselayer1_conv1 self_features_4_denselayer1_conv1 (self_features_4_denselayer1_relu1,) {}
call_module self_features_4_denselayer1_norm2 self_features_4_denselayer1_norm2 (self_features_4_denselayer1_conv1,) {}
call_module self_features_4_denselayer1_relu2 self_features_4_denselayer1_relu2 (self_features_4_denselayer1_norm2,) {}
call_module self_features_4_denselayer1_conv2 self_features_4_denselayer1_conv2 (self_features_4_denselayer1_relu2,) {}
call_function cat_1 <built-in method cat of type object at 0x7f457ba18540> ([self_features_3, self_features_4_denselayer1_conv2], 1) {}
call_module self_features_4_denselayer2_norm1 self_features_4_denselayer2_norm1 (cat_1,) {}
call_module self_features_4_denselayer2_relu1 self_features_4_denselayer2_relu1 (self_features_4_denselayer2_norm1,) {}
call_module self_features_4_denselayer2_conv1 self_features_4_denselayer2_conv1 (self_features_4_denselayer2_relu1,) {}
call_module self_features_4_denselayer2_norm2 self_features_4_denselayer2_norm2 (self_features_4_denselayer2_conv1,) {}
call_module self_features_4_denselayer2_relu2 self_features_4_denselayer2_relu2 (self_features_4_denselayer2_norm2,) {}
call_module self_features_4_denselayer2_conv2 self_features_4_denselayer2_conv2 (self_features_4_denselayer2_relu2,) {}
call_function cat_2 <built-in method cat of type object at 0x7f457ba18540> ([self_features_3, self_features_4_denselayer1_conv2, self_features_4_denselayer2_conv2], 1) {}
call_module self_features_4_denselayer3_norm1 self_features_4_denselayer3_norm1 (cat_2,) {}
call_module self_features_4_denselayer3_relu1 self_features_4_denselayer3_relu1 (self_features_4_denselayer3_norm1,) {}
call_module self_features_4_denselayer3_conv1 self_features_4_denselayer3_conv1 (self_features_4_denselayer3_relu1,) {}
call_module self_features_4_denselayer3_norm2 self_features_4_denselayer3_norm2 (self_features_4_denselayer3_conv1,) {}
call_module self_features_4_denselayer3_relu2 self_features_4_denselayer3_relu2 (self_features_4_denselayer3_norm2,) {}
call_module self_features_4_denselayer3_conv2 self_features_4_denselayer3_conv2 (self_features_4_denselayer3_relu2,) {}
call_function cat_3 <built-in method cat of type object at 0x7f457ba18540> ([self_features_3, self_features_4_denselayer1_conv2, self_features_4_denselayer2_conv2, self_features_4_denselayer3_conv2], 1) {}
call_module self_features_4_denselayer4_norm1 self_features_4_denselayer4_norm1 (cat_3,) {}
call_module self_features_4_denselayer4_relu1 self_features_4_denselayer4_relu1 (self_features_4_denselayer4_norm1,) {}
call_module self_features_4_denselayer4_conv1 self_features_4_denselayer4_conv1 (self_features_4_denselayer4_relu1,) {}
call_module self_features_4_denselayer4_norm2 self_features_4_denselayer4_norm2 (self_features_4_denselayer4_conv1,) {}
call_module self_features_4_denselayer4_relu2 self_features_4_denselayer4_relu2 (self_features_4_denselayer4_norm2,) {}
call_module self_features_4_denselayer4_conv2 self_features_4_denselayer4_conv2 (self_features_4_denselayer4_relu2,) {}
call_function cat_4 <built-in method cat of type object at 0x7f457ba18540> ([self_features_3, self_features_4_denselayer1_conv2, self_features_4_denselayer2_conv2, self_features_4_denselayer3_conv2, self_features_4_denselayer4_conv2], 1) {}
call_module self_features_4_denselayer5_norm1 self_features_4_denselayer5_norm1 (cat_4,) {}
call_module self_features_4_denselayer5_relu1 self_features_4_denselayer5_relu1 (self_features_4_denselayer5_norm1,) {}
call_module self_features_4_denselayer5_conv1 self_features_4_denselayer5_conv1 (self_features_4_denselayer5_relu1,) {}
call_module self_features_4_denselayer5_norm2 self_features_4_denselayer5_norm2 (self_features_4_denselayer5_conv1,) {}
call_module self_features_4_denselayer5_relu2 self_features_4_denselayer5_relu2 (self_features_4_denselayer5_norm2,) {}
call_module self_features_4_denselayer5_conv2 self_features_4_denselayer5_conv2 (self_features_4_denselayer5_relu2,) {}
call_function cat_5 <built-in method cat of type object at 0x7f457ba18540> ([self_features_3, self_features_4_denselayer1_conv2, self_features_4_denselayer2_conv2, self_features_4_denselayer3_conv2, self_features_4_denselayer4_conv2, self_features_4_denselayer5_conv2], 1) {}
call_module self_features_4_denselayer6_norm1 self_features_4_denselayer6_norm1 (cat_5,) {}
call_module self_features_4_denselayer6_relu1 self_features_4_denselayer6_relu1 (self_features_4_denselayer6_norm1,) {}
call_module self_features_4_denselayer6_conv1 self_features_4_denselayer6_conv1 (self_features_4_denselayer6_relu1,) {}
call_module self_features_4_denselayer6_norm2 self_features_4_denselayer6_norm2 (self_features_4_denselayer6_conv1,) {}
call_module self_features_4_denselayer6_relu2 self_features_4_denselayer6_relu2 (self_features_4_denselayer6_norm2,) {}
call_module self_features_4_denselayer6_conv2 self_features_4_denselayer6_conv2 (self_features_4_denselayer6_relu2,) {}
call_function cat_6 <built-in method cat of type object at 0x7f457ba18540> ([self_features_3, self_features_4_denselayer1_conv2, self_features_4_denselayer2_conv2, self_features_4_denselayer3_conv2, self_features_4_denselayer4_conv2, self_features_4_denselayer5_conv2, self_features_4_denselayer6_conv2], 1) {}
call_module self_features_5_0 self_features_5_0 (cat_6,) {}
call_module self_features_5_1 self_features_5_1 (self_features_5_0,) {}
call_module self_features_5_2 self_features_5_2 (self_features_5_1,) {}
call_module self_features_5_3 self_features_5_3 (self_features_5_2,) {}
call_function cat_7 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3], 1) {}
call_module self_features_6_denselayer1_norm1 self_features_6_denselayer1_norm1 (cat_7,) {}
call_module self_features_6_denselayer1_relu1 self_features_6_denselayer1_relu1 (self_features_6_denselayer1_norm1,) {}
call_module self_features_6_denselayer1_conv1 self_features_6_denselayer1_conv1 (self_features_6_denselayer1_relu1,) {}
call_module self_features_6_denselayer1_norm2 self_features_6_denselayer1_norm2 (self_features_6_denselayer1_conv1,) {}
call_module self_features_6_denselayer1_relu2 self_features_6_denselayer1_relu2 (self_features_6_denselayer1_norm2,) {}
call_module self_features_6_denselayer1_conv2 self_features_6_denselayer1_conv2 (self_features_6_denselayer1_relu2,) {}
call_function cat_8 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3, self_features_6_denselayer1_conv2], 1) {}
call_module self_features_6_denselayer2_norm1 self_features_6_denselayer2_norm1 (cat_8,) {}
call_module self_features_6_denselayer2_relu1 self_features_6_denselayer2_relu1 (self_features_6_denselayer2_norm1,) {}
call_module self_features_6_denselayer2_conv1 self_features_6_denselayer2_conv1 (self_features_6_denselayer2_relu1,) {}
call_module self_features_6_denselayer2_norm2 self_features_6_denselayer2_norm2 (self_features_6_denselayer2_conv1,) {}
call_module self_features_6_denselayer2_relu2 self_features_6_denselayer2_relu2 (self_features_6_denselayer2_norm2,) {}
call_module self_features_6_denselayer2_conv2 self_features_6_denselayer2_conv2 (self_features_6_denselayer2_relu2,) {}
call_function cat_9 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3, self_features_6_denselayer1_conv2, self_features_6_denselayer2_conv2], 1) {}
call_module self_features_6_denselayer3_norm1 self_features_6_denselayer3_norm1 (cat_9,) {}
call_module self_features_6_denselayer3_relu1 self_features_6_denselayer3_relu1 (self_features_6_denselayer3_norm1,) {}
call_module self_features_6_denselayer3_conv1 self_features_6_denselayer3_conv1 (self_features_6_denselayer3_relu1,) {}
call_module self_features_6_denselayer3_norm2 self_features_6_denselayer3_norm2 (self_features_6_denselayer3_conv1,) {}
call_module self_features_6_denselayer3_relu2 self_features_6_denselayer3_relu2 (self_features_6_denselayer3_norm2,) {}
call_module self_features_6_denselayer3_conv2 self_features_6_denselayer3_conv2 (self_features_6_denselayer3_relu2,) {}
call_function cat_10 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3, self_features_6_denselayer1_conv2, self_features_6_denselayer2_conv2, self_features_6_denselayer3_conv2], 1) {}
call_module self_features_6_denselayer4_norm1 self_features_6_denselayer4_norm1 (cat_10,) {}
call_module self_features_6_denselayer4_relu1 self_features_6_denselayer4_relu1 (self_features_6_denselayer4_norm1,) {}
call_module self_features_6_denselayer4_conv1 self_features_6_denselayer4_conv1 (self_features_6_denselayer4_relu1,) {}
call_module self_features_6_denselayer4_norm2 self_features_6_denselayer4_norm2 (self_features_6_denselayer4_conv1,) {}
call_module self_features_6_denselayer4_relu2 self_features_6_denselayer4_relu2 (self_features_6_denselayer4_norm2,) {}
call_module self_features_6_denselayer4_conv2 self_features_6_denselayer4_conv2 (self_features_6_denselayer4_relu2,) {}
call_function cat_11 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3, self_features_6_denselayer1_conv2, self_features_6_denselayer2_conv2, self_features_6_denselayer3_conv2, self_features_6_denselayer4_conv2], 1) {}
call_module self_features_6_denselayer5_norm1 self_features_6_denselayer5_norm1 (cat_11,) {}
call_module self_features_6_denselayer5_relu1 self_features_6_denselayer5_relu1 (self_features_6_denselayer5_norm1,) {}
call_module self_features_6_denselayer5_conv1 self_features_6_denselayer5_conv1 (self_features_6_denselayer5_relu1,) {}
call_module self_features_6_denselayer5_norm2 self_features_6_denselayer5_norm2 (self_features_6_denselayer5_conv1,) {}
call_module self_features_6_denselayer5_relu2 self_features_6_denselayer5_relu2 (self_features_6_denselayer5_norm2,) {}
call_module self_features_6_denselayer5_conv2 self_features_6_denselayer5_conv2 (self_features_6_denselayer5_relu2,) {}
call_function cat_12 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3, self_features_6_denselayer1_conv2, self_features_6_denselayer2_conv2, self_features_6_denselayer3_conv2, self_features_6_denselayer4_conv2, self_features_6_denselayer5_conv2], 1) {}
call_module self_features_6_denselayer6_norm1 self_features_6_denselayer6_norm1 (cat_12,) {}
call_module self_features_6_denselayer6_relu1 self_features_6_denselayer6_relu1 (self_features_6_denselayer6_norm1,) {}
call_module self_features_6_denselayer6_conv1 self_features_6_denselayer6_conv1 (self_features_6_denselayer6_relu1,) {}
call_module self_features_6_denselayer6_norm2 self_features_6_denselayer6_norm2 (self_features_6_denselayer6_conv1,) {}
call_module self_features_6_denselayer6_relu2 self_features_6_denselayer6_relu2 (self_features_6_denselayer6_norm2,) {}
call_module self_features_6_denselayer6_conv2 self_features_6_denselayer6_conv2 (self_features_6_denselayer6_relu2,) {}
call_function cat_13 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3, self_features_6_denselayer1_conv2, self_features_6_denselayer2_conv2, self_features_6_denselayer3_conv2, self_features_6_denselayer4_conv2, self_features_6_denselayer5_conv2, self_features_6_denselayer6_conv2], 1) {}
call_module self_features_6_denselayer7_norm1 self_features_6_denselayer7_norm1 (cat_13,) {}
call_module self_features_6_denselayer7_relu1 self_features_6_denselayer7_relu1 (self_features_6_denselayer7_norm1,) {}
call_module self_features_6_denselayer7_conv1 self_features_6_denselayer7_conv1 (self_features_6_denselayer7_relu1,) {}
call_module self_features_6_denselayer7_norm2 self_features_6_denselayer7_norm2 (self_features_6_denselayer7_conv1,) {}
call_module self_features_6_denselayer7_relu2 self_features_6_denselayer7_relu2 (self_features_6_denselayer7_norm2,) {}
call_module self_features_6_denselayer7_conv2 self_features_6_denselayer7_conv2 (self_features_6_denselayer7_relu2,) {}
call_function cat_14 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3, self_features_6_denselayer1_conv2, self_features_6_denselayer2_conv2, self_features_6_denselayer3_conv2, self_features_6_denselayer4_conv2, self_features_6_denselayer5_conv2, self_features_6_denselayer6_conv2, self_features_6_denselayer7_conv2], 1) {}
call_module self_features_6_denselayer8_norm1 self_features_6_denselayer8_norm1 (cat_14,) {}
call_module self_features_6_denselayer8_relu1 self_features_6_denselayer8_relu1 (self_features_6_denselayer8_norm1,) {}
call_module self_features_6_denselayer8_conv1 self_features_6_denselayer8_conv1 (self_features_6_denselayer8_relu1,) {}
call_module self_features_6_denselayer8_norm2 self_features_6_denselayer8_norm2 (self_features_6_denselayer8_conv1,) {}
call_module self_features_6_denselayer8_relu2 self_features_6_denselayer8_relu2 (self_features_6_denselayer8_norm2,) {}
call_module self_features_6_denselayer8_conv2 self_features_6_denselayer8_conv2 (self_features_6_denselayer8_relu2,) {}
call_function cat_15 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3, self_features_6_denselayer1_conv2, self_features_6_denselayer2_conv2, self_features_6_denselayer3_conv2, self_features_6_denselayer4_conv2, self_features_6_denselayer5_conv2, self_features_6_denselayer6_conv2, self_features_6_denselayer7_conv2, self_features_6_denselayer8_conv2], 1) {}
call_module self_features_6_denselayer9_norm1 self_features_6_denselayer9_norm1 (cat_15,) {}
call_module self_features_6_denselayer9_relu1 self_features_6_denselayer9_relu1 (self_features_6_denselayer9_norm1,) {}
call_module self_features_6_denselayer9_conv1 self_features_6_denselayer9_conv1 (self_features_6_denselayer9_relu1,) {}
call_module self_features_6_denselayer9_norm2 self_features_6_denselayer9_norm2 (self_features_6_denselayer9_conv1,) {}
call_module self_features_6_denselayer9_relu2 self_features_6_denselayer9_relu2 (self_features_6_denselayer9_norm2,) {}
call_module self_features_6_denselayer9_conv2 self_features_6_denselayer9_conv2 (self_features_6_denselayer9_relu2,) {}
call_function cat_16 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3, self_features_6_denselayer1_conv2, self_features_6_denselayer2_conv2, self_features_6_denselayer3_conv2, self_features_6_denselayer4_conv2, self_features_6_denselayer5_conv2, self_features_6_denselayer6_conv2, self_features_6_denselayer7_conv2, self_features_6_denselayer8_conv2, self_features_6_denselayer9_conv2], 1) {}
call_module self_features_6_denselayer10_norm1 self_features_6_denselayer10_norm1 (cat_16,) {}
call_module self_features_6_denselayer10_relu1 self_features_6_denselayer10_relu1 (self_features_6_denselayer10_norm1,) {}
call_module self_features_6_denselayer10_conv1 self_features_6_denselayer10_conv1 (self_features_6_denselayer10_relu1,) {}
call_module self_features_6_denselayer10_norm2 self_features_6_denselayer10_norm2 (self_features_6_denselayer10_conv1,) {}
call_module self_features_6_denselayer10_relu2 self_features_6_denselayer10_relu2 (self_features_6_denselayer10_norm2,) {}
call_module self_features_6_denselayer10_conv2 self_features_6_denselayer10_conv2 (self_features_6_denselayer10_relu2,) {}
call_function cat_17 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3, self_features_6_denselayer1_conv2, self_features_6_denselayer2_conv2, self_features_6_denselayer3_conv2, self_features_6_denselayer4_conv2, self_features_6_denselayer5_conv2, self_features_6_denselayer6_conv2, self_features_6_denselayer7_conv2, self_features_6_denselayer8_conv2, self_features_6_denselayer9_conv2, self_features_6_denselayer10_conv2], 1) {}
call_module self_features_6_denselayer11_norm1 self_features_6_denselayer11_norm1 (cat_17,) {}
call_module self_features_6_denselayer11_relu1 self_features_6_denselayer11_relu1 (self_features_6_denselayer11_norm1,) {}
call_module self_features_6_denselayer11_conv1 self_features_6_denselayer11_conv1 (self_features_6_denselayer11_relu1,) {}
call_module self_features_6_denselayer11_norm2 self_features_6_denselayer11_norm2 (self_features_6_denselayer11_conv1,) {}
call_module self_features_6_denselayer11_relu2 self_features_6_denselayer11_relu2 (self_features_6_denselayer11_norm2,) {}
call_module self_features_6_denselayer11_conv2 self_features_6_denselayer11_conv2 (self_features_6_denselayer11_relu2,) {}
call_function cat_18 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3, self_features_6_denselayer1_conv2, self_features_6_denselayer2_conv2, self_features_6_denselayer3_conv2, self_features_6_denselayer4_conv2, self_features_6_denselayer5_conv2, self_features_6_denselayer6_conv2, self_features_6_denselayer7_conv2, self_features_6_denselayer8_conv2, self_features_6_denselayer9_conv2, self_features_6_denselayer10_conv2, self_features_6_denselayer11_conv2], 1) {}
call_module self_features_6_denselayer12_norm1 self_features_6_denselayer12_norm1 (cat_18,) {}
call_module self_features_6_denselayer12_relu1 self_features_6_denselayer12_relu1 (self_features_6_denselayer12_norm1,) {}
call_module self_features_6_denselayer12_conv1 self_features_6_denselayer12_conv1 (self_features_6_denselayer12_relu1,) {}
call_module self_features_6_denselayer12_norm2 self_features_6_denselayer12_norm2 (self_features_6_denselayer12_conv1,) {}
call_module self_features_6_denselayer12_relu2 self_features_6_denselayer12_relu2 (self_features_6_denselayer12_norm2,) {}
call_module self_features_6_denselayer12_conv2 self_features_6_denselayer12_conv2 (self_features_6_denselayer12_relu2,) {}
call_function cat_19 <built-in method cat of type object at 0x7f457ba18540> ([self_features_5_3, self_features_6_denselayer1_conv2, self_features_6_denselayer2_conv2, self_features_6_denselayer3_conv2, self_features_6_denselayer4_conv2, self_features_6_denselayer5_conv2, self_features_6_denselayer6_conv2, self_features_6_denselayer7_conv2, self_features_6_denselayer8_conv2, self_features_6_denselayer9_conv2, self_features_6_denselayer10_conv2, self_features_6_denselayer11_conv2, self_features_6_denselayer12_conv2], 1) {}
call_module self_features_7_0 self_features_7_0 (cat_19,) {}
call_module self_features_7_1 self_features_7_1 (self_features_7_0,) {}
call_module self_features_7_2 self_features_7_2 (self_features_7_1,) {}
call_module self_features_7_3 self_features_7_3 (self_features_7_2,) {}
call_function cat_20 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3], 1) {}
call_module self_features_8_denselayer1_norm1 self_features_8_denselayer1_norm1 (cat_20,) {}
call_module self_features_8_denselayer1_relu1 self_features_8_denselayer1_relu1 (self_features_8_denselayer1_norm1,) {}
call_module self_features_8_denselayer1_conv1 self_features_8_denselayer1_conv1 (self_features_8_denselayer1_relu1,) {}
call_module self_features_8_denselayer1_norm2 self_features_8_denselayer1_norm2 (self_features_8_denselayer1_conv1,) {}
call_module self_features_8_denselayer1_relu2 self_features_8_denselayer1_relu2 (self_features_8_denselayer1_norm2,) {}
call_module self_features_8_denselayer1_conv2 self_features_8_denselayer1_conv2 (self_features_8_denselayer1_relu2,) {}
call_function cat_21 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2], 1) {}
call_module self_features_8_denselayer2_norm1 self_features_8_denselayer2_norm1 (cat_21,) {}
call_module self_features_8_denselayer2_relu1 self_features_8_denselayer2_relu1 (self_features_8_denselayer2_norm1,) {}
call_module self_features_8_denselayer2_conv1 self_features_8_denselayer2_conv1 (self_features_8_denselayer2_relu1,) {}
call_module self_features_8_denselayer2_norm2 self_features_8_denselayer2_norm2 (self_features_8_denselayer2_conv1,) {}
call_module self_features_8_denselayer2_relu2 self_features_8_denselayer2_relu2 (self_features_8_denselayer2_norm2,) {}
call_module self_features_8_denselayer2_conv2 self_features_8_denselayer2_conv2 (self_features_8_denselayer2_relu2,) {}
call_function cat_22 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2], 1) {}
call_module self_features_8_denselayer3_norm1 self_features_8_denselayer3_norm1 (cat_22,) {}
call_module self_features_8_denselayer3_relu1 self_features_8_denselayer3_relu1 (self_features_8_denselayer3_norm1,) {}
call_module self_features_8_denselayer3_conv1 self_features_8_denselayer3_conv1 (self_features_8_denselayer3_relu1,) {}
call_module self_features_8_denselayer3_norm2 self_features_8_denselayer3_norm2 (self_features_8_denselayer3_conv1,) {}
call_module self_features_8_denselayer3_relu2 self_features_8_denselayer3_relu2 (self_features_8_denselayer3_norm2,) {}
call_module self_features_8_denselayer3_conv2 self_features_8_denselayer3_conv2 (self_features_8_denselayer3_relu2,) {}
call_function cat_23 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2], 1) {}
call_module self_features_8_denselayer4_norm1 self_features_8_denselayer4_norm1 (cat_23,) {}
call_module self_features_8_denselayer4_relu1 self_features_8_denselayer4_relu1 (self_features_8_denselayer4_norm1,) {}
call_module self_features_8_denselayer4_conv1 self_features_8_denselayer4_conv1 (self_features_8_denselayer4_relu1,) {}
call_module self_features_8_denselayer4_norm2 self_features_8_denselayer4_norm2 (self_features_8_denselayer4_conv1,) {}
call_module self_features_8_denselayer4_relu2 self_features_8_denselayer4_relu2 (self_features_8_denselayer4_norm2,) {}
call_module self_features_8_denselayer4_conv2 self_features_8_denselayer4_conv2 (self_features_8_denselayer4_relu2,) {}
call_function cat_24 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2], 1) {}
call_module self_features_8_denselayer5_norm1 self_features_8_denselayer5_norm1 (cat_24,) {}
call_module self_features_8_denselayer5_relu1 self_features_8_denselayer5_relu1 (self_features_8_denselayer5_norm1,) {}
call_module self_features_8_denselayer5_conv1 self_features_8_denselayer5_conv1 (self_features_8_denselayer5_relu1,) {}
call_module self_features_8_denselayer5_norm2 self_features_8_denselayer5_norm2 (self_features_8_denselayer5_conv1,) {}
call_module self_features_8_denselayer5_relu2 self_features_8_denselayer5_relu2 (self_features_8_denselayer5_norm2,) {}
call_module self_features_8_denselayer5_conv2 self_features_8_denselayer5_conv2 (self_features_8_denselayer5_relu2,) {}
call_function cat_25 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2], 1) {}
call_module self_features_8_denselayer6_norm1 self_features_8_denselayer6_norm1 (cat_25,) {}
call_module self_features_8_denselayer6_relu1 self_features_8_denselayer6_relu1 (self_features_8_denselayer6_norm1,) {}
call_module self_features_8_denselayer6_conv1 self_features_8_denselayer6_conv1 (self_features_8_denselayer6_relu1,) {}
call_module self_features_8_denselayer6_norm2 self_features_8_denselayer6_norm2 (self_features_8_denselayer6_conv1,) {}
call_module self_features_8_denselayer6_relu2 self_features_8_denselayer6_relu2 (self_features_8_denselayer6_norm2,) {}
call_module self_features_8_denselayer6_conv2 self_features_8_denselayer6_conv2 (self_features_8_denselayer6_relu2,) {}
call_function cat_26 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2], 1) {}
call_module self_features_8_denselayer7_norm1 self_features_8_denselayer7_norm1 (cat_26,) {}
call_module self_features_8_denselayer7_relu1 self_features_8_denselayer7_relu1 (self_features_8_denselayer7_norm1,) {}
call_module self_features_8_denselayer7_conv1 self_features_8_denselayer7_conv1 (self_features_8_denselayer7_relu1,) {}
call_module self_features_8_denselayer7_norm2 self_features_8_denselayer7_norm2 (self_features_8_denselayer7_conv1,) {}
call_module self_features_8_denselayer7_relu2 self_features_8_denselayer7_relu2 (self_features_8_denselayer7_norm2,) {}
call_module self_features_8_denselayer7_conv2 self_features_8_denselayer7_conv2 (self_features_8_denselayer7_relu2,) {}
call_function cat_27 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2], 1) {}
call_module self_features_8_denselayer8_norm1 self_features_8_denselayer8_norm1 (cat_27,) {}
call_module self_features_8_denselayer8_relu1 self_features_8_denselayer8_relu1 (self_features_8_denselayer8_norm1,) {}
call_module self_features_8_denselayer8_conv1 self_features_8_denselayer8_conv1 (self_features_8_denselayer8_relu1,) {}
call_module self_features_8_denselayer8_norm2 self_features_8_denselayer8_norm2 (self_features_8_denselayer8_conv1,) {}
call_module self_features_8_denselayer8_relu2 self_features_8_denselayer8_relu2 (self_features_8_denselayer8_norm2,) {}
call_module self_features_8_denselayer8_conv2 self_features_8_denselayer8_conv2 (self_features_8_denselayer8_relu2,) {}
call_function cat_28 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2], 1) {}
call_module self_features_8_denselayer9_norm1 self_features_8_denselayer9_norm1 (cat_28,) {}
call_module self_features_8_denselayer9_relu1 self_features_8_denselayer9_relu1 (self_features_8_denselayer9_norm1,) {}
call_module self_features_8_denselayer9_conv1 self_features_8_denselayer9_conv1 (self_features_8_denselayer9_relu1,) {}
call_module self_features_8_denselayer9_norm2 self_features_8_denselayer9_norm2 (self_features_8_denselayer9_conv1,) {}
call_module self_features_8_denselayer9_relu2 self_features_8_denselayer9_relu2 (self_features_8_denselayer9_norm2,) {}
call_module self_features_8_denselayer9_conv2 self_features_8_denselayer9_conv2 (self_features_8_denselayer9_relu2,) {}
call_function cat_29 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2], 1) {}
call_module self_features_8_denselayer10_norm1 self_features_8_denselayer10_norm1 (cat_29,) {}
call_module self_features_8_denselayer10_relu1 self_features_8_denselayer10_relu1 (self_features_8_denselayer10_norm1,) {}
call_module self_features_8_denselayer10_conv1 self_features_8_denselayer10_conv1 (self_features_8_denselayer10_relu1,) {}
call_module self_features_8_denselayer10_norm2 self_features_8_denselayer10_norm2 (self_features_8_denselayer10_conv1,) {}
call_module self_features_8_denselayer10_relu2 self_features_8_denselayer10_relu2 (self_features_8_denselayer10_norm2,) {}
call_module self_features_8_denselayer10_conv2 self_features_8_denselayer10_conv2 (self_features_8_denselayer10_relu2,) {}
call_function cat_30 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2], 1) {}
call_module self_features_8_denselayer11_norm1 self_features_8_denselayer11_norm1 (cat_30,) {}
call_module self_features_8_denselayer11_relu1 self_features_8_denselayer11_relu1 (self_features_8_denselayer11_norm1,) {}
call_module self_features_8_denselayer11_conv1 self_features_8_denselayer11_conv1 (self_features_8_denselayer11_relu1,) {}
call_module self_features_8_denselayer11_norm2 self_features_8_denselayer11_norm2 (self_features_8_denselayer11_conv1,) {}
call_module self_features_8_denselayer11_relu2 self_features_8_denselayer11_relu2 (self_features_8_denselayer11_norm2,) {}
call_module self_features_8_denselayer11_conv2 self_features_8_denselayer11_conv2 (self_features_8_denselayer11_relu2,) {}
call_function cat_31 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2], 1) {}
call_module self_features_8_denselayer12_norm1 self_features_8_denselayer12_norm1 (cat_31,) {}
call_module self_features_8_denselayer12_relu1 self_features_8_denselayer12_relu1 (self_features_8_denselayer12_norm1,) {}
call_module self_features_8_denselayer12_conv1 self_features_8_denselayer12_conv1 (self_features_8_denselayer12_relu1,) {}
call_module self_features_8_denselayer12_norm2 self_features_8_denselayer12_norm2 (self_features_8_denselayer12_conv1,) {}
call_module self_features_8_denselayer12_relu2 self_features_8_denselayer12_relu2 (self_features_8_denselayer12_norm2,) {}
call_module self_features_8_denselayer12_conv2 self_features_8_denselayer12_conv2 (self_features_8_denselayer12_relu2,) {}
call_function cat_32 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2], 1) {}
call_module self_features_8_denselayer13_norm1 self_features_8_denselayer13_norm1 (cat_32,) {}
call_module self_features_8_denselayer13_relu1 self_features_8_denselayer13_relu1 (self_features_8_denselayer13_norm1,) {}
call_module self_features_8_denselayer13_conv1 self_features_8_denselayer13_conv1 (self_features_8_denselayer13_relu1,) {}
call_module self_features_8_denselayer13_norm2 self_features_8_denselayer13_norm2 (self_features_8_denselayer13_conv1,) {}
call_module self_features_8_denselayer13_relu2 self_features_8_denselayer13_relu2 (self_features_8_denselayer13_norm2,) {}
call_module self_features_8_denselayer13_conv2 self_features_8_denselayer13_conv2 (self_features_8_denselayer13_relu2,) {}
call_function cat_33 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2, self_features_8_denselayer13_conv2], 1) {}
call_module self_features_8_denselayer14_norm1 self_features_8_denselayer14_norm1 (cat_33,) {}
call_module self_features_8_denselayer14_relu1 self_features_8_denselayer14_relu1 (self_features_8_denselayer14_norm1,) {}
call_module self_features_8_denselayer14_conv1 self_features_8_denselayer14_conv1 (self_features_8_denselayer14_relu1,) {}
call_module self_features_8_denselayer14_norm2 self_features_8_denselayer14_norm2 (self_features_8_denselayer14_conv1,) {}
call_module self_features_8_denselayer14_relu2 self_features_8_denselayer14_relu2 (self_features_8_denselayer14_norm2,) {}
call_module self_features_8_denselayer14_conv2 self_features_8_denselayer14_conv2 (self_features_8_denselayer14_relu2,) {}
call_function cat_34 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2, self_features_8_denselayer13_conv2, self_features_8_denselayer14_conv2], 1) {}
call_module self_features_8_denselayer15_norm1 self_features_8_denselayer15_norm1 (cat_34,) {}
call_module self_features_8_denselayer15_relu1 self_features_8_denselayer15_relu1 (self_features_8_denselayer15_norm1,) {}
call_module self_features_8_denselayer15_conv1 self_features_8_denselayer15_conv1 (self_features_8_denselayer15_relu1,) {}
call_module self_features_8_denselayer15_norm2 self_features_8_denselayer15_norm2 (self_features_8_denselayer15_conv1,) {}
call_module self_features_8_denselayer15_relu2 self_features_8_denselayer15_relu2 (self_features_8_denselayer15_norm2,) {}
call_module self_features_8_denselayer15_conv2 self_features_8_denselayer15_conv2 (self_features_8_denselayer15_relu2,) {}
call_function cat_35 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2, self_features_8_denselayer13_conv2, self_features_8_denselayer14_conv2, self_features_8_denselayer15_conv2], 1) {}
call_module self_features_8_denselayer16_norm1 self_features_8_denselayer16_norm1 (cat_35,) {}
call_module self_features_8_denselayer16_relu1 self_features_8_denselayer16_relu1 (self_features_8_denselayer16_norm1,) {}
call_module self_features_8_denselayer16_conv1 self_features_8_denselayer16_conv1 (self_features_8_denselayer16_relu1,) {}
call_module self_features_8_denselayer16_norm2 self_features_8_denselayer16_norm2 (self_features_8_denselayer16_conv1,) {}
call_module self_features_8_denselayer16_relu2 self_features_8_denselayer16_relu2 (self_features_8_denselayer16_norm2,) {}
call_module self_features_8_denselayer16_conv2 self_features_8_denselayer16_conv2 (self_features_8_denselayer16_relu2,) {}
call_function cat_36 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2, self_features_8_denselayer13_conv2, self_features_8_denselayer14_conv2, self_features_8_denselayer15_conv2, self_features_8_denselayer16_conv2], 1) {}
call_module self_features_8_denselayer17_norm1 self_features_8_denselayer17_norm1 (cat_36,) {}
call_module self_features_8_denselayer17_relu1 self_features_8_denselayer17_relu1 (self_features_8_denselayer17_norm1,) {}
call_module self_features_8_denselayer17_conv1 self_features_8_denselayer17_conv1 (self_features_8_denselayer17_relu1,) {}
call_module self_features_8_denselayer17_norm2 self_features_8_denselayer17_norm2 (self_features_8_denselayer17_conv1,) {}
call_module self_features_8_denselayer17_relu2 self_features_8_denselayer17_relu2 (self_features_8_denselayer17_norm2,) {}
call_module self_features_8_denselayer17_conv2 self_features_8_denselayer17_conv2 (self_features_8_denselayer17_relu2,) {}
call_function cat_37 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2, self_features_8_denselayer13_conv2, self_features_8_denselayer14_conv2, self_features_8_denselayer15_conv2, self_features_8_denselayer16_conv2, self_features_8_denselayer17_conv2], 1) {}
call_module self_features_8_denselayer18_norm1 self_features_8_denselayer18_norm1 (cat_37,) {}
call_module self_features_8_denselayer18_relu1 self_features_8_denselayer18_relu1 (self_features_8_denselayer18_norm1,) {}
call_module self_features_8_denselayer18_conv1 self_features_8_denselayer18_conv1 (self_features_8_denselayer18_relu1,) {}
call_module self_features_8_denselayer18_norm2 self_features_8_denselayer18_norm2 (self_features_8_denselayer18_conv1,) {}
call_module self_features_8_denselayer18_relu2 self_features_8_denselayer18_relu2 (self_features_8_denselayer18_norm2,) {}
call_module self_features_8_denselayer18_conv2 self_features_8_denselayer18_conv2 (self_features_8_denselayer18_relu2,) {}
call_function cat_38 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2, self_features_8_denselayer13_conv2, self_features_8_denselayer14_conv2, self_features_8_denselayer15_conv2, self_features_8_denselayer16_conv2, self_features_8_denselayer17_conv2, self_features_8_denselayer18_conv2], 1) {}
call_module self_features_8_denselayer19_norm1 self_features_8_denselayer19_norm1 (cat_38,) {}
call_module self_features_8_denselayer19_relu1 self_features_8_denselayer19_relu1 (self_features_8_denselayer19_norm1,) {}
call_module self_features_8_denselayer19_conv1 self_features_8_denselayer19_conv1 (self_features_8_denselayer19_relu1,) {}
call_module self_features_8_denselayer19_norm2 self_features_8_denselayer19_norm2 (self_features_8_denselayer19_conv1,) {}
call_module self_features_8_denselayer19_relu2 self_features_8_denselayer19_relu2 (self_features_8_denselayer19_norm2,) {}
call_module self_features_8_denselayer19_conv2 self_features_8_denselayer19_conv2 (self_features_8_denselayer19_relu2,) {}
call_function cat_39 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2, self_features_8_denselayer13_conv2, self_features_8_denselayer14_conv2, self_features_8_denselayer15_conv2, self_features_8_denselayer16_conv2, self_features_8_denselayer17_conv2, self_features_8_denselayer18_conv2, self_features_8_denselayer19_conv2], 1) {}
call_module self_features_8_denselayer20_norm1 self_features_8_denselayer20_norm1 (cat_39,) {}
call_module self_features_8_denselayer20_relu1 self_features_8_denselayer20_relu1 (self_features_8_denselayer20_norm1,) {}
call_module self_features_8_denselayer20_conv1 self_features_8_denselayer20_conv1 (self_features_8_denselayer20_relu1,) {}
call_module self_features_8_denselayer20_norm2 self_features_8_denselayer20_norm2 (self_features_8_denselayer20_conv1,) {}
call_module self_features_8_denselayer20_relu2 self_features_8_denselayer20_relu2 (self_features_8_denselayer20_norm2,) {}
call_module self_features_8_denselayer20_conv2 self_features_8_denselayer20_conv2 (self_features_8_denselayer20_relu2,) {}
call_function cat_40 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2, self_features_8_denselayer13_conv2, self_features_8_denselayer14_conv2, self_features_8_denselayer15_conv2, self_features_8_denselayer16_conv2, self_features_8_denselayer17_conv2, self_features_8_denselayer18_conv2, self_features_8_denselayer19_conv2, self_features_8_denselayer20_conv2], 1) {}
call_module self_features_8_denselayer21_norm1 self_features_8_denselayer21_norm1 (cat_40,) {}
call_module self_features_8_denselayer21_relu1 self_features_8_denselayer21_relu1 (self_features_8_denselayer21_norm1,) {}
call_module self_features_8_denselayer21_conv1 self_features_8_denselayer21_conv1 (self_features_8_denselayer21_relu1,) {}
call_module self_features_8_denselayer21_norm2 self_features_8_denselayer21_norm2 (self_features_8_denselayer21_conv1,) {}
call_module self_features_8_denselayer21_relu2 self_features_8_denselayer21_relu2 (self_features_8_denselayer21_norm2,) {}
call_module self_features_8_denselayer21_conv2 self_features_8_denselayer21_conv2 (self_features_8_denselayer21_relu2,) {}
call_function cat_41 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2, self_features_8_denselayer13_conv2, self_features_8_denselayer14_conv2, self_features_8_denselayer15_conv2, self_features_8_denselayer16_conv2, self_features_8_denselayer17_conv2, self_features_8_denselayer18_conv2, self_features_8_denselayer19_conv2, self_features_8_denselayer20_conv2, self_features_8_denselayer21_conv2], 1) {}
call_module self_features_8_denselayer22_norm1 self_features_8_denselayer22_norm1 (cat_41,) {}
call_module self_features_8_denselayer22_relu1 self_features_8_denselayer22_relu1 (self_features_8_denselayer22_norm1,) {}
call_module self_features_8_denselayer22_conv1 self_features_8_denselayer22_conv1 (self_features_8_denselayer22_relu1,) {}
call_module self_features_8_denselayer22_norm2 self_features_8_denselayer22_norm2 (self_features_8_denselayer22_conv1,) {}
call_module self_features_8_denselayer22_relu2 self_features_8_denselayer22_relu2 (self_features_8_denselayer22_norm2,) {}
call_module self_features_8_denselayer22_conv2 self_features_8_denselayer22_conv2 (self_features_8_denselayer22_relu2,) {}
call_function cat_42 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2, self_features_8_denselayer13_conv2, self_features_8_denselayer14_conv2, self_features_8_denselayer15_conv2, self_features_8_denselayer16_conv2, self_features_8_denselayer17_conv2, self_features_8_denselayer18_conv2, self_features_8_denselayer19_conv2, self_features_8_denselayer20_conv2, self_features_8_denselayer21_conv2, self_features_8_denselayer22_conv2], 1) {}
call_module self_features_8_denselayer23_norm1 self_features_8_denselayer23_norm1 (cat_42,) {}
call_module self_features_8_denselayer23_relu1 self_features_8_denselayer23_relu1 (self_features_8_denselayer23_norm1,) {}
call_module self_features_8_denselayer23_conv1 self_features_8_denselayer23_conv1 (self_features_8_denselayer23_relu1,) {}
call_module self_features_8_denselayer23_norm2 self_features_8_denselayer23_norm2 (self_features_8_denselayer23_conv1,) {}
call_module self_features_8_denselayer23_relu2 self_features_8_denselayer23_relu2 (self_features_8_denselayer23_norm2,) {}
call_module self_features_8_denselayer23_conv2 self_features_8_denselayer23_conv2 (self_features_8_denselayer23_relu2,) {}
call_function cat_43 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2, self_features_8_denselayer13_conv2, self_features_8_denselayer14_conv2, self_features_8_denselayer15_conv2, self_features_8_denselayer16_conv2, self_features_8_denselayer17_conv2, self_features_8_denselayer18_conv2, self_features_8_denselayer19_conv2, self_features_8_denselayer20_conv2, self_features_8_denselayer21_conv2, self_features_8_denselayer22_conv2, self_features_8_denselayer23_conv2], 1) {}
call_module self_features_8_denselayer24_norm1 self_features_8_denselayer24_norm1 (cat_43,) {}
call_module self_features_8_denselayer24_relu1 self_features_8_denselayer24_relu1 (self_features_8_denselayer24_norm1,) {}
call_module self_features_8_denselayer24_conv1 self_features_8_denselayer24_conv1 (self_features_8_denselayer24_relu1,) {}
call_module self_features_8_denselayer24_norm2 self_features_8_denselayer24_norm2 (self_features_8_denselayer24_conv1,) {}
call_module self_features_8_denselayer24_relu2 self_features_8_denselayer24_relu2 (self_features_8_denselayer24_norm2,) {}
call_module self_features_8_denselayer24_conv2 self_features_8_denselayer24_conv2 (self_features_8_denselayer24_relu2,) {}
call_function cat_44 <built-in method cat of type object at 0x7f457ba18540> ([self_features_7_3, self_features_8_denselayer1_conv2, self_features_8_denselayer2_conv2, self_features_8_denselayer3_conv2, self_features_8_denselayer4_conv2, self_features_8_denselayer5_conv2, self_features_8_denselayer6_conv2, self_features_8_denselayer7_conv2, self_features_8_denselayer8_conv2, self_features_8_denselayer9_conv2, self_features_8_denselayer10_conv2, self_features_8_denselayer11_conv2, self_features_8_denselayer12_conv2, self_features_8_denselayer13_conv2, self_features_8_denselayer14_conv2, self_features_8_denselayer15_conv2, self_features_8_denselayer16_conv2, self_features_8_denselayer17_conv2, self_features_8_denselayer18_conv2, self_features_8_denselayer19_conv2, self_features_8_denselayer20_conv2, self_features_8_denselayer21_conv2, self_features_8_denselayer22_conv2, self_features_8_denselayer23_conv2, self_features_8_denselayer24_conv2], 1) {}
call_module self_features_9_0 self_features_9_0 (cat_44,) {}
call_module self_features_9_1 self_features_9_1 (self_features_9_0,) {}
call_module self_features_9_2 self_features_9_2 (self_features_9_1,) {}
call_module self_features_9_3 self_features_9_3 (self_features_9_2,) {}
call_function cat_45 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3], 1) {}
call_module self_features_10_denselayer1_norm1 self_features_10_denselayer1_norm1 (cat_45,) {}
call_module self_features_10_denselayer1_relu1 self_features_10_denselayer1_relu1 (self_features_10_denselayer1_norm1,) {}
call_module self_features_10_denselayer1_conv1 self_features_10_denselayer1_conv1 (self_features_10_denselayer1_relu1,) {}
call_module self_features_10_denselayer1_norm2 self_features_10_denselayer1_norm2 (self_features_10_denselayer1_conv1,) {}
call_module self_features_10_denselayer1_relu2 self_features_10_denselayer1_relu2 (self_features_10_denselayer1_norm2,) {}
call_module self_features_10_denselayer1_conv2 self_features_10_denselayer1_conv2 (self_features_10_denselayer1_relu2,) {}
call_function cat_46 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2], 1) {}
call_module self_features_10_denselayer2_norm1 self_features_10_denselayer2_norm1 (cat_46,) {}
call_module self_features_10_denselayer2_relu1 self_features_10_denselayer2_relu1 (self_features_10_denselayer2_norm1,) {}
call_module self_features_10_denselayer2_conv1 self_features_10_denselayer2_conv1 (self_features_10_denselayer2_relu1,) {}
call_module self_features_10_denselayer2_norm2 self_features_10_denselayer2_norm2 (self_features_10_denselayer2_conv1,) {}
call_module self_features_10_denselayer2_relu2 self_features_10_denselayer2_relu2 (self_features_10_denselayer2_norm2,) {}
call_module self_features_10_denselayer2_conv2 self_features_10_denselayer2_conv2 (self_features_10_denselayer2_relu2,) {}
call_function cat_47 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2], 1) {}
call_module self_features_10_denselayer3_norm1 self_features_10_denselayer3_norm1 (cat_47,) {}
call_module self_features_10_denselayer3_relu1 self_features_10_denselayer3_relu1 (self_features_10_denselayer3_norm1,) {}
call_module self_features_10_denselayer3_conv1 self_features_10_denselayer3_conv1 (self_features_10_denselayer3_relu1,) {}
call_module self_features_10_denselayer3_norm2 self_features_10_denselayer3_norm2 (self_features_10_denselayer3_conv1,) {}
call_module self_features_10_denselayer3_relu2 self_features_10_denselayer3_relu2 (self_features_10_denselayer3_norm2,) {}
call_module self_features_10_denselayer3_conv2 self_features_10_denselayer3_conv2 (self_features_10_denselayer3_relu2,) {}
call_function cat_48 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2], 1) {}
call_module self_features_10_denselayer4_norm1 self_features_10_denselayer4_norm1 (cat_48,) {}
call_module self_features_10_denselayer4_relu1 self_features_10_denselayer4_relu1 (self_features_10_denselayer4_norm1,) {}
call_module self_features_10_denselayer4_conv1 self_features_10_denselayer4_conv1 (self_features_10_denselayer4_relu1,) {}
call_module self_features_10_denselayer4_norm2 self_features_10_denselayer4_norm2 (self_features_10_denselayer4_conv1,) {}
call_module self_features_10_denselayer4_relu2 self_features_10_denselayer4_relu2 (self_features_10_denselayer4_norm2,) {}
call_module self_features_10_denselayer4_conv2 self_features_10_denselayer4_conv2 (self_features_10_denselayer4_relu2,) {}
call_function cat_49 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2], 1) {}
call_module self_features_10_denselayer5_norm1 self_features_10_denselayer5_norm1 (cat_49,) {}
call_module self_features_10_denselayer5_relu1 self_features_10_denselayer5_relu1 (self_features_10_denselayer5_norm1,) {}
call_module self_features_10_denselayer5_conv1 self_features_10_denselayer5_conv1 (self_features_10_denselayer5_relu1,) {}
call_module self_features_10_denselayer5_norm2 self_features_10_denselayer5_norm2 (self_features_10_denselayer5_conv1,) {}
call_module self_features_10_denselayer5_relu2 self_features_10_denselayer5_relu2 (self_features_10_denselayer5_norm2,) {}
call_module self_features_10_denselayer5_conv2 self_features_10_denselayer5_conv2 (self_features_10_denselayer5_relu2,) {}
call_function cat_50 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2, self_features_10_denselayer5_conv2], 1) {}
call_module self_features_10_denselayer6_norm1 self_features_10_denselayer6_norm1 (cat_50,) {}
call_module self_features_10_denselayer6_relu1 self_features_10_denselayer6_relu1 (self_features_10_denselayer6_norm1,) {}
call_module self_features_10_denselayer6_conv1 self_features_10_denselayer6_conv1 (self_features_10_denselayer6_relu1,) {}
call_module self_features_10_denselayer6_norm2 self_features_10_denselayer6_norm2 (self_features_10_denselayer6_conv1,) {}
call_module self_features_10_denselayer6_relu2 self_features_10_denselayer6_relu2 (self_features_10_denselayer6_norm2,) {}
call_module self_features_10_denselayer6_conv2 self_features_10_denselayer6_conv2 (self_features_10_denselayer6_relu2,) {}
call_function cat_51 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2, self_features_10_denselayer5_conv2, self_features_10_denselayer6_conv2], 1) {}
call_module self_features_10_denselayer7_norm1 self_features_10_denselayer7_norm1 (cat_51,) {}
call_module self_features_10_denselayer7_relu1 self_features_10_denselayer7_relu1 (self_features_10_denselayer7_norm1,) {}
call_module self_features_10_denselayer7_conv1 self_features_10_denselayer7_conv1 (self_features_10_denselayer7_relu1,) {}
call_module self_features_10_denselayer7_norm2 self_features_10_denselayer7_norm2 (self_features_10_denselayer7_conv1,) {}
call_module self_features_10_denselayer7_relu2 self_features_10_denselayer7_relu2 (self_features_10_denselayer7_norm2,) {}
call_module self_features_10_denselayer7_conv2 self_features_10_denselayer7_conv2 (self_features_10_denselayer7_relu2,) {}
call_function cat_52 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2, self_features_10_denselayer5_conv2, self_features_10_denselayer6_conv2, self_features_10_denselayer7_conv2], 1) {}
call_module self_features_10_denselayer8_norm1 self_features_10_denselayer8_norm1 (cat_52,) {}
call_module self_features_10_denselayer8_relu1 self_features_10_denselayer8_relu1 (self_features_10_denselayer8_norm1,) {}
call_module self_features_10_denselayer8_conv1 self_features_10_denselayer8_conv1 (self_features_10_denselayer8_relu1,) {}
call_module self_features_10_denselayer8_norm2 self_features_10_denselayer8_norm2 (self_features_10_denselayer8_conv1,) {}
call_module self_features_10_denselayer8_relu2 self_features_10_denselayer8_relu2 (self_features_10_denselayer8_norm2,) {}
call_module self_features_10_denselayer8_conv2 self_features_10_denselayer8_conv2 (self_features_10_denselayer8_relu2,) {}
call_function cat_53 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2, self_features_10_denselayer5_conv2, self_features_10_denselayer6_conv2, self_features_10_denselayer7_conv2, self_features_10_denselayer8_conv2], 1) {}
call_module self_features_10_denselayer9_norm1 self_features_10_denselayer9_norm1 (cat_53,) {}
call_module self_features_10_denselayer9_relu1 self_features_10_denselayer9_relu1 (self_features_10_denselayer9_norm1,) {}
call_module self_features_10_denselayer9_conv1 self_features_10_denselayer9_conv1 (self_features_10_denselayer9_relu1,) {}
call_module self_features_10_denselayer9_norm2 self_features_10_denselayer9_norm2 (self_features_10_denselayer9_conv1,) {}
call_module self_features_10_denselayer9_relu2 self_features_10_denselayer9_relu2 (self_features_10_denselayer9_norm2,) {}
call_module self_features_10_denselayer9_conv2 self_features_10_denselayer9_conv2 (self_features_10_denselayer9_relu2,) {}
call_function cat_54 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2, self_features_10_denselayer5_conv2, self_features_10_denselayer6_conv2, self_features_10_denselayer7_conv2, self_features_10_denselayer8_conv2, self_features_10_denselayer9_conv2], 1) {}
call_module self_features_10_denselayer10_norm1 self_features_10_denselayer10_norm1 (cat_54,) {}
call_module self_features_10_denselayer10_relu1 self_features_10_denselayer10_relu1 (self_features_10_denselayer10_norm1,) {}
call_module self_features_10_denselayer10_conv1 self_features_10_denselayer10_conv1 (self_features_10_denselayer10_relu1,) {}
call_module self_features_10_denselayer10_norm2 self_features_10_denselayer10_norm2 (self_features_10_denselayer10_conv1,) {}
call_module self_features_10_denselayer10_relu2 self_features_10_denselayer10_relu2 (self_features_10_denselayer10_norm2,) {}
call_module self_features_10_denselayer10_conv2 self_features_10_denselayer10_conv2 (self_features_10_denselayer10_relu2,) {}
call_function cat_55 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2, self_features_10_denselayer5_conv2, self_features_10_denselayer6_conv2, self_features_10_denselayer7_conv2, self_features_10_denselayer8_conv2, self_features_10_denselayer9_conv2, self_features_10_denselayer10_conv2], 1) {}
call_module self_features_10_denselayer11_norm1 self_features_10_denselayer11_norm1 (cat_55,) {}
call_module self_features_10_denselayer11_relu1 self_features_10_denselayer11_relu1 (self_features_10_denselayer11_norm1,) {}
call_module self_features_10_denselayer11_conv1 self_features_10_denselayer11_conv1 (self_features_10_denselayer11_relu1,) {}
call_module self_features_10_denselayer11_norm2 self_features_10_denselayer11_norm2 (self_features_10_denselayer11_conv1,) {}
call_module self_features_10_denselayer11_relu2 self_features_10_denselayer11_relu2 (self_features_10_denselayer11_norm2,) {}
call_module self_features_10_denselayer11_conv2 self_features_10_denselayer11_conv2 (self_features_10_denselayer11_relu2,) {}
call_function cat_56 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2, self_features_10_denselayer5_conv2, self_features_10_denselayer6_conv2, self_features_10_denselayer7_conv2, self_features_10_denselayer8_conv2, self_features_10_denselayer9_conv2, self_features_10_denselayer10_conv2, self_features_10_denselayer11_conv2], 1) {}
call_module self_features_10_denselayer12_norm1 self_features_10_denselayer12_norm1 (cat_56,) {}
call_module self_features_10_denselayer12_relu1 self_features_10_denselayer12_relu1 (self_features_10_denselayer12_norm1,) {}
call_module self_features_10_denselayer12_conv1 self_features_10_denselayer12_conv1 (self_features_10_denselayer12_relu1,) {}
call_module self_features_10_denselayer12_norm2 self_features_10_denselayer12_norm2 (self_features_10_denselayer12_conv1,) {}
call_module self_features_10_denselayer12_relu2 self_features_10_denselayer12_relu2 (self_features_10_denselayer12_norm2,) {}
call_module self_features_10_denselayer12_conv2 self_features_10_denselayer12_conv2 (self_features_10_denselayer12_relu2,) {}
call_function cat_57 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2, self_features_10_denselayer5_conv2, self_features_10_denselayer6_conv2, self_features_10_denselayer7_conv2, self_features_10_denselayer8_conv2, self_features_10_denselayer9_conv2, self_features_10_denselayer10_conv2, self_features_10_denselayer11_conv2, self_features_10_denselayer12_conv2], 1) {}
call_module self_features_10_denselayer13_norm1 self_features_10_denselayer13_norm1 (cat_57,) {}
call_module self_features_10_denselayer13_relu1 self_features_10_denselayer13_relu1 (self_features_10_denselayer13_norm1,) {}
call_module self_features_10_denselayer13_conv1 self_features_10_denselayer13_conv1 (self_features_10_denselayer13_relu1,) {}
call_module self_features_10_denselayer13_norm2 self_features_10_denselayer13_norm2 (self_features_10_denselayer13_conv1,) {}
call_module self_features_10_denselayer13_relu2 self_features_10_denselayer13_relu2 (self_features_10_denselayer13_norm2,) {}
call_module self_features_10_denselayer13_conv2 self_features_10_denselayer13_conv2 (self_features_10_denselayer13_relu2,) {}
call_function cat_58 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2, self_features_10_denselayer5_conv2, self_features_10_denselayer6_conv2, self_features_10_denselayer7_conv2, self_features_10_denselayer8_conv2, self_features_10_denselayer9_conv2, self_features_10_denselayer10_conv2, self_features_10_denselayer11_conv2, self_features_10_denselayer12_conv2, self_features_10_denselayer13_conv2], 1) {}
call_module self_features_10_denselayer14_norm1 self_features_10_denselayer14_norm1 (cat_58,) {}
call_module self_features_10_denselayer14_relu1 self_features_10_denselayer14_relu1 (self_features_10_denselayer14_norm1,) {}
call_module self_features_10_denselayer14_conv1 self_features_10_denselayer14_conv1 (self_features_10_denselayer14_relu1,) {}
call_module self_features_10_denselayer14_norm2 self_features_10_denselayer14_norm2 (self_features_10_denselayer14_conv1,) {}
call_module self_features_10_denselayer14_relu2 self_features_10_denselayer14_relu2 (self_features_10_denselayer14_norm2,) {}
call_module self_features_10_denselayer14_conv2 self_features_10_denselayer14_conv2 (self_features_10_denselayer14_relu2,) {}
call_function cat_59 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2, self_features_10_denselayer5_conv2, self_features_10_denselayer6_conv2, self_features_10_denselayer7_conv2, self_features_10_denselayer8_conv2, self_features_10_denselayer9_conv2, self_features_10_denselayer10_conv2, self_features_10_denselayer11_conv2, self_features_10_denselayer12_conv2, self_features_10_denselayer13_conv2, self_features_10_denselayer14_conv2], 1) {}
call_module self_features_10_denselayer15_norm1 self_features_10_denselayer15_norm1 (cat_59,) {}
call_module self_features_10_denselayer15_relu1 self_features_10_denselayer15_relu1 (self_features_10_denselayer15_norm1,) {}
call_module self_features_10_denselayer15_conv1 self_features_10_denselayer15_conv1 (self_features_10_denselayer15_relu1,) {}
call_module self_features_10_denselayer15_norm2 self_features_10_denselayer15_norm2 (self_features_10_denselayer15_conv1,) {}
call_module self_features_10_denselayer15_relu2 self_features_10_denselayer15_relu2 (self_features_10_denselayer15_norm2,) {}
call_module self_features_10_denselayer15_conv2 self_features_10_denselayer15_conv2 (self_features_10_denselayer15_relu2,) {}
call_function cat_60 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2, self_features_10_denselayer5_conv2, self_features_10_denselayer6_conv2, self_features_10_denselayer7_conv2, self_features_10_denselayer8_conv2, self_features_10_denselayer9_conv2, self_features_10_denselayer10_conv2, self_features_10_denselayer11_conv2, self_features_10_denselayer12_conv2, self_features_10_denselayer13_conv2, self_features_10_denselayer14_conv2, self_features_10_denselayer15_conv2], 1) {}
call_module self_features_10_denselayer16_norm1 self_features_10_denselayer16_norm1 (cat_60,) {}
call_module self_features_10_denselayer16_relu1 self_features_10_denselayer16_relu1 (self_features_10_denselayer16_norm1,) {}
call_module self_features_10_denselayer16_conv1 self_features_10_denselayer16_conv1 (self_features_10_denselayer16_relu1,) {}
call_module self_features_10_denselayer16_norm2 self_features_10_denselayer16_norm2 (self_features_10_denselayer16_conv1,) {}
call_module self_features_10_denselayer16_relu2 self_features_10_denselayer16_relu2 (self_features_10_denselayer16_norm2,) {}
call_module self_features_10_denselayer16_conv2 self_features_10_denselayer16_conv2 (self_features_10_denselayer16_relu2,) {}
call_function cat_61 <built-in method cat of type object at 0x7f457ba18540> ([self_features_9_3, self_features_10_denselayer1_conv2, self_features_10_denselayer2_conv2, self_features_10_denselayer3_conv2, self_features_10_denselayer4_conv2, self_features_10_denselayer5_conv2, self_features_10_denselayer6_conv2, self_features_10_denselayer7_conv2, self_features_10_denselayer8_conv2, self_features_10_denselayer9_conv2, self_features_10_denselayer10_conv2, self_features_10_denselayer11_conv2, self_features_10_denselayer12_conv2, self_features_10_denselayer13_conv2, self_features_10_denselayer14_conv2, self_features_10_denselayer15_conv2, self_features_10_denselayer16_conv2], 1) {}
call_module self_features_11 self_features_11 (cat_61,) {}
call_function relu <function relu at 0x7f44f0734b80> (self_features_11,) {'inplace': True}
call_function adaptive_avg_pool2d <function adaptive_avg_pool2d at 0x7f44f0734670> (relu, (1, 1)) {}
call_function flatten <built-in method flatten of type object at 0x7f457ba18540> (adaptive_avg_pool2d, 1) {}
call_module self_classifier self_classifier (flatten,) {}
output output output ((self_classifier,),) {}
tensor([[ 0.0617, -0.4023, -0.2796, ..., -0.5545, 0.0970, -0.0639],
[-0.2033, -0.2713, -0.0938, ..., -0.4816, 0.0760, -0.1036],
[ 0.0635, -0.3497, -0.1489, ..., -0.4842, 0.1774, -0.0726],
...,
[-0.1053, -0.3395, 0.0095, ..., -0.4864, 0.0555, -0.1056],
[ 0.0018, -0.2435, -0.1660, ..., -0.5070, 0.0975, -0.1385],
[ 0.1196, -0.3560, -0.1148, ..., -0.4838, 0.1769, -0.0659]],
device='cuda:0', grad_fn=<AddmmBackward0>)
Using our custom backend, we can now see how TorchDynamo is able to handle
data-dependent control flow. Consider the function below, where the line
if b.sum() < 0 is the source of data-dependent control flow.
def bar(a, b):
x = a / (torch.abs(a) + 1)
if b.sum() < 0:
b = b * -1
return x * b
opt_bar = torch.compile(bar, backend=custom_backend)
inp1 = torch.randn(10)
inp2 = torch.randn(10)
opt_bar(inp1, inp2)
opt_bar(inp1, -inp2)
custom backend called with FX graph:
opcode name target args kwargs
------------- ------- ------------------------------------------------------ ---------------- --------
placeholder a a () {}
placeholder b b () {}
call_function abs_1 <built-in method abs of type object at 0x7f457ba18540> (a,) {}
call_function add <built-in function add> (abs_1, 1) {}
call_function truediv <built-in function truediv> (a, add) {}
call_method sum_1 sum (b,) {}
call_function lt <built-in function lt> (sum_1, 0) {}
output output output ((truediv, lt),) {}
custom backend called with FX graph:
opcode name target args kwargs
------------- ------ ----------------------- --------- --------
placeholder b b () {}
placeholder x x () {}
call_function mul <built-in function mul> (x, b) {}
output output output ((mul,),) {}
custom backend called with FX graph:
opcode name target args kwargs
------------- ------ ----------------------- ----------- --------
placeholder b b () {}
placeholder x x () {}
call_function mul <built-in function mul> (b, -1) {}
call_function mul_1 <built-in function mul> (x, mul) {}
output output output ((mul_1,),) {}
tensor([-0.0176, 1.0753, 0.0282, 0.0756, -0.0176, 0.0633, -0.9161, 0.1333,
-0.1971, -0.3406])
The output reveals that TorchDynamo extracted 3 different FX graphs corresponding the following code (order may differ from the output above):
x = a / (torch.abs(a) + 1)b = b * -1; return x * breturn x * b
When TorchDynamo encounters unsupported Python features, such as data-dependent control flow, it breaks the computation graph, lets the default Python interpreter handle the unsupported code, then resumes capturing the graph.
Let’s investigate by example how TorchDynamo would step through bar.
If b.sum() < 0, then TorchDynamo would run graph 1, let
Python determine the result of the conditional, then run
graph 2. On the other hand, if not b.sum() < 0, then TorchDynamo
would run graph 1, let Python determine the result of the conditional, then
run graph 3.
This highlights a major difference between TorchDynamo and previous PyTorch compiler solutions. When encountering unsupported Python features, previous solutions either raise an error or silently fail. TorchDynamo, on the other hand, will break the computation graph.
We can see where TorchDynamo breaks the graph by using torch._dynamo.explain:
# Reset since we are using a different backend.
torch._dynamo.reset()
explanation, out_guards, graphs, ops_per_graph, break_reasons, explanation_verbose = torch._dynamo.explain(
bar, torch.randn(10), torch.randn(10)
)
print(explanation_verbose)
Dynamo produced 2 graphs with 1 graph break and 6 ops
Break reasons:
1. generic_jump TensorVariable()
File "/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py", line 428, in bar
if b.sum() < 0:
2. return_value
File "/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py", line 430, in <graph break in bar>
return x * b
TorchDynamo compilation metrics:
Function Runtimes (s)
------------------------------ --------------
_compile 0.0083, 0.0036
OutputGraph.call_user_compiler 0.0000, 0.0000
In order to maximize speedup, graph breaks should be limited.
We can force TorchDynamo to raise an error upon the first graph
break encountered by using fullgraph=True:
opt_bar = torch.compile(bar, fullgraph=True)
try:
opt_bar(torch.randn(10), torch.randn(10))
except:
tb.print_exc()
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py", line 478, in <module>
opt_bar(torch.randn(10), torch.randn(10))
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 337, in catch_errors
return callback(frame, cache_size, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
return _compile(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
out_code = transform_code_object(code, transform)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 445, in transform_code_object
transformations(instructions, code_options)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
tracer.run()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1726, in run
super().run()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 576, in run
and self.step()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 540, in step
getattr(self, inst.opname)(inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 327, in inner
unimplemented(f"generic_jump {typestr(value)}")
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 71, in unimplemented
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: generic_jump TensorVariable()
from user code:
File "/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py", line 428, in bar
if b.sum() < 0:
Set torch._dynamo.config.verbose=True for more information
You can suppress this exception and fall back to eager by setting:
torch._dynamo.config.suppress_errors = True
And below, we demonstrate that TorchDynamo does not break the graph on the model we used above for demonstrating speedups.
opt_model = torch.compile(init_model(), fullgraph=True)
print(opt_model(generate_data(16)[0]))
tensor([[ 0.1344, 0.1985, -0.2056, ..., 0.1870, -0.1624, -0.1645],
[ 0.2678, 0.2351, -0.1903, ..., 0.2161, -0.0053, 0.0306],
[ 0.0162, 0.2179, -0.1113, ..., 0.1712, -0.1681, -0.0637],
...,
[ 0.0808, 0.2685, -0.1883, ..., 0.0379, -0.2079, -0.1445],
[-0.0210, 0.0856, -0.2461, ..., 0.1860, -0.1278, -0.0285],
[-0.0386, 0.0725, -0.1967, ..., 0.0863, -0.2202, -0.1483]],
device='cuda:0', grad_fn=<CompiledFunctionBackward>)
Finally, if we simply want TorchDynamo to output the FX graph for export,
we can use torch._dynamo.export. Note that torch._dynamo.export, like
fullgraph=True, raises an error if TorchDynamo breaks the graph.
try:
torch._dynamo.export(bar, torch.randn(10), torch.randn(10))
except:
tb.print_exc()
model_exp = torch._dynamo.export(init_model(), generate_data(16)[0])
print(model_exp[0](generate_data(16)[0]))
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py", line 495, in <module>
torch._dynamo.export(bar, torch.randn(10), torch.randn(10))
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 601, in export
result_traced = opt_f(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 337, in catch_errors
return callback(frame, cache_size, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
return _compile(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
out_code = transform_code_object(code, transform)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 445, in transform_code_object
transformations(instructions, code_options)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
tracer.run()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1726, in run
super().run()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 576, in run
and self.step()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 540, in step
getattr(self, inst.opname)(inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 327, in inner
unimplemented(f"generic_jump {typestr(value)}")
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 71, in unimplemented
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: generic_jump TensorVariable()
from user code:
File "/var/lib/jenkins/workspace/intermediate_source/torch_compile_tutorial.py", line 428, in bar
if b.sum() < 0:
Set torch._dynamo.config.verbose=True for more information
You can suppress this exception and fall back to eager by setting:
torch._dynamo.config.suppress_errors = True
tensor([[ 0.4993, 0.0861, -0.2805, ..., 0.2436, 0.0687, -0.4413],
[ 0.4826, -0.0365, -0.2340, ..., 0.3676, 0.0351, -0.3603],
[ 0.2845, 0.0281, -0.1829, ..., 0.3191, 0.1543, -0.4259],
...,
[ 0.5315, 0.1066, -0.2218, ..., 0.2003, 0.0758, -0.3831],
[ 0.4143, 0.0588, -0.2935, ..., 0.1921, 0.0568, -0.3246],
[ 0.4940, 0.0450, -0.2889, ..., 0.2579, 0.0710, -0.4879]],
device='cuda:0', grad_fn=<AddmmBackward0>)
Conclusion¶
In this tutorial, we introduced torch.compile by covering
basic usage, demonstrating speedups over eager mode, comparing to previous
PyTorch compiler solutions, and briefly investigating TorchDynamo and its interactions
with FX graphs. We hope that you will give torch.compile a try!
Total running time of the script: ( 2 minutes 32.516 seconds)
