Quickstart

Packaging a model for torch::deploy

torch::deploy can load and run Python models that are packaged with torch.package. You can find torch.package’s documentation here.

For now, let’s create a simple model that we can load and run in torch::deploy.

import torchvision
from torch.package import PackageExporter

# Instantiate some model
model = torchvision.models.resnet.resnet18()

# Package and export it.
with PackageExporter("my_package.pt") as e:
    e.intern("torchvision.**")
    e.extern("numpy.**")
    e.extern("sys")
    e.extern("PIL.*")
    e.extern("typing_extensions")
    e.save_pickle("model", "model.pkl", model)

Note that since “numpy”, “sys”, “PIL” were marked as “extern”, torch.package will look for these dependencies on the system that loads this package. They will not be packaged with the model.

Now, there should be a file named my_package.pt in your working directory.

Load the model in C++

#include <multipy/runtime/deploy.h>
#include <multipy/runtime/path_environment.h>
#include <torch/script.h>
#include <torch/torch.h>

#include <iostream>
#include <memory>

int main(int argc, const char* argv[]) {
  if (argc != 2) {
    std::cerr << "usage: example-app <path-to-exported-script-module>\n";
    return -1;
  }

  // Start an interpreter manager governing 4 embedded interpreters.
  torch::deploy::InterpreterManager manager(4);
  torch::deploy::ReplicatedObj model;
  try {
    // Load the model from the multipy.package.
    torch::deploy::Package package = manager.loadPackage(argv[1]);
    model = package.loadPickle("model", "model.pkl");
  } catch (const c10::Error& e) {
    std::cerr << "error loading the model\n";
    std::cerr << e.msg();
    return -1;
  }

  // Create a vector of inputs.
  std::vector<torch::jit::IValue> inputs;
  inputs.push_back(torch::ones({1, 3, 224, 224}));

  // Execute the model and turn its output into a tensor.
  at::Tensor output = model(inputs).toTensor();
  std::cout << output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) << '\n';

  std::cout << "ok\n";
}

This small program introduces many of the core concepts of torch::deploy.

An InterpreterManager abstracts over a collection of independent Python interpreters, allowing you to load balance across them when running your code.

Using the InterpreterManager::loadPackage method, you can load a torch.package from disk and make it available to all interpreters.

Package::loadPickle allows you to retrieve specific Python objects from the package, like the ResNet model we saved earlier.

Finally, the model itself is a ReplicatedObj. This is an abstract handle to an object that is replicated across multiple interpreters. When you interact with a ReplicatedObj (for example, by calling forward), it will select an free interpreter to execute that interaction.

Build and execute the C++ example

Assuming the above C++ program was stored in a file called example-app.cpp, a minimal CMakeLists.txt file would look like:

cmake_minimum_required(VERSION 3.12 FATAL_ERROR)
project(multipy_tutorial)

set(MULTIPY_PATH ".." CACHE PATH "The repo where multipy is built or the PYTHONPATH")

# include the multipy utils to help link against
include(${MULTIPY_PATH}/multipy/runtime/utils.cmake)

# add headers from multipy
include_directories(${MULTIPY_PATH})

# link the multipy prebuilt binary
add_library(multipy_internal STATIC IMPORTED)
set_target_properties(multipy_internal
    PROPERTIES
    IMPORTED_LOCATION
    ${MULTIPY_PATH}/multipy/runtime/build/libtorch_deploy.a)
caffe2_interface_library(multipy_internal multipy)

# build our examples
add_executable(quickstart quickstart/quickstart.cpp)
target_link_libraries(quickstart PUBLIC "-Wl,--no-as-needed -rdynamic" dl pthread util multipy c10 torch_cpu)

The -rdynamic and --no-as-needed flags are needed when linking to the executable to ensure that symbols are exported to the dynamic table, making them accessible to the torch::deploy interpreters (which are dynamically loaded).

The last step is configuring and building the project. Assuming that our code directory is laid out like this:

example-app/
    CMakeLists.txt
    quickstart.cpp

We can now run the following commands to build the application from within the example-app/ folder:

cmake -S . -B build -DMULTIPY_PATH="/home/user/repos/multipy" # the parent directory of multipy (i.e. the git repo)
cmake --build build --config Release -j

Now we can run our app:

./example-app /path/to/my_package.pt