# Neuropod Tutorial¶

In this tutorial, we’re going to build a simple Neuropod model for addition in TensorFlow, PyTorch, and TorchScript. We'll also show how to run inference from Python and C++.

Almost all of the examples/code in this tutorial come from the Neuropod unit and integration tests. Please read through them for complete working examples.

The Neuropod packaging and inference interfaces also have comprehensive docstrings and provide a more detailed usage of the API than this tutorial.

Make sure to follow the instructions for installing Neuropod before continuing

## Package a Model¶

The first step for packaging a model is to define a “problem” (e.g. 2d object detection).

A “problem” is composed of 4 things:

- an
`input_spec`

- A list of dicts specifying the name, datatype, and shape of an input tensor

- an
`output_spec`

- A list of dicts specifying the name, datatype, and shape of an output tensor

`test_input_data`

*(optional)*- If provided, Neuropod will run inference immediately after packaging to verify that the model was packaged correctly. Must be provided if
`test_output_data`

is provided

- If provided, Neuropod will run inference immediately after packaging to verify that the model was packaged correctly. Must be provided if
`test_output_data`

*(optional)*- If provided, Neuropod will test that the output of inference with
`test_input_data`

matches`test_output_data`

- If provided, Neuropod will test that the output of inference with

The shape of a tensor can include `None`

in which case any value is acceptable. You can also use “symbols” in these shape definitions. Every instance of that symbol must resolve to the same value at runtime.

For example, here’s a problem definition for our addition model:

INPUT_SPEC = [ # A one dimensional tensor of any size with dtype float32 {"name": "x", "dtype": "float32", "shape": ("num_inputs",)}, # A one dimensional tensor of the same size with dtype float32 {"name": "y", "dtype": "float32", "shape": ("num_inputs",)}, ] OUTPUT_SPEC = [ # The sum of the two tensors {"name": "out", "dtype": "float32", "shape": (None,)}, ] TEST_INPUT_DATA = { "x": np.arange(5, dtype=np.float32), "y": np.arange(5, dtype=np.float32), } TEST_EXPECTED_OUT = { "out": np.arange(5) + np.arange(5) }

The symbol `num_inputs`

in the shapes of `x`

and `y`

must resolve to the same value at runtime.

For a definition of a “real” problem, see the example problem definitions section in the appendix.

Now that we have a problem defined, we’re going to see how to package a model in each of the currently supported DL frameworks.

### TensorFlow¶

There are two ways to package a TensorFlow model. One is with a `GraphDef`

the other is with a path to a frozen graph. Both of these require a `node_name_mapping`

that maps a tensor name in the problem definition (see above) to a node in the TensorFlow graph. See the examples below for more detail.

#### GraphDef¶

If we have a function that returns a `GraphDef`

like:

import tensorflow as tf def create_tf_addition_model(): """ A simple addition model """ g = tf.Graph() with g.as_default(): with tf.name_scope("some_namespace"): x = tf.placeholder(tf.float32, name="in_x") y = tf.placeholder(tf.float32, name="in_y") out = tf.add(x, y, name="out") return g.as_graph_def()

we can package the model as follows:

from neuropod.packagers import create_tensorflow_neuropod create_tensorflow_neuropod( neuropod_path=neuropod_path, model_name="addition_model", graph_def=create_tf_addition_model(), node_name_mapping={ "x": "some_namespace/in_x:0", "y": "some_namespace/in_y:0", "out": "some_namespace/out:0", }, input_spec=addition_problem_definition.INPUT_SPEC, output_spec=addition_problem_definition.OUTPUT_SPEC, test_input_data=addition_problem_definition.TEST_INPUT_DATA, test_expected_out=addition_problem_definition.TEST_EXPECTED_OUT, )

Note

`create_tensorflow_neuropod`

runs inference with the test data immediately after creating the neuropod. Raises a `ValueError`

if the model output does not match the expected output.

#### Path to a Frozen Graph¶

If you already have a frozen graph, you can package the model like this:

from neuropod.packagers import create_tensorflow_neuropod create_tensorflow_neuropod( neuropod_path=neuropod_path, model_name="addition_model", frozen_graph_path="/path/to/my/frozen.graph", node_name_mapping={ "x": "some_namespace/in_x:0", "y": "some_namespace/in_y:0", "out": "some_namespace/out:0", }, input_spec=addition_problem_definition.INPUT_SPEC, output_spec=addition_problem_definition.OUTPUT_SPEC, test_input_data=addition_problem_definition.TEST_INPUT_DATA, test_expected_out=addition_problem_definition.TEST_EXPECTED_OUT, )

Note

`create_tensorflow_neuropod`

runs inference with the test data immediately after creating the neuropod. Raises a `ValueError`

if the model output does not match the expected output.

### PyTorch¶

Tip

Packaging a PyTorch model is a bit more complicated because you need python code and the weights in order to run the network.

Converting your model to TorchScript is recommended if possible.

In order to create a PyTorch neuropod package, we need to follow a few guidelines:

- Absolute imports (e.g.
`import torch`

) are okay as long as your runtime environment has the package installed - For Python 3, all other imports within your package must be relative

This type of neuropod package is a bit less flexible than TensorFlow/TorchScript/Keras packages because absolute imports introduce a dependency on the runtime environment. This will be improved in a future release.

Let's say our addition model looks like this (and is stored at `/my/model/code/dir/main.py`

):

import torch import torch.nn as nn class AdditionModel(nn.Module): def forward(self, x, y): return { "out": x + y } def get_model(data_root): return AdditionModel()

In order to package it, we need 4 things:

- The paths to any data we want to store (e.g. the model weights)
- The path to the
`python_root`

of the code along with relative paths for any dirs within the`python_root`

we want to package - An entrypoint function that returns a model given a path to the packaged data. See the docstring for
`create_pytorch_neuropod`

for more details and examples. - The dependencies of our model. These should be python packages.

Tip

See the API docs for `create_pytorch_neuropod`

for detailed descriptions of every parameter

For our model:

- We don't need to store any data (because our model has no weights)
- Our python root is
`/my/model/code/dir`

and we want to store all the code in it - Our entrypoint function is
`get_model`

and our entrypoint_package is`main`

(because the code is in`main.py`

in the python root)

This translates to the following:

from neuropod.packagers import create_pytorch_neuropod create_pytorch_neuropod( neuropod_path=neuropod_path, model_name="addition_model", data_paths=[], code_path_spec=[{ "python_root": '/my/model/code/dir', "dirs_to_package": [ "" # Package everything in the python_root ], }], entrypoint_package="main", entrypoint="get_model", input_spec=addition_problem_definition.INPUT_SPEC, output_spec=addition_problem_definition.OUTPUT_SPEC, test_input_data=addition_problem_definition.TEST_INPUT_DATA, test_expected_out=addition_problem_definition.TEST_EXPECTED_OUT, )

Note

`create_pytorch_neuropod`

runs inference with the test data immediately after creating the neuropod. Raises a `ValueError`

if the model output does not match the expected output.

### TorchScript¶

TorchScript is much easier to package than PyTorch (since we don't need to store any python code).

If we have an addition model that looks like:

import torch class AdditionModel(torch.jit.ScriptModule): """ A simple addition model """ @torch.jit.script_method def forward(self, x, y): return { "out": x + y }

We can package it by running:

from neuropod.packagers import create_torchscript_neuropod create_torchscript_neuropod( neuropod_path=neuropod_path, model_name="addition_model", module=AdditionModel(), input_spec=addition_problem_definition.INPUT_SPEC, output_spec=addition_problem_definition.OUTPUT_SPEC, test_input_data=addition_problem_definition.TEST_INPUT_DATA, test_expected_out=addition_problem_definition.TEST_EXPECTED_OUT, )

Note

`create_torchscript_neuropod`

runs inference with the test data immediately after creating the neuropod. Raises a `ValueError`

if the model output does not match the expected output.

### Keras¶

If we have a Keras addition model that looks like:

def create_keras_addition_model(): """ A simple addition model """ x = Input(batch_shape=(None,), name="x") y = Input(batch_shape=(None,), name="y") out = Add(name="out")([x, y]) model = Model(inputs=[x, y], outputs=[out]) return model

We can package it by running:

from neuropod.packagers import create_keras_neuropod create_keras_neuropod( neuropod_path=neuropod_path, model_name="addition_model", sess=tf.keras.backend.get_session(), model=create_keras_addition_model(), input_spec=addition_problem_definition.INPUT_SPEC, output_spec=addition_problem_definition.OUTPUT_SPEC, test_input_data=addition_problem_definition.TEST_INPUT_DATA, test_expected_out=addition_problem_definition.TEST_EXPECTED_OUT, )

Note

`create_keras_neuropod`

runs inference with the test data immediately after creating the neuropod. Raises a `ValueError`

if the model output does not match the expected output.

### Python¶

Packaging aribtrary Python code has the same interface as packaging PyTorch above.

For an example, see the PyTorch section above and use `create_python_neuropod`

instead of `create_pytorch_neuropod`

## Run Inference¶

Inference is the exact same no matter what the underlying DL framework is

### From Python¶

x = np.array([1, 2, 3, 4]) y = np.array([5, 6, 7, 8]) with load_neuropod(ADDITION_MODEL_PATH) as neuropod: results = neuropod.infer({"x": x, "y": y}) # array([6, 8, 10, 12]) print results["out"]

### From C++¶

#include "neuropod/neuropod.hh" int main() { const std::vector<int64_t> shape = {4}; // To show two different ways of adding data, one of our inputs is an array // and the other is a vector. const float[] x_data = {1, 2, 3, 4}; const std::vector<float> y_data = {5, 6, 7, 8}; // Load the neuropod Neuropod neuropod(ADDITION_MODEL_PATH); // Add the input data using two different signatures of `copy_from` // (one with a pointer and size, one with a vector) auto x_tensor = neuropod.allocate_tensor<float>(shape); x_tensor->copy_from(x_data, 4); auto y_tensor = neuropod.allocate_tensor<float>(shape); y_tensor->copy_from(y_data); // Run inference const auto output_data = neuropod.infer({ {"x", x_tensor}, {"y", y_tensor} }); const auto out_tensor = output_data->at("out"); // {6, 8, 10, 12} const auto out_vector = out_tensor->as_typed_tensor<float>()->get_data_as_vector(); // {4} const auto out_shape = out_tensor->get_dims(); }

Note

This shows basic usage of the C++ API. For more flexible and memory-efficient usage, please see the C++ API docs

## Appendix¶

### Example Problem Definitions¶

The problem definition for 2d object detection may look something like this:

INPUT_SPEC = [ # BGR image {"name": "image", "dtype": "uint8", "shape": (1200, 1920, 3)}, ] OUTPUT_SPEC = [ # shape: (num_detections, 4): (xmin, ymin, xmax, ymax) # These values are in units of pixels. The origin is the top left corner # with positive X to the right and positive Y towards the bottom of the image {"name": "boxes", "dtype": "float32", "shape": ("num_detections", 4)}, # The list of classes that the network can output # This must be some subset of ['vehicle', 'person', 'motorcycle', 'bicycle'] {"name": "supported_object_classes", "dtype": "string", "shape": ("num_classes",)}, # The probability of each class for each detection # These should all be floats between 0 and 1 {"name": "object_class_probability", "dtype": "float32", "shape": ("num_detections", "num_classes")}, ]