Add a new inference framework¶

Introduction¶

Inference is a multi-framework inference sub-module of nndeploy, which abstracts the differences among various inference frameworks and provides a unified interface experience. Currently, nndeploy has already supported TensorRT, OpenVINO, ONNXRuntime, MNN, TNN, ncnn, coreML, paddle-lite, AscendCL, RKNN, and other inference frameworks.

Steps¶

Adding a new inference framework mainly involves the following five steps:

(1) Add inference framework related class types
(2) Inherit the base class InferenceParam
(3) Inherit the base class Inference
(4) Write Converter
(5) Modify cmake

Step one: Add new device type class types¶

1.1 Add ModelType class types¶

(1) Modify the file \include\nndeploy\base\common.h, add new model format class types in ModelType, format kModelTypeXxx
(2) Modify the file \source\nndeploy\base\common.cc, add string conversion to new model format class types in the function stringToModelType(const std::string &src)

1.2 Add InferenceType class types¶

(1) Modify the file \include\nndeploy\base\common.h, add new inference framework format class types in InferenceType, format kInferenceTypeXxx
(2) Modify the file \source\nndeploy\base\common.cc, add string conversion to new inference framework format in the function stringToInferenceType(const std::string &src)

1.3 Add error class types¶

(1) Modify the file \include\nndeploy\base\status.h, add new error class types in StatusCode, format kStatusCodeErrorInferenceXxx

Step two: Inherit the base class InferenceParam¶

(1) In the \include\nndeploy\inference\newxxx\xxx_inference_param.h file, you can refer to MNN(\include/nndeploy/inference/mnn/mnn_inference_param.h) or TensorRT(\include/nndeploy/inference/tensorrt/tensorrt_inference_param.h)
(2) In the \source\nndeploy\inference\newxxx\xxx_inference_param.cc file, you can refer to MNN(\source/nndeploy/inference/mnn/mnn_inference_param.c) or TensorRT(\include/nndeploy/inference/tensorrt/tensorrt_inference_param.cc)

Step three: Inherit the base class Inference¶

(1) In the \include\nndeploy\inference\newxxx\xxx_inference.h file, you can refer to MNN(\include/nndeploy/inference/mnn/mnn_inference.h) or TensorRT(\include/nndeploy/inference/tensorrt/tensorrt_inference.h)
(2) In the \source\nndeploy\inference\newxxx\xxx_inference.cc file, you can refer to MNN(\source/nndeploy/inference/mnn/mnn_inference.c) or TensorRT(\include/nndeploy/inference/tensorrt/tensorrt_inference.cc)

Step four: Write Converter¶

nndeploy provides a unified Tensor and the hyperparameter data structure required by the inference, each inference framework has its own defined Tensor and hyperparameter data structure, in order to ensure the unified interface experience of the call, it is necessary to write the conversion module.

In the file <path>\include\nndeploy\inference\new_xx\xxx_converter.h, you can refer to MNN(<path>\include/nndeploy/inference/mnn/mnn_converter.h) or TensorRT(<path>\include/nndeploy/inference/tensorrt/tensorrt_converter.h).
In the file <path>\source\nndeploy\inference\new_xx\xxx_inference.cc, you can refer to MNN(<path>\source/nndeploy/inference/mnn/mnn_converter.c) or TensorRT(<path>\include/nndeploy/inference/tensorrt/tensorrt_converter.cc).

Step Five: Modify CMake¶

Modify the CMakeList file <path>\CMakeLists.txt,
- Add the inference framework translation option nndeploy_option(ENABLE_NNDEPLOY_INFERENCE_XXX “ENABLE_NNDEPLOY_INFERENCE_XXX” OFF).
- Due to the addition of the new device, source files and header files have been added, which need to be included in the translation file. It is necessary to add the following CMake source code within the if(ENABLE_NNDEPLOY_INFERENCE) endif() code block.
```
if (ENABLE_NNDEPLOY_INFERENCE_XXX)
  file(GLOB_RECURSE INFERENCE_XXX_SOURCE
    "${ROOT_PATH}/include/nndeploy/inference/xxx/*.h"
    "${ROOT_PATH}/source/nndeploy/inference/xxx/*.cc"
  )
  set(INFERENCE_SOURCE ${INFERENCE_SOURCE} ${INFERENCE_XXX_SOURCE})
endif()
```
Link the three libraries of the inference framework.
- You need to create a new xxx.cmake under the <path>\cmake directory, similar to <path>\cmake\mnn.cmake or <path>\cmake\xxx.cmake.
- Modify <path>\cmake\nndeploy.cmake, add include(”${ROOT_PATH}/cmake/xxx.cmake”).
Modify <path>\build\config.cmake, add the device translation option set(ENABLE_NNDEPLOY_INFERENCE_XXX ON).