nndeploy - An End-to-End Model Deployment Framework

1 Requirements Analysis

First is the requirements analysis, which is why we need to do nndeploy, what are the actual scenarios for multi-end model deployment, and what are the current pain points in model multi-end deployment and model deployment.

1.1 Multi-End Deployment Actual Cases

This is an actual case of AI intelligent matting multi-end deployment, through a portrait segmentation model, extracting the Mona Lisa batch from the image, using a certain domestic image software, which has versions for iOS, Android, web, and computer (Win, Mac, Linux), etc. This case illustrates the actual need for multi-end deployment of models.

1.2 Model Multi-End Deployment and Model Deployment Pain Points

1.2.1 Fragmentation of Inference Frameworks

The first pain point of model multi-end deployment - the fragmentation of inference frameworks. Currently, there is no inference framework in the industry that surpasses its counterparts in all aspects. Different inference frameworks have their own advantages on different platforms and hardware. For example, on Nvidia GPU machines, TensorRT is the best performing inference framework; on x86 CPU machines, OpenVINO is the best performing inference framework; in the Apple ecosystem, CoreML is the best performing inference framework; on ARM Android, there are choices like ncnn, MNN, TFLite, TNN, etc.; under Rockchip, RKNN is the best performing inference framework. In summary: on specific hardware, the inference framework usually recommended by the hardware company is adopted.

1.2.2 Learning Costs, Development Costs, and Maintenance Costs of Multiple Inference Frameworks

The second pain point of model multi-end deployment - the learning costs, development costs, and maintenance costs of multiple inference frameworks. Different inference frameworks have different inference interfaces, hyperparameter configurations, Tensors, etc. If a model needs multi-end deployment, it requires writing a set of code for each different inference framework, which will bring significant learning costs, development costs, and maintenance costs to the model deployment engineers.

1.2.3 Diversity of Models

The first two pain points are about the pain points of model multi-end deployment, the third pain point is about the pain point of model deployment itself - the diversity of models. From the perspective of model deployment, it can be divided into single input, multiple inputs, single output, multiple outputs, static shape input, dynamic shape input, static shape output, dynamic shape output, etc. When combining the above differences with memory zero-copy optimization (directly operating the internal allocation of inputs and outputs of the inference framework), usually only engineers with rich model deployment experience can quickly find the optimal solution.

The following is a table combining model characteristics, descriptions, TensorRT manual graph construction, and actual algorithm examples:

模型特性	描述	TensorRT手动构图	实际算法例子
单输入	模型只有一个输入张量。	确保NetworkDefinition中只有一个输入节点。	图像分类模型ResNet，它接收单张图像作为输入，并输出图像的分类结果。
多输入	模型有多个输入张量。	在NetworkDefinition中定义多个输入节点，并在推理后处理时获取所有输入。	划痕修复模型，它接收原始图像以及划痕检测mask作为输入
单输出	模型只有一个输出张量。	确保NetworkDefinition中只有一个输出节点。	图像检测模型YOLOv5，将后处理融合到模型内部
多输出	模型有多个输出张量。	在NetworkDefinition中定义多个输出节点，并在推理后处理时获取所有输出。	图像检测模型YOLOv5，将后处理不融合到模型内部
静态形状输入	输入张量的形状在推理前已知且不变。	在BuilderConfig中设置固定的输入形状。	上述模型基本都为静态输入模型
动态形状输入	输入张量的形状在推理时可能变化。	使用IOptimizationProfile定义输入张量的动态形状，并在ExecutionContext中动态设置输入形状。	自适应的图像超分辨率模型，它能够接收不同尺寸的低分辨率图像作为输入，并输出高分辨率的图像。
静态形状输出	输出张量的形状在推理前已知且不变。	不需要在推理时动态调整。	除动态形状输入模型外，上述模型基本都为静态输出模型
动态形状输出	输出张量的形状在推理时可能变化。	需要在推理后处理时动态获取输出形状，并据此处理输出数据。	机器翻译模型，如Transformer，它接收任意长度的文本作为输入，并输出相应长度的目标语言翻译文本。

1.2.4 Pre- and Post-Processing for Model High Performance

The fourth pain point is also about the pain point of model deployment itself - the pre- and post-processing of models. Model deployment is not just about model inference, but also includes pre-processing and post-processing. Inference frameworks often only provide the functionality of model inference. Usually, deployment engineers need to develop the pre- and post-processing algorithms based on their understanding of the original algorithms through C++. For CV class algorithms, pre-processing is usually composed of operators such as cvtcolor, resize, padding, warp_affine, crop, normalize, transpose, etc. For most CV class models, pre-processing has a lot in common. For a certain category of algorithms, post-processing algorithms are particularly similar, so pre- and post-processing can be reused. When a certain pre- or post-processing is heavily reused, it can be considered for key optimization, thereby achieving further acceleration.

1.2.5 Complex Scenarios of Multiple Models

The fifth pain point is also about model deployment - the complex scenarios of multiple model combinations. Currently, many scenarios require solving business problems through the combination of multiple models. For example, old photo restoration, this algorithm has 6 models + 1 traditional algorithm (old photo->scratch detection->scratch restoration->super-resolution->condition(loop(face detection->face alignment->face restoration->face pasting))->restored photo) combination. Without the support of a deployment framework, there will be a lot of business code, high model coupling, poor flexibility, code not suitable for parallel processing, and other issues (prone to bugs, maintainability).

2 Overview

nndeploy is an end-to-end model deployment framework. The following figure shows the overall architecture of nndeploy, based on multi-end inference and directed acyclic graph model deployment, striving to provide users with a cross-platform, simple to use, high-performance model deployment experience.

2.1 Features

2.1.1 Ready-to-Use Algorithms

Currently, the deployment of YOLOV5, YOLOV6, YOLOV8, and SAM models has been completed, available for your direct use. We will continue to deploy other open-source models, allowing you to use them out of the box.

model	Inference	developer	remarks
YOLOV5	TensorRt/OpenVINO/ONNXRuntime/MNN	02200059Z、Always
YOLOV6	TensorRt/OpenVINO/ONNXRuntime	02200059Z、Always
YOLOV8	TensorRt/OpenVINO/ONNXRuntime/MNN	02200059Z、Always
SAM	ONNXRuntime	youxiudeshouyeren、Always

2.1.2 Support for Cross-Platform and Multiple Inference Frameworks

One set of code for multi-end deployment: by switching inference configurations, one set of code can complete the deployment of models across multiple platforms and multiple inference frameworks. Mainly targeting pain point one (fragmentation of inference frameworks) and pain point two (learning costs, development costs, and maintenance costs of multiple inference frameworks).

Currently supported inference frameworks are as follows:

Inference/OS	Linux	Windows	Android	MacOS	IOS	developer	remarks
TensorRT	√	-	-	-	-	Always
OpenVINO	√	√	-	-	-	Always
ONNXRuntime	√	√	-	-	-	Always
MNN	√	√	√	-	-	Always
TNN	√	√	√	-	-	02200059Z
ncnn	-	-	√	-	-	Always
coreML	-	-	-	√	-	JoDio-zd、jaywlinux
paddle-lite	-	-	-	-	-	qixuxiang
AscendCL	√	-	-	-	-	CYYAI
RKNN	√	-	-	-	-	100312dog

2.1.3 Simple and Easy to Use

• Based on directed acyclic graph model deployment: abstracting the deployment of AI algorithms end-to-end (pre-processing->inference->post-processing) into a directed acyclic graph Graph, pre-processing as a Node, inference also as a Node, post-processing also as a Node. Mainly targeting pain point four (reusing model pre- and post-processing).

• Inference template Infer: Based on the multi-end inference module Inference + directed acyclic graph node Node redesign, the powerful inference template Infer can help you internally handle different model variations, such as single input, multiple inputs, single output, multiple outputs, static shape input, dynamic shape input, static shape output, dynamic shape output, etc. Mainly targeting pain point three (model diversity).

• Efficiently solving complex scenarios of multiple models: In complex scenarios where multiple models are combined to complete a task (e.g., old photo restoration), each model can be an independent Graph. nndeploy’s directed acyclic graph supports embedding graphs flexibly and powerfully, breaking down big problems into small ones, quickly solving complex scenarios of multiple models through combination.

• Quickly constructing demo: For already deployed models, writing a demo to showcase effects is necessary, and the demo needs to handle multiple format inputs, such as image input output, folder containing multiple images input output, video input output, etc. By node-izing the above codec, it can be more versatile and efficient to complete the demo writing, achieving the goal of quickly showcasing effects (currently mainly implemented OpenCV-based codec node-ization).

2.1.4 High Performance

• Abstraction of the inference framework’s high performance: Each inference framework has its own characteristics, which need to be respected and understood to ensure the abstraction does not lose the framework’s features, providing a unified usage experience. nndeploy can configure most parameters of third-party inference frameworks, ensuring inference performance. It can directly operate the input and output allocated inside the inference framework, achieving zero-copy of pre and post-processing, improving the end-to-end performance of model deployment.

• Thread pool: Improves the concurrency performance and resource utilization rate of model deployment (thread pool). Additionally, it supports automatic parallelization of CPU operators, improving CPU operator execution performance (parallel_for).

• Memory pool: After completion, efficient memory allocation and release can be achieved (TODO).

• A set of high-performance operators: After completion, will accelerate your model’s pre and post-processing speed (TODO).

2.1.5 Parallel

• Serial: Executes each node in order according to the topological sorting of the model deployment’s directed acyclic graph.

• Pipeline parallel: In scenarios processing multiple frames, based on the model deployment method of directed acyclic graph, the pre-processing Node, inference Node, and post-processing Node can be bound to three different threads, each thread can also be bound to different hardware devices, enabling the three Nodes to process in pipeline parallel. In complex scenarios of multiple models and multiple hardware devices, the advantage of pipeline parallel can be more utilized, significantly improving overall throughput.

• Task parallel: In complex scenarios of multiple models and multiple hardware devices, based on the model deployment method of directed acyclic graph, the parallelism in model deployment can be fully explored, reducing the runtime overhead of a single algorithm’s full process.

• Combination parallel of the above modes: In complex scenarios of multiple models, multiple hardware devices, and processing multiple frames, nndeploy’s directed acyclic graph supports embedding graphs, each graph can have an independent parallel mode, allowing users to arbitrarily combine the parallel modes of model deployment tasks, with powerful expressive ability and can fully utilize hardware performance.

3 Architecture Introduction

nndeploy is a model end-to-end deployment framework based on multi-end inference and directed acyclic graph model deployment. The architecture introduction starts with multi-end inference and directed acyclic graph model deployment as the leads to introduce the overall architecture.

Three modules related to multi-end inference

3.1 Multi-end Inference

Multi-end inference submodule (Inference). Provides a unified method of model inference to operate different inference backends. The following diagram outlines the overall flow of nndeploy integrating a new inference framework, here taking MNN as an example. 1. First, understand MNN; 2. Understand the Inference submodule (inference super parameter configuration class InferenceParam, inference base class Inference); 3. Based on understanding MNN and the Inference base class, write the inference adapter (inherit the base class Inference, write MnnInference; inherit the base class InferenceParam, write MnnInferenceParam; write the inference-related data structure conversion tool class MnnConvert); 4. Run YOLOV5s based on the MNN backend.

3.1.1 Model Inference Class Inference

The corresponding files are \include\nndeploy\inference.h and \source\nndeploy\inference.cc, with detailed comments in the files, main functions as shown in the figure.

3.1.2 InferenceParam Inference Super Parameter Configuration Class

The corresponding files are \include\nndeploy\inference_param.h and \source\nndeploy\inference_param.cc. Each inference instance requires super parameter configuration, such as model inference precision, whether it is dynamic shape, etc. Detailed functions as shown in the figure.

3.1.3 Inference-related Data Structure Conversion Tool Class

nndeploy provides a unified Tensor and the hyperparameter data structure required for inference. Each inference framework has its own custom Tensor and hyperparameter data structure. To ensure a unified interface calling experience, a converter module needs to be written. Since each inference framework’s definitions differ, this tool class cannot define a base class; its main purpose is to serve the internal use of the inference framework adapter, and it does not require a base class. For reference, see \include/nndeploy/inference/mnn/mnn_converter.h and \source/nndeploy/inference/mnn/mnn_converter.c. The specific implementation is as shown in the following diagram.

3.2 Data Containers Tensor && Buffer

Each inference framework has different data interaction methods, such as TensorRT’s io_binding method, OpenVINO’s ov::Tensor, TNN’s TNN::Blob. Not only is it necessary to provide a unified inference class and inference hyperparameter configuration class, but also to design a universal Tensor, with the member variables of Tensor and TensorDesc as shown in the diagram.

The input and output of model inference can be data on heterogeneous devices, such as TensorRT’s input being CUDA memory. Introducing Buffer, Tensor is decoupled from heterogeneous devices. The member variables of Buffer and BufferDesc are as shown in the diagram.

3.3 Device Management

The device is nndeploy’s abstraction of hardware devices. Through the abstraction of hardware devices, it shields the differences brought by different hardware programming models. nndeploy currently supports devices such as CPU, X86, ARM, CUDA, AscendCL, etc. The main functionalities are as follows:

• Unified memory allocation: provides a unified memory allocation interface for different devices, simplifying the memory allocation for data containers Buffer, Mat, Tensor.

• Unified memory copy: provides a unified memory copy interface for different devices (device-to-device copy, host-to-device upload/download), simplifying the memory copy for data containers Buffer, Mat, Tensor.

• Unified synchronization operation: provides a unified synchronization operation interface for different devices, simplifying device-side model inference, operator synchronization operations.

• Unified hardware device information query: provides a unified hardware device information query interface for different devices, helping users better choose the runtime devices for the entire model deployment process.

Three modules related to model deployment based on directed acyclic graphs.

3.4 Model Deployment Based on Directed Acyclic Graphs

The following diagram is an actual example of YOLOv8n.

This is a very typical directed acyclic graph, model preprocessing->model inference->model inference constitutes NNDEPLOY_YOLOV8 DAG (a library that can be called externally), this DAG and the decoding node as well as the drawing box node can together form a new DAG (a demo executable program).

Note: For already deployed models, it’s necessary to write a demo to show the effect, and the demo needs to handle inputs of multiple formats, such as image input/output, multiple images in a folder input/output, video input/output, etc. By modularizing the above decoding nodes, it can be more universal and efficient to complete the writing of the demo, achieving the goal of quickly showing the effect.

3.5 Pipeline Parallelism

In scenarios processing multiple frames, based on the model deployment method of directed acyclic graphs, the preprocessing Node, inference Node, and postprocessing Node can be bound to three different threads, each thread can also be bound to different hardware devices, thus the three Nodes can process in pipeline parallel. In complex scenarios with multiple models and multiple hardware devices, the advantages of pipeline parallelism can be more fully utilized, significantly improving overall throughput. The following diagram is an actual example of directed acyclic graph + pipeline parallelism optimized YOLOv8n.

3.6 Complex Scenarios with Multiple Models

The following diagram is an actual example of an old photo restoration algorithm, which includes 6 models + 1 traditional algorithm (old photo->scratch detection->scratch restoration->super-resolution->condition(loop(face detection->face correction->face restoration->face pasting))->restored photo) combination. Based on nndeploy, deploying through dag is very straightforward and the mental burden of development is very small. If dag is not used for deployment, in actual code, each model needs to be manually concatenated, which would involve a large amount of business code, high model coupling, poor flexibility, code not suitable for parallelization, and other issues.

4 Next Steps Planning

• Inference backend

  • Improve the already integrated inference framework coreml

  • Improve the already integrated inference framework paddle-lite

  • Integrate the new inference framework TFLite

• Device management module

  • Add OpenCL’s device management module

  • Add ROCM’s device management module

  • Add OpenGL’s device management module

• Memory optimization

  • Master-slave memory copy optimization: for the unified memory architecture, replace master-slave memory copy with master-slave memory mapping, master-slave memory address sharing, etc.

  • Memory pool: for nndeploy’s internal data containers Buffer, Mat, Tensor, establish a memory pool for heterogeneous devices, achieving high-performance memory allocation and release.

  • Multi-node shared memory mechanism: for multi-model serial scenarios, based on the directed acyclic graph of model deployment, support multi-inference node shared memory mechanism in serial execution mode.

  • Edge circular queue memory reuse mechanism: based on the directed acyclic graph of model deployment, support edge circular queue shared memory mechanism in pipeline parallel execution mode.

• stable diffusion model

  • Deploy stable diffusion model

  • For the stable diffusion model with stable_diffusion.cpp (inference module, manually constructing the computation graph)

  • High-performance op

  • Distributed

    • In scenarios where multiple models jointly complete a task, distribute multiple models to multiple machines for distributed execution

    • In the context of large models, by splitting the large model into multiple sub-models, distribute multiple sub-models to multiple machines for distributed execution

5 References

• TNN

• FastDeploy

• opencv

• CGraph

• CThreadPool

• tvm

• mmdeploy

• FlyCV

• torchpipe

6 Join Us

• nndeploy is developed and maintained by a group of like-minded netizens together. We regularly discuss technology and share industry insights. Currently, nndeploy is in the development stage. If you love open source, enjoy challenges, whether for learning purposes or have better ideas, you are welcome to join us.

• WeChat: titian5566 (You can add my WeChat to join the nndeploy discussion group, note: nndeploy+name)