Open-source AI deployment framework nndeploy - from requirement analysis to architecture design

Project address: https://github.com/nndeploy/nndeploy Welcome star and PR

1. 模型部署实际案例

Through practical cases of model deployment, to answer what a deployment framework is.

This is a practical case of multi-end deployment of an AI intelligent cropping model, extracting from the Mona Lisa image through a portrait segmentation model. The training framework is responsible for training this algorithm, and the inference framework completes the online deployment in the production environment. Training frameworks include pytorch, tf, etc., and major manufacturers also launch their own inference frameworks, such as NVIDIA’s TensorRT, Intel’s OpenVINO, Ali’s mnn, etc. There are many pain points to be solved between the training framework and the inference framework, and the deployment framework is designed to solve these pain points. From this practical case, a clear pain point in model deployment can be identified: this image software has many versions such as iOS, Android, web, and PC (Win Mac Linux), and without a deployment framework, it will be difficult to solve the pain point of multi-end deployment. Specific pain points will be detailed in the next chapter’s requirement analysis.

1 Requirement Analysis

First is the requirement analysis, which is why we need to do nndeploy, what are the actual scenarios for multi-end model deployment, and what are the current pain points in multi-end model deployment and model deployment.

1.2 Multi-end model deployment and model deployment pain points

1.2.1 Fragmentation in inference frameworks

The first pain point in multi-end model deployment - fragmentation in inference frameworks. Currently, there is no inference framework in the industry that surpasses its counterparts in all aspects, and different inference frameworks have their respective advantages on different platforms and hardware. For example, on NVidia GPU machines, TensorRT is the best performing inference framework; on x86 CPU machines, OpenVINO is the best performing inference framework; in the Apple ecosystem, coreml is the best performing inference framework; on ARM Android, there are choices like ncnn, MNN, TFLite, TNN, etc.; on Rockchip microcontrollers, RKNN is the best performing inference framework. In summary: under specific hardware, the inference framework provided by the hardware company is usually adopted.

1.2.2 Learning cost, development cost, maintenance cost of multiple inference frameworks

The second pain point in multi-end model deployment - the learning cost, development cost, maintenance cost of multiple inference frameworks. Different inference frameworks have different inference interfaces, hyperparameter configurations, Tensors, etc. If a model needs to be deployed on multiple ends, a set of code needs to be written for each different inference framework, which will bring significant learning cost, development cost, and maintenance cost to model deployment engineers.

1.2.3 Diversity of models

The above two pain points are for multi-end model deployment, the third pain point is the pain point of model deployment itself - the diversity of models. From the perspective of model deployment, it can be divided into single input, multiple inputs, single output, multiple outputs, static shape input, dynamic shape input, static shape output, dynamic shape output, etc. When the above differences are combined with memory zero-copy optimization (directly operating the internal allocation of input and output of the inference framework), usually only engineers with rich model deployment experience can quickly find the optimal solution.

Below is a table combining model characteristics, descriptions, TensorRT manual graph construction, and actual algorithm examples:

模型特性	描述	TensorRT手动构图	实际算法例子
单输入	模型只有一个输入张量。	确保NetworkDefinition中只有一个输入节点。	图像分类模型ResNet，它接收单张图像作为输入，并输出图像的分类结果。
多输入	模型有多个输入张量。	在NetworkDefinition中定义多个输入节点，并在推理后处理时获取所有输入。	划痕修复模型，它接收原始图像以及划痕检测mask作为输入
单输出	模型只有一个输出张量。	确保NetworkDefinition中只有一个输出节点。	图像检测模型YOLOv5，将后处理融合到模型内部
多输出	模型有多个输出张量。	在NetworkDefinition中定义多个输出节点，并在推理后处理时获取所有输出。	图像检测模型YOLOv5，将后处理不融合到模型内部
静态形状输入	输入张量的形状在推理前已知且不变。	在BuilderConfig中设置固定的输入形状。	上述模型基本都为静态输入模型
动态形状输入	输入张量的形状在推理时可能变化。	使用IOptimizationProfile定义输入张量的动态形状，并在ExecutionContext中动态设置输入形状。	自适应的图像超分辨率模型，它能够接收不同尺寸的低分辨率图像作为输入，并输出高分辨率的图像。
静态形状输出	输出张量的形状在推理前已知且不变。	不需要在推理时动态调整。	除动态形状输入模型外，上述模型基本都为静态输出模型
动态形状输出	输出张量的形状在推理时可能变化。	需要在推理后处理时动态获取输出形状，并据此处理输出数据。	机器翻译模型，如Transformer，它接收任意长度的文本作为输入，并输出相应长度的目标语言翻译文本。

1.2.4 High-performance pre and post-processing of models

The fourth pain point is also a pain point of model deployment itself - the pre and post-processing of models. Model deployment is not just about model inference, it also includes pre-processing and post-processing, and inference frameworks often only provide the function of model inference. Usually, deployment engineers need to understand the original algorithm, develop the algorithm’s pre and post-processing through C++. For CV class algorithms, pre-processing is usually composed of operators such as cvtcolor, resize, padding, warp_affine, crop, normalize, transpose. For most CV models, pre-processing has a lot in common, and for a certain category of algorithms, post-processing algorithms are particularly similar, so pre and post-processing can be reused. When a certain pre or post-processing is heavily reused, it can be considered for key optimization to obtain further acceleration.

1.2.5 Complex scenarios with multiple models

The fifth pain point is also a pain point of model deployment - complex scenarios with multiple model combinations. Currently, many scenarios require solving business problems with multiple model combinations, such as old photo restoration, which involves 6 models + 1 traditional algorithm (old photo->scratch detection->scratch repair->super resolution->condition(loop(face detection->face correction->face repair->face paste back))->restored photo). Without the support of a deployment framework, there will be a lot of business code, high model coupling, poor flexibility, code not suitable for parallelization, and other issues (easy to bug, maintainability).

2 Overview

nndeploy is an end-to-end model deployment framework. The figure below shows the overall architecture of nndeploy, based on multi-end inference and directed acyclic graph model deployment, aiming to provide users with a cross-platform, simple to use, high-performance model deployment experience.

2.1 Features

2.1.1 Ready-to-use algorithms out of the box

Currently, the deployment of YOLOV5, YOLOV6, YOLOV8, SAM models has been completed, available for your direct use. We will continue to deploy other open-source models, allowing you to use them out of the box.

model	Inference	developer	remarks
YOLOV5	TensorRt/OpenVINO/ONNXRuntime/MNN	02200059Z、Always
YOLOV6	TensorRt/OpenVINO/ONNXRuntime	02200059Z、Always
YOLOV8	TensorRt/OpenVINO/ONNXRuntime/MNN	02200059Z、Always
SAM	ONNXRuntime	youxiudeshouyeren、Always

2.1.2 Support for cross-platform and multiple inference frameworks

One code, multiple deployments: By switching inference configurations, one set of code can complete model deployment across multiple platforms and multiple inference frameworks. Mainly targeting pain point one (fragmentation of inference frameworks) and pain point two (the learning cost, development cost, and maintenance cost of multiple inference frameworks).

Currently supported inference frameworks are as follows:

Inference/OS	Linux	Windows	Android	MacOS	IOS	developer	remarks
TensorRT	√	-	-	-	-	Always
OpenVINO	√	√	-	-	-	Always
ONNXRuntime	√	√	-	-	-	Always
MNN	√	√	√	-	-	Always
TNN	√	√	√	-	-	02200059Z
ncnn	-	-	√	-	-	Always
coreML	-	-	-	√	-	JoDio-zd、jaywlinux
paddle-lite	-	-	-	-	-	qixuxiang
AscendCL	√	-	-	-	-	CYYAI
RKNN	√	-	-	-	-	100312dog

2.1.3 Easy to use

• Based on directed acyclic graph deployment model: Deploying AI algorithms end-to-end (pre-processing -> inference -> post-processing) is abstracted as a directed acyclic graph Graph, where pre-processing is one Node, inference is another Node, and post-processing is also a Node. Mainly targeting pain point four (reuse of model’s pre and post processing).

• Inference template Infer: Based on multi-end inference module Inference + directed acyclic graph node Node, redesigned to be a powerful inference template Infer. The Infer inference template can help you internally handle the differences brought by different models, such as single input, multiple inputs, single output, multiple outputs, static shape input, dynamic shape input, static shape output, dynamic shape output, and a series of differences. Mainly targeting pain point three (diversity of models).

• Efficiently solving complex scenarios with multiple models: In complex scenarios where multiple models work together to complete a task (such as old photo restoration), each model can be an independent Graph. nndeploy’s directed acyclic graph supports embedding graphs flexibly and powerfully, breaking down big problems into small ones, quickly solving complex multi-model scenario problems through combination.

• Quick demo construction: For models already deployed, writing a demo to showcase the effect is necessary, and the demo needs to handle inputs in multiple formats, such as image input output, input output of multiple images in a folder, video input output, etc. By modularizing the above encoding and decoding nodes, it can be more universal and efficient to complete the demo writing, achieving the purpose of quickly showcasing the effect (currently mainly implemented the encoding and decoding node modularization based on OpenCV).

2.1.4 High performance

• Abstraction of inference framework’s high performance: Each inference framework also has its own characteristics, which need to be respected and understood enough to not lose the characteristics of the inference framework in abstraction, and to achieve a unified user experience. nndeploy can configure most parameters of third-party inference frameworks, ensuring inference performance. It can directly operate the input and output allocated by the inference framework internally, achieving zero copy of pre and post processing, improving the end-to-end performance of model deployment.

• Thread pool: Improve the concurrency performance and resource utilization rate of model deployment (thread pool). In addition, it also supports automatic parallelization of CPU operators, which can improve the execution performance of CPU operators (parallel_for).

• Memory pool: After completion, efficient memory allocation and release can be achieved (TODO).

• A set of high-performance operators: After completion, it will accelerate the speed of your model’s pre and post processing (TODO).

2.1.5 Parallel

• Serial: According to the topological sorting of the model deployment’s directed acyclic graph, execute each node in sequence.

• Pipeline parallel: In scenarios processing multiple frames, based on the directed acyclic graph model deployment method, the pre-processing Node, inference Node, and post-processing Node can be bound to three different threads, each thread can also be bound to different hardware devices, thus the three Nodes can process in pipeline parallel. In complex scenarios with multiple models and multiple hardware devices, the advantages of pipeline parallel can be more utilized, significantly improving the overall throughput.

• Task parallel: In complex scenarios with multiple models and multiple hardware devices, based on the directed acyclic graph model deployment method, the parallelism in model deployment can be fully exploited to shorten the runtime of a single algorithm’s full process.

• Combination of the above modes’ parallel: In complex scenarios with multiple models, multiple hardware devices, and processing multiple frames, nndeploy’s directed acyclic graph supports the function of embedding graphs, each graph can have an independent parallel mode, so users can freely combine the parallel modes of model deployment tasks, with powerful expression ability and can fully exploit hardware performance.

3 Architecture Introduction

nndeploy is a model end-to-end deployment framework based on multi-end inference and directed acyclic graph model deployment. The architecture introduction introduces the overall architecture from two aspects: multi-end inference and directed acyclic graph model deployment.

Three modules related to multi-end inference

3.1 Multi-end inference

Multi-end inference sub-module (Inference). Provides a unified method of model inference to operate different inference backends. The following figure outlines the overall process of nndeploy accessing a new inference framework, here taking MNN as an example. 1. First is understanding MNN; 2. Understanding the Inference sub-module (inference super parameter configuration class InferenceParam, inference base class Inference); 3. Based on understanding MNN and the base class Inference, writing the inference adapter (inheriting the base class Inference, writing MnnInference; inheriting the base class InferenceParam, writing MnnInferenceParam; writing the conversion tool class MnnConvert for inference-related data structures); 4. Running YOLOV5s based on MNN backend.

3.1.1 Model inference class Inference

The corresponding files are \include\nndeploy\inference.h and \source\nndeploy\inference.cc, the files have more detailed annotation instructions, the main functions are as shown in the figure

3.1.2 InferenceParam Inference super parameter configuration class

The corresponding files are \include\nndeploy\inference_param.h and \source\nndeploy\inference_param.cc. Each inference instance requires hyperparameter configuration, such as model inference precision, whether it is dynamic shape, etc. Detailed functions are as shown in the figure.

3.1.3 Inference-related data structure conversion tool class

nndeploy provides a unified Tensor and the hyperparameter data structure required for inference. Each inference framework has its own custom Tensor and hyperparameter data structure. To ensure a unified interface calling experience, a converter module needs to be written. Since each inference framework definition is different, this tool class cannot define a base class. The main purpose of this tool class is also to serve the inference framework adapter internally, and a base class is not required. Refer to \include/nndeploy/inference/mnn/mnn_converter.h and \source/nndeploy/inference/mnn/mnn_converter.c. The specific implementation is as shown in the figure.

3.2 Data container Tensor && Buffer

Each inference framework has different data interaction methods, such as TensorRT’s io_binding method, OpenVINO’s ov::Tensor, TNN’s TNN::Blob. Not only is it necessary to provide a unified inference class and inference hyperparameter configuration class, but also to design a universal Tensor. The member variables of Tensor and TensorDesc are as shown in the figure.

The input/output of model inference can be data on heterogeneous devices, such as TensorRT’s input being CUDA memory. Introducing Buffer to decouple Tensor from heterogeneous devices. The member variables of Buffer and BufferDesc are as shown in the figure.

3.3 Device management

Device is nndeploy’s abstraction of hardware devices. Through the abstraction of hardware devices, it shields the differences brought by different hardware programming models. nndeploy currently supports devices such as CPU, X86, ARM, CUDA, AscendCL, etc. The main functions are as follows

• Unified memory allocation: Provide a unified memory allocation interface for different devices, thereby simplifying the memory allocation of data containers Buffer, Mat, Tensor

• Unified memory copy: Provide a unified memory copy interface for different devices (copy between devices, upload/download between host and device), thereby simplifying the memory copy of data containers Buffer, Mat, Tensor

• Unified synchronization operation: Provide a unified synchronization operation interface for different devices, simplifying device-side model inference, operator synchronization operations, etc.

• Unified hardware device information query: Provide a unified hardware device information query interface for different devices, helping users better choose the running device for the entire process deployment of the model

Three modules related to model deployment based on directed acyclic graph

3.4 Model deployment based on directed acyclic graph

The following figure is an actual example of YOLOv8n.

This is a very typical directed acyclic graph, model preprocessing->model inference->model inference constitutes NNDEPLOY_YOLOV8 DAG (a library that can be called externally). This DAG, along with the decoding node and drawing box node, can form a new DAG (a demo executable program).

Note: For already deployed models, a demo needs to be written to display the effect. The demo needs to handle multiple formats of input, such as image input/output, multiple images in a folder input/output, video input/output, etc. By modularizing the above decoding node, it can be more universal and efficient to complete the writing of the demo, achieving the purpose of quickly displaying the effect.

3.5 Pipeline parallel

In scenarios processing multiple frames, based on the model deployment method of directed acyclic graph, the preprocessing Node, inference Node, and postprocessing Node can be bound to three different threads. Each thread can also be bound to different hardware devices, so the three Nodes can process in pipeline parallel. In complex scenarios with multiple models and multiple hardware devices, the advantages of pipeline parallel can be more utilized, significantly improving the overall throughput. The following figure is an actual example of optimizing YOLOv8n with directed acyclic graph + pipeline parallel.

3.6 Complex scenarios with multiple models

The following figure is an actual example of an old photo restoration algorithm. This algorithm has 6 models + 1 traditional algorithm (old photo->scratch detection->scratch restoration->super resolution->condition(loop(face detection->face correction->face restoration->face paste back))->restored photo) combination. Based on nndeploy, deploying through dag is very direct and the mental burden of development is very small. If dag is not used for deployment, in the actual code, each model needs to be manually concatenated, which will have a lot of business code, high model coupling, poor flexibility, code not suitable for parallel and other issues.

4 Next step planning

• Inference backend

  • Complete the already integrated inference framework coreml

  • Complete the already integrated inference framework paddle-lite

  • Integrate the new inference framework TFLite

• Device management module

  • Add a new device management module for OpenCL

  • Add a new device management module for ROCM

  • Add a new device management module for OpenGL

• Memory optimization

  • Master-slave memory copy optimization: For the unified memory architecture, through master-slave memory mapping, master-slave memory address sharing and other methods to replace master-slave memory copy

  • Memory Pool: For the internal data containers Buffer, Mat, and Tensor of nndeploy, establish a memory pool for heterogeneous devices to achieve high-performance memory allocation and deallocation.

  • Multi-node Shared Memory Mechanism: In scenarios involving multiple model serial connections, based on the directed acyclic graph of model deployment, under the mode of serial execution, support the shared memory mechanism among multiple inference nodes.

  • Edge Circular Queue Memory Reuse Mechanism: Based on the directed acyclic graph of model deployment, under the mode of pipeline parallel execution, support the shared memory mechanism of edge circular queues.

• stable diffusion model

  • Deploy stable diffusion model

  • For stable diffusion model, integrate stable_diffusion.cpp (inference submodule, manually construct the computation graph)

  • High-performance op

  • Distributed

    • In scenarios where multiple models jointly complete a task, schedule multiple models to multiple machines for distributed execution.

    • In scenarios of large models, by splitting the large model into multiple sub-models, schedule multiple sub-models to multiple machines for distributed execution.

5 References

• TNN

• FastDeploy

• opencv

• CGraph

• CThreadPool

• tvm

• mmdeploy

• FlyCV

• torchpipe

6 Join Us

• nndeploy is jointly developed and maintained by a group of like-minded netizens. We discuss technology and share industry insights from time to time. Currently, nndeploy is in the development stage. If you love open source, enjoy challenges, whether for learning purposes or have better ideas, you are welcome to join us.

• WeChat: titian5566 (You can add my WeChat to join the nndeploy exchange group, note: nndeploy + name)