English | Simplified Chinese

Introduction¶

nndeploy is an end-to-end model deployment framework. Based on multi-end inference and deployment on directed acyclic graph models, it aims to provide users with a cross-platform, simple-to-use, and high-performance model deployment experience.

Architecture¶

Architecture

Features¶

1. 开箱即用的算法¶

Currently, deployment for models such as YOLOV5, YOLOV6, YOLOV8 has been completed and is ready for your direct use. We will continue to deploy other open-source models, allowing you to use them right out of the box.

model	Inference	developer
YOLOV5	TensorRt/OpenVINO/ONNXRuntime/MNN	02200059Z、Always
YOLOV6	TensorRt/OpenVINO/ONNXRuntime	02200059Z、Always
YOLOV8	TensorRt/OpenVINO/ONNXRuntime/MNN	02200059Z、Always
SAM	ONNXRuntime	youxiudeshouyeren、Always

2. 支持跨平台和多推理框架¶

One code for multi-end deployment: By switching inference configurations, one set of code can complete model deployment across multiple platforms and multiple inference frameworks.

Currently supported inference frameworks include:

Inference/OS	Linux	Windows	Android	MacOS	IOS	developer
TensorRT	√	-	-	-	-	Always
OpenVINO	√	√	-	-	-	Always
ONNXRuntime	√	√	-	-	-	Always
MNN	√	√	√	-	-	Always
TNN	√	√	√	-	-	02200059Z
ncnn	-	-	√	-	-	Always
coreML	-	-	-	√	-	JoDio-zd、jaywlinux
paddle-lite	-	-	-	-	-	qixuxiang
AscendCL	√	-	-	-	-	CYYAI
RKNN	√	-	-	-	-	100312dog

Notice:

TFLite, TVM, OpenPPL, sophgo, Horizon are under development. We are striving to cover the vast majority of mainstream inference frameworks.
When selecting the corresponding inference backend, the loaded model file must still be in the format required by that inference backend.

3. 简单易用¶

Based on directed acyclic graph model deployment: The deployment of AI algorithms end-to-end (pre-processing -> inference -> post-processing) is abstracted as a directed acyclic graph Graph, with pre-processing as one Node, inference as another Node, and post-processing as yet another Node.
Inference template Infer: Based on multi-end inference module Inference + directed acyclic graph node Node redesign, a powerful inference template Infer is created. The Infer inference template can help you internally handle the differences brought by different models, such as single input, multiple inputs, single output, multiple outputs, static shape input, dynamic shape input, static shape output, dynamic shape output, and a series of different scenarios.
Efficiently solving complex scenarios of multiple models: In complex scenarios where multiple models combine to complete a task (e.g., old photo restoration), each model can be an independent Graph. nndeploy’s directed acyclic graph supports embedding graphs within graphs, offering flexible and powerful functionality, breaking down big problems into small ones, and quickly solving complex multi-model scenario problems through combination.
Quickly build demo: For already deployed models, writing a demo to showcase the results is necessary. The demo needs to handle multiple formats of input, such as image input/output, folder with multiple images input/output, video input/output, etc. By node-izing the above codecs, it can be more universal and efficient to complete the writing of the demo, achieving the goal of quickly showcasing results (currently mainly implemented node-ization based on OpenCV’s codecs).

4. 高性能¶

Abstraction of inference framework’s high performance: Each inference framework has its own characteristics. It’s essential to respect and understand these inference frameworks to avoid losing their features during abstraction and achieve a unified user experience. nndeploy can configure the vast majority of parameters of third-party inference frameworks, ensuring inference performance. It can directly operate the internal inputs and outputs allocated by the inference framework, achieving zero-copy of pre- and post-processing, thereby improving the end-to-end performance of model deployment.
Thread pool: Improves the concurrency performance and resource utilization of model deployment (thread pool). Additionally, it supports automatic parallelization of CPU-side operators, which can enhance the execution performance of CPU operators (parallel_for).
Memory pool: After completion, it can achieve efficient memory allocation and release (TODO).
A set of high-performance operators: After completion, it will accelerate the speed of your model’s pre- and post-processing (TODO).

5. 并行¶

Serial: According to the topological order of the model’s deployment directed acyclic graph, execute each node in sequence.
Pipeline parallel: In scenarios processing multiple frames, based on the model deployment method of directed acyclic graph, the pre-processing Node, inference Node, and post-processing Node can be bound to three different threads, with each thread bound to different hardware devices, allowing the three Nodes to process in pipeline parallel. In complex scenarios with multiple models and multiple hardware devices, the advantages of pipeline parallel can be more fully utilized, significantly improving overall throughput.
Task parallel: In complex scenarios with multiple models and multiple hardware devices, based on the model deployment method of directed acyclic graph, the parallelism in model deployment can be fully exploited, reducing the runtime overhead of a single algorithm’s entire process.
Combination of the above parallel modes: In complex scenarios with multiple models, multiple hardware devices, and processing multiple frames, nndeploy’s directed acyclic graph supports embedding graphs within graphs, allowing each graph to have its own parallel mode. Thus, users can arbitrarily combine the parallel modes of model deployment tasks, with strong expressive power and the ability to fully utilize hardware performance.

Resource repository¶

We have uploaded third-party libraries, model repositories, and test data to HuggingFace. If needed, you are welcome to download and use them.

Next steps¶

Inference backend
- Complete the already integrated inference framework coreml
- Complete the already integrated inference framework paddle-lite
- Integrate the new inference framework TFLite
Device management module
- Add OpenCL’s device management module
- Add ROCM’s device management module
- Add OpenGL’s device management module
Memory optimization
- Master-slave memory copy optimization: For architectures with unified memory, replace master-slave memory copying with methods like master-slave memory mapping and master-slave memory address sharing.
- Memory pool: For nndeploy’s internal data containers Buffer, Mat, Tensor, establish a memory pool for heterogeneous devices to achieve high-performance memory allocation and release.
- Multi-node shared memory mechanism: In scenarios of multiple model series, based on the directed acyclic graph of model deployment, support a shared memory mechanism for multiple inference nodes under the serial execution mode.
- Edge circular queue memory reuse mechanism: Based on the directed acyclic graph of model deployment, support an edge circular queue shared memory mechanism under the pipeline parallel execution mode.
stable diffusion model
- Deploy stable diffusion model
- For stable diffusion model with stable_diffusion.cpp (inference module, manually constructing computation graph approach)
- High-performance op
- Distributed
  - In scenarios where multiple models collaboratively complete a task, scheduling multiple models to multiple machines for distributed execution
  - In large model scenarios, by splitting the large model into multiple sub-models, scheduling multiple sub-models to multiple machines for distributed execution

References¶

Join us¶

nndeploy is jointly developed and maintained by a group of like-minded friends. We regularly discuss technology and share industry insights. Currently, nndeploy is in the development stage. Whether you are passionate about open source, enjoy tinkering, aiming to learn, or have better ideas, you are welcome to join us.
WeChat: titian5566 (You can add my WeChat to join the nndeploy discussion group, note: nndeploy + name)