v1.0.0.0

After seven months, with 15 developers and 245 commits, nndeploy welcomes the stable version v1.0.0.0. This version improves the architecture, adds new features, builds examples, and documents the process. The current version better aligns with our goal - a multi-platform, high-performance, simple and easy-to-use machine learning deployment framework.

We hope the v1.0.0.0 version can be applied in more practical scenarios, generating business value. Welcome to experience the new version, we look forward to your feedback and more importantly, your joining: https://github.com/nndeploy/nndeploy.

nndeploy v1.0.0.0 mainly includes the following new features, highlights, and optimizations:

1. 优化后的全新架构

Architecture

2. 新增模型支持

We have added support for the segmentation model SAM (segment anything), below is an example result of SAM segmentation:

sam

3. 新增多种并行模式

  • Serial: Executes each node in order according to the topological sorting of the model deployment’s directed acyclic graph.

  • Task parallel: In complex scenarios with multiple models and multiple hardware devices, based on the directed acyclic graph model deployment method, it can fully exploit the parallelism in model deployment, reducing the total runtime of a single algorithm process.

  • Pipeline parallel: In scenarios processing multiple frames, based on the directed acyclic graph model deployment method, it can bind the preprocessing Node, inference Node, and post-processing Node to three different threads, each thread can also be bound to different hardware devices, enabling the three Nodes to process in pipeline parallel. In complex scenarios with multiple models and multiple hardware devices, it can better leverage the advantages of pipeline parallel, significantly improving overall throughput.

  • Combination of the above parallel modes: In complex scenarios with multiple models, multiple hardware devices, and processing multiple frames, nndeploy’s directed acyclic graph supports embedding graphs that are flexible and powerful, each graph can have independent parallel modes, allowing users to freely combine the parallel modes of model deployment tasks in your complex scenarios, with strong expressive power and fully utilizing hardware performance.

  • Performance comparison between YOLOv8n serial and pipeline parallel

    The performance comparison results between YOLOv8n processing 24 images in serial and pipeline parallel are shown in the table below

模型 推理引擎 运行方式 耗时（ms）
YOLOv8n onnxruntime 串行 936.359
YOLOv8n onnxruntime 流水线并行 796.763
YOLOv8n TensorRT 串行 327.416
YOLOv8n TensorRT 流水线并行 120.463

  • Serial vs task-level parallel

    Due to the current model’s inference process having little internal parallelism, insufficient to reflect task-level parallel, we constructed a virtual graph; when each node’s time consumption is set to 10ms, task-level parallel can reduce the serial’s 90ms to 50ms.

    sam

4. 新增线程池

The thread pool plays a crucial role in nndeploy. It supports nndeploy’s parallel requirements, providing a stable foundation for task-level parallel and pipeline parallel. The implementation of the thread pool allows the framework to manage and dispatch tasks more efficiently, effectively utilizing computing resources, and improving the framework’s overall performance.

Thread pool

5. 有向无环图的优化与重构

  • Node reconstruction and optimization: Node is a node in the directed acyclic graph, representing performing a certain operation or action. Preprocessing, inference, post-processing, and decoding are all nodes, the entire directed acyclic graph can also be seen as a node. The node itself does not hold any data, data flows in from the input edge, after calculation, flows out to the output edge.

  • Edge reconstruction and optimization: Edge is the edge in the directed acyclic graph, representing the direction of data flow, each node’s output edge is responsible for managing memory. The edge may hold data such as Mat, Tensor, Buffer. Edges are divided into fixed edges and pipeline edges, only fixed edges are used in serial and task-level parallel; only pipeline edges are used in pipeline parallel.

  • Graph reconstruction and optimization: Graph is the directed acyclic graph. Composed of a series of Nodes, Edges, and their topological relationships, a Graph can also embed a Graph.

  • Loop Graph: Inherits Graph, adds Loop Graph, enhances the expressive power of the directed acyclic graph, improves development efficiency in complex multi-model deployment scenarios.

  • Condition Graph: Inherits Graph, adds Loop Graph, enhances the expressive power of the directed acyclic graph, improves development efficiency in complex multi-model deployment scenarios, in processing multiple frames and deploying single models on multiple devices, through combination with pipeline parallel, can fully utilize all hardware performance, significantly improving overall throughput.

6. 推理模板的重构

Reconstructed the inference template, based on the multi-end inference module Inference + directed acyclic graph node Node redesigned the powerful inference template Infer, the Infer inference template can help you internally handle differences brought by different models, such as single input, multiple outputs, static shape input, dynamic shape input, static shape output, dynamic shape output, whether to operate the inference framework’s internal allocation of inputs and outputs, etc., a series of differences, facing different models’ differences, usually requires engineers with rich model deployment experience to quickly and efficiently solve.

7. 新增编解码节点化

For already deployed models, writing a demo to show the effect is necessary, and the demo needs to handle multiple formats of input, such as image input output, folder containing multiple images’ input output, video’s input output, etc., by nodeizing the above decoding, can more universally and efficiently complete the writing of the demo, achieving the purpose of quickly showing the effect.

8.多端推理子模块

The current version newly adds support for the following inference frameworks:

Inference/OS Linux Windows Android MacOS IOS developer remarks
ncnn - - √ - - Always
coreML - - - √ - JoDio-zd、jaywlinux
paddle-lite - - - - - qixuxiang
AscendCL √ - - - - CYYAI
RKNN √ - - - - 100312dog

9. 设备管理模块

The current version newly adds support for the following Huawei Ascend device management module:

Device Manager developer remarks
AscendCL CYYAI

10. 文档

Constructed friendly, comprehensive documentation, the documentation does not aim to explain the usage of API interfaces, but focuses on exploring the design concepts and working principles behind the framework, allowing users to more comprehensively understand the framework’s operation mode, thus better leveraging nndeploy’s development efficiency and high-performance advantages in model deployment.

This document includes overview, developer guidelines, architecture guidelines and other module contents. In future updates, we will continue to refine the documentation, striving to provide users with clear and understandable documentation.

Next steps planning

  • Inference backend

    • Complete the already integrated inference framework CoreML

    • Complete the already integrated inference framework Paddle-Lite

    • Integrate the new inference framework TFLite

  • Device management module

    • Add OpenCL device management module

    • Add ROCM device management module

    • Add OpenGL device management module

  • Memory optimization

    • Master-slave memory copy optimization: For architectures with unified memory, replace master-slave memory copy with methods like master-slave memory mapping and shared master-slave memory addresses.

    • Memory pool: For NNDeply’s internal data containers Buffer, Mat, Tensor, establish a memory pool for heterogeneous devices to achieve high-performance memory allocation and release.

    • Multi-node shared memory mechanism: For scenarios with multiple model series, based on the directed acyclic graph of model deployment, support a shared memory mechanism for multiple inference nodes in serial execution mode.

    • Edge circular queue memory reuse mechanism: Based on the directed acyclic graph of model deployment, support an edge circular queue shared memory mechanism in pipeline parallel execution mode.

  • Stable Diffusion Model

    • Products similar to ComfyUI

    • Deploy Stable Diffusion Model

    • For Stable Diffusion Model, integrate stable_diffusion.cpp (inference submodule, manually constructing computation graphs).

    • High-performance OP

    • Distributed

      • In scenarios where multiple models jointly complete a task, schedule multiple models to multiple machines for distributed execution.

      • In large model scenarios, by splitting the large model into multiple sub-models, schedule multiple sub-models to multiple machines for distributed execution.

  • Open source operation

    • Promotion

    • Attract developers

    • Balance your time and nurturing developers:

    • Technical depth or product

    • Company landing

Contributors

Thanks to the following contributors:

@Alwaysssssss, @youxiudeshouyeren, @02200059Z, @JoDio-zd, @qixuxiang, @100312dog, @CYYAI, @PeterH0323, @zjhellofss, @zhouhao03, @jaywlinux, @ChunelFeng, @acheerfulish, @wangzhaode, @wenxin-zhao

Join us

  • NNDeply is developed and maintained by a group of like-minded friends. We regularly discuss technology and share industry insights. Currently, NNDeply is in the development stage. If you love open source, enjoy challenges, whether for learning purposes or have better ideas, welcome to join us.

  • WeChat: titian5566 (You can add my WeChat to join the NNDeply exchange group, note: nndeploy + name)