常见问题

问题一 来自 nndeploy交流群

Q: 我想了解一下：onnx runtime也可以选择不同的execution provider: https://onnxruntime.ai/docs/execution-providers/ nndeploy与其思路的差别主要在哪里呢？

A: ONNXRuntime与nndeploy最大的区别是：ONNXRuntime首先是一个的推理框架，然后可以接入其他推理框架，nndeploy明确定位就是一个多端部署框架。这个会带来的差异主要是如下两点

  • ONNXRuntime为了保证其他推理框架兼容其自身，在功能上会对其他推理框架有一定程度的阉割（比如TNN、mnn操作推理框架内部分配的输入输出、OpenVINO CPU-GPU异构模式、模型量化等等），导致模型通过EP只是能跑起来，性能应该不好。nndeploy希望类似：你搞了很久的trt，然后你就trt搭建了自己的一个小型框架，可以完全操纵trt;

  • ONNXRuntime强绑定ONNX以及protobuf，会导致包体积变大，尤其是对于移动端不友好，对于不能直接读取onnx模型文件的框架应该也不太友好，还有就是现在很多框架都是pt->自定义模型文件（比如ncnnn\mnn），以达到极致的性能

总结而言，都是调用第三方推理框架，使用nndeploy功能更全面、性能更好、体验更丝滑。

问题二 来自 nndeploy交流群

Q: nndeploy相对于openmmlab的mmdeploy以及paddle的fastdeploy会有什么优势？

A: 非常开心可以和openmmlab的mmdeploy以及paddle的fastdeploy放在一起比较，毫无疑问，openmmlab的mmdeploy以及paddle的fastdeploy都是行业巨擘，有非常大影响力，据我所知都有非常多公司在使用这两个库。因为我们是nndeploy的开发者，所以我们这里主要谈谈我们的优势。

  • 更加开放。目前fd家重点是paddle的模型仓库以及主推自家推理框架，mmdp家主要是openmmlab的模型仓库以及主推自家推理框架，而nndeploy积极集成各家的推理框架，不设限的部署热门的开源模型。nndeploy想跳出这个“特定优先支持”，做一个更加通用普适的部署框架

  • 更加易用，在多端推理的基础功能的基础上nndeploy还做了设备管理，让你可以有统一的方式操作内存以及执行流等；还通过有向无环图来管理模型部署的前处理、推理、后处理

  • 更加适合部署多模型，nndeploy以有向无环图的形式来实现多个模型的部署，通过这种方式来部署多模型算法，可以少写大量的业务代码，模块性以及鲁棒性都会更好。

  • 推理框架的高性能抽象：每个推理框架也都有其各自的特性，需要足够尊重以及理解这些推理框架，才能在抽象中不丢失推理框架的特性，并做到统一的使用体验。nndeploy可配置第三方推理框架绝大部分参数，保证了推理性能。可直接操作理框架内部分配的输入输出，实现前后处理的零拷贝，提升模型部署端到端的性能。我们正在实现更多性能方面的优化，比如在图的基础上加上线程池、内存池、高性能算子库等，未来性能方面的功能会逐渐更加完善

  • nndeploy更加专注模型c++部署，更轻量化一些

问题三 来自 nndeploy交流群

Q: nndeploy与triton的区别

A: 主要有以下四点

  • triton侧重于对模型推理阶段的统一管理；nndeploy关注模型的端到端（包括前后处理）部署的统一管理；

  • triton屏蔽了内部推理框架接口，注重于对用户的推理框架黑盒，假定受众为不了解推理框架使用的普通用户；nndeploy注重于屏蔽不同推理框架的使用差异，但为开发人员保留对推理框架的控制能力，可以服务于高级用户，只是侧重于减少代码开发工作量，关注代码在不同推理框架的兼容性；

  • triton的优化主要是推理阶段的资源/任务调度，nndeploy的优化来自于全流程，包括资源/任务调度、前后处理与模型推理中的全流程内存优化、模型优化等；

  • 目前triton将所关注的功能封装为server，nndeploy当前版本暂未实现（未来将会有）

问题四 来自内部讨论

Q: 当前模型部署有哪些痛点

A: 主要有以下四点

  • 现在业界尚不存在各方面都远超其同类产品的推理框架，不同推理框架在不同平台、硬件下分别具有各自的优势。例如，在 Linux + NVidia 显卡机器推理，TensorRT 是性能最好的推理框架；在 Windows + x86 CPU 机器推理，OpenVINO 是性能最好的推理框架；在 ARM Android 下，有 ncnn、MNN、TFLite、TNN等一系列选择。

  • Different inference frameworks have varying inference interfaces and configuration APIs, requiring a separate set of code for each framework, which imposes significant learning, development, and maintenance costs on model deployment engineers.

  • Model deployment is not just about model inference; it also includes pre-processing and post-processing. Inference frameworks often only provide the functionality for model inference.

  • Currently, many scenarios require the combination of multiple models to solve business problems (such as stable diffusion, old photo restoration, face recognition, etc.). Directly using the native interfaces of inference frameworks can lead to a large amount of inefficient business code writing.

Discussion and exchange with HelloGithub egg yolk

1 Framework Introduction

Below are introductions to three open-source frameworks on their GitHub homepage.

1.1 nndeploy: nndeploy is an end-to-end model deployment framework. Based on multi-end inference and directed acyclic graph model deployment, it aims to provide users with a cross-platform, easy-to-use, high-performance model deployment experience.

1.2 fastdeploy: FastDeploy is an all-scenario, easy-to-use, extremely efficient AI inference deployment tool, supporting cloud-edge-end deployment. It offers out-of-the-box deployment experiences for over 160+ models in Text, Vision, Speech, and cross-modal categories, achieving end-to-end inference performance optimization. Including object detection, character recognition (OCR), face, image matting, multi-target tracking system, NLP, Stable Diffusion text-to-image generation, TTS, and dozens of other task scenarios, meeting developers’ needs for multi-scenario, multi-hardware, multi-platform industrial deployment.

1.2 mmdeploy: MMDeploy is OpenMMLab’s model deployment toolbox, providing a unified deployment experience for various algorithm libraries. Based on MMDeploy, developers can easily generate the required SDK for specific hardware from the training repo, saving a significant amount of adaptation time.

2 Framework Comparison

名称 易用性 性能 并行 开放性 模型支持 扩展性
nndeploy 高 高 支持 高 低 低
fastdeploy 中 中 不支持 中 高 高
mmdeploy 中 中 不支持 中 高 高

2.1 Ease of Use: Based on directed acyclic graph model deployment, the entire model deployment process (pre-processing -> inference -> post-processing) is straightforward; building inference templates Infer can shield the complexity of models; efficiently solving complex scenarios with multiple models by supporting graph embedding with flexible and powerful features, breaking down big problems into small ones, quickly solving complex multi-model scenario problems through combination; can quickly build demo

2.2 Performance: Abstracting the high performance of the inference framework; efficient thread pool

2.3 Parallelism: Based on directed acyclic graph model deployment, supporting various parallel modes, further enhancing the performance of model deployment

2.4 Openness: Currently, fastdeploy focuses on paddle’s model repository and mainly promotes its own inference framework, mmdeploy mainly on openmmlab’s model repository and its own inference framework, while nndeploy actively integrates various inference frameworks, not limiting the deployment of hot open-source models. nndeploy aims to jump out of this ‘specific priority support’, to become a more universal and widely applicable deployment framework

2.5 Model Support: Currently, nndeploy supports fewer models; fastdeploy supports the vast majority of models in paddle’s model repository; mmdeploy supports the vast majority of models in openmmlab’s model repository

2.6 Extensibility: nndeploy has a better architecture, can be extended into an inference framework, and also supports distributed inference deployment

About Me

Inference Framework: On Android Tee (Trusted Execution Environment), developed a pure C inference framework

Deployment Framework: From scratch, developed a deployment framework, currently serving hundreds of models online

Model Deployment: Completed the deployment of dozens of CV models on multiple ends

Detection Tracking: Have practical experience with traditional detection tracking algorithms

Open Source Framework:

  • Have read in detail four well-known inference frameworks: tnn, mnn, tengine, nnlib

  • Used inference frameworks like openvin, tensorrt, onnxruntime

  • ONNX model optimizers: onnxoptimizer, onnxsim

The Purpose of Open Source

Attempt to solve deployment pain points: Have been engaged in model deployment, inference, and high-performance computing related fields, discovered some pain points in model deployment, and want to try to solve these problems

Technical pursuit: Based on nndeploy, can realize some ideas on AI Infra, keeping pace with the rhythm of AI development

Broaden knowledge boundaries: Meet some friends, broaden knowledge boundaries through exchange

Influence: Maybe can provide some help to the industry, and also gain some influence