Architecture

Architecture Introduction

Architecture

Architecture Details

  • Directed Acyclic Graph: The model end-to-end deployment flow can be abstracted into 3 subgraphs: Model Frontend Processing+> Model Inference+> Model Backend Processing, which is an extremely typical directed acyclic graph. For multi-model ensemble algorithms, it is a more complex directed acyclic graph, directly writing business code to connect the entire process is not only error-prone but also has low efficiency. Using the directed acyclic graph method can greatly reduce the writing of business code.

  • Process Template: Front and backend processing templates as well as inference sub-templates. We hope to further simplify your deployment process, therefore in the model end-to-end deployment of Model Frontend Processing+> Model Inference+> Model Backend Processing, we take a step to calculate the template. Particularly, a lot of thought has been put into the inference template, corresponding to different models, there are many differences, such as single input, multi-output, static shape input, dynamic shape input, static shape output, dynamic shape output, whether the operable inference framework’s internal allocation of input and output etc., etc., only engineers with rich model deployment experience can quickly solve the above issues. In the past, we were based on multi-end inference modules Inference + Directed Acyclic Graph Node we then calculated a powerful inference template Infer, this template can help you describe the differences brought by different models internally.

  • Resource Pool: Resource management sub-module. Currently developing line program pool as well as memory pool (this section is a module that nndeploy is actively developing, looking forward to Dahu together to get things done). The line program pool can implement directed acyclic graph’s water line and execute, memory pool can achieve efficient memory allocation and release.

  • Parallel: Parallel module. Based on the directed acyclic graph’s parallel module, task parallel can shorten single algorithm full process execution time, water line parallel can increase algorithm full process execution throughput.

  • Inference: Multi-end inference sub-module (nndeploy still needs to build more inference framework modules, looking forward to Dahu together to get things done). Provide a unified inference interface to operate different inference backends, when packaging each inference framework module, we spent a lot of time to understand and study the characteristics of each inference framework module, such as TensorRT can use external memory inference, OpenVINO has a high throughput model, TNN can operate internal allocation of input and output etc. etc. In the process of abstraction, we will not lose the characteristics of the inference framework module, and achieve a unified usage experience, and also ensure performance.

  • OP: High performance compute sub-module. We plan to develop a high-performance front and backend compute module (looking forward to Dahu together to get things done), improve the performance from model end to end, also plan to develop an nn compute library or package oneDNN, QNN etc. compute libraries (not sure if there will be an inference framework module in nndeploy).

  • Data Container: Data container sub-module. The packaging of the inference framework’s API interface is not only the packaging, but also needs to design a Tensor, used to interact data with the third-party inference framework’s Tensor. nndeploy also designs a data container Mat for image processing, and designs multiple devices’ unified memory Buffer.

  • Device: Device management sub-module. Provide unified memory allocation, memory copy, execution flow management etc. operations for different devices.