Developer Discussion

mac

  • Compilation without enabling any inference framework

  • Improve CoreML

    • Code

    • Compile

    • Invoke YOLOv5

  • Invoke MNN

    • Compile

    • Invoke YOLOv5

Nndeploy’s GitHub Actions

  • Nndeploy uses GitHub’s official Actions for compilation CI testing, encountering the following issues

    • Compilation fails on Mac, possibly due to Mac compilation issues

    • Unable to perform compilation testing with every push

MDC

  • Huawei’s chips can use device MDC and inference MDC

  • Device MDC

    • Is calling MDC reasonable? Should it be called its own programming model Ascend or ACL? (Similar to programming models CUDA, OpenCL) (This host-side API is very similar to CUDA, is this a kind of CUDA compatibility? Does its device-side code resemble CUDA?)

    • Can it be used on other chips? (For example, chips from the Kunpeng series)

  • Inference Framework MDC

    • Is calling MDC reasonable? What is its relationship with CANN?

    • Can it be used on other chips? (For example, chips from the Kunpeng series)

  • What is the relationship between this and MindSpore-Lite?

  • Variable names are all lowercase?

  • Is OrtValueInfo better to be changed?

  • The README needs to note the following, mainly which operating systems it can run on, which chips it can run on, and whether it should be added to the main page’s README

  • aclrt -> CUDA runtime API

  • aclmdl -> What does this mdl mean?

V2 modifications

  • Some header files use MDC and mdc

  • Variable names are all lowercase

  • Fields named ASCENDCL and AscendCL in the code are changed to -> AscendCL, for the following reasons

    • AscendCL is the official writing

    • Internal writing tends towards AscendCL, for example, OpenCL in devices, TensorRT in inference frameworks

  • Device class adds void* getContext() interface

    • For devices, there are usually concepts of context environment and execution queue

    • There is a requirement to get context in inference/ascend_cl

  • In the run function, data sets are recreated each time without destruction, is there a problem here?

base::Status AscendCLInference::run() {
  base::Status status = base::kStatusCodeOk;

  // Always: 
  input_dataset_ = aclmdlCreateDataset();
  output_dataset_ = aclmdlCreateDataset();
}

Pipeline Parallelism

Pipeline Parallelism Implementation Details Discussion

What is pipeline parallelism in nndeploy

Example description. On a certain Linux server, there are N 4090 graphics cards.

Photo album header clustering task: (Original image Edge) -> Face detection Node -> (Face Edge) -> Face keypoints Node -> (Face keypoints Edge) -> Face comparison Node -> (Face id Edge)

Note: Assume there are 1000 images in the album

Further explanation:

    1. Allocate three threads, each thread handles face detection (Node), face keypoints (Node), face comparison (Node) respectively

    1. (Original image) There is a data warehouse (Edge), when the data warehouse has data, face detection can start processing

    1. (Face) There is a data warehouse (Edge), when the data warehouse has data, face keypoints can start processing

    1. (Face keypoints) There is a data warehouse (Edge), when the data warehouse has data, face comparison can start processing

    1. (Face id) There is a data warehouse (Edge), when the data warehouse has data, the caller can get the data

Preliminary solution

  • Open a thread pool with three threads

  • Put all three nodes into the thread pool

  • Each thread execution must wait for (Original image Edge) (Face Edge) (Face keypoints Edge) to have data and the data has been successfully written before it can execute

  • Caller thread (Face id Edge) has data and the data has been successfully written

Specific difficulties and questions

  • For users, pipeline parallelism and serial execution are the same execution mode

  • When executing graph->run, just put the three nodes into the thread pool once

  • How does the main thread know it’s the last image, and then where to wait

  • The requirements for data between the start node and intermediate nodes are inconsistent, the start node input Edge has data to proceed, intermediate node input Edge has data and the data has been written (i.e., the previous node has completed writing to the output Edge). (Original image) There is a data warehouse (Edge) when the data warehouse has data, the first task can run

  • How to achieve data destruction

Document Structure Discussion

To achieve the following goals

  • Suitable for ReadTheDocs layout

  • Distinguish between Chinese and English documents

  • Be able to share deployment-related articles (I prefer to write articles under this folder)

  • TODO - API auto export

Document structure

  • TVM format - This is not ReadTheDocs format

  • MNN format

  • MMDeploy format

Based on the directory modification of Shouye Dazuo

  • Start

    • README (Want to write all these in the README, mainly to improve development efficiency, don’t want to repeatedly write the same articles)

      • Development and Prospects (Do these count as README?)

      • Currently supported hardware, inference frameworks (Do these count as README?)

      • Currently run-through demo algorithms (Do these count as README?)

      • Current existing features (Do these count as README?)

      • Things to do in the near future (Roadmap) (Do these count as README?)

      • Final vision (Do these count as README?)

    • Compile and install nndeploy

    • How to run nndeploy

    • Contribution Guide

      • PR rules, code specifications, WeChat group, perspectives from which contributions can be made, etc.

  • User Guide

    • Deploy a new algorithm

      • Export model

      • Construct pipeline

      • Pre-processing, post-processing

    • Add a new inference framework (Does this count as a user guide?)

    • Add a new device (Does this count as a user guide?)

    • Add a new op (Does this count as a user guide?)

  • Developer Guide

    • Overall architecture

    • The role and dependency relationships of major components corresponding to folders

    • Data container

      • Tensor, Mat, Buffer

    • Directed acyclic graph

      • graph, node, edge

    • Resource pool

      • Threadpool design

      • Memorypool design

    • Parallel mode

      • Pipeline Parallelism

      • Task parallelism

  • Knowledge sharing

    • Model export

    • Quantization

  • Problem discussion

DAG code review

  • Condition and loop are properties of Graph rather than Node

    • Graph inherits from Node

    • Compared to Node, the main differences of Graph are as follows

      • Graph has the capability to manage Nodes

      • Execution delegation of Graph to executor

  • Move executor.h to the main directory

  • And place the helper functions from executor.h into util

  • Naming modifications

    • condition_is_running -> runnint_condition

    • graph directory -> executor directory

    • condition_parallel_pipeline_executor -> parallel_pipeline_condition_executor

  • Resolve the issue of sporadic errors caused by Graph embedding Graph

  • Functionality verification

    • Each sub-module in a graph has its own parallel mode (serial, task parallel, pipeline parallel)

    • When an Edge is both the output of the entire graph and the input of some intermediate node, how to resolve this (main issue lies in pipeline parallel mode, to be reviewed and resolved)(yi)

DAG pipeline edge

Sub-graph within a graph

Main graph

  • Parameter Validation Phase

  • Mark Predecessors And Successors Phase

    • Modify the parallel attributes for each child node (node or graph)

      • If the child node already has parallel attributes, do not modify; otherwise, modify

  • Construct edge

Pre-release tasks

  • Code review

    • base - completed

    • device - completed

    • thread_pool - completed

    • inference - completed

    • DAG

      • Will embedding a graph within another graph cause errors during resource release?

        • It should not, Node and Edge are wrapped with a layer that includes resource management fields

      • Whether to add a field within the edge

        • Input edge - no producer

        • Input edge of intermediate node - has consumer and is not the output of the entire graph

        • Output edge - no consumer and is the output of the entire graph

        • Output edge is also the input edge of an intermediate node - has consumer and is the output of the entire graph

    • Model

Operator discussion (discussion with Night Owl)

  1. Implementation of Op classification, e.g., binary operators, nn class operators, mathematical class operators, activation function class operators, can share some functions like type checking

    1. Unary

    1. Binary

  • Elementwise

  • nn - compute

  • shape - concat

  1. Registration mechanism: register Op based on device

  • op_type + device (only keep this)

Distinguish within Op (not compilation, nor multiple subclasses - the most traditional way of functions, if else)

  • data_type + data_format (implement in one or two minutes)

  • opencl - image2d_t buffer

  1. Attribute layer: define attrs (param) for each Op, for unified parameter passing, also convenient for type checking (TVM’s way)

  2. Type checking includes the legality of data types (whether supported on a certain device), infer the size and dimensions of the output relay

Called during init

  1. Calculation layer (I also think) assumes all inputs, attributes, memory are legal, only focus on calculation

  2. Calling method () how to provide a clear interface, can easily see the parameters and content, order needed to pass

relay.nn.conv2d([inputs],kernel,stride)

Mapping relationship

conv2d_attrs={

} relay.call(”conv2d”,conv2d_attrs)

  1. Memory management: how Op and Edge collaborate no net op tensor

Modify: (this depends on your design) Does each operator need a header file?? Unify parameters as vector inputs, vector outputs, whether weights are passed as a member of inputs (unified interface layer)

attrs as an attribute assignment of the specific calculation class

Future planning

  • Inference submodule

  • Framework improvement

  • Python export

Model file

  • text -> json (rapidjson)

  • bin -> safetensors ()

llama

Graph construction

  • Model file - llama.safetensors

  • Model structure

    • llama.json

    • Expr mechanism

  • Note: ONNX will split the model structure very scattered

Add llama ir (rmsnorm\attention\matmul…)

  • D:\github\nndeploy\framework\include\nndeploy\ir\op_param.h

  • OpType

  • NNDEPLOY_CC_API std::string opTypeToString(OpType op_type);

  • NNDEPLOY_CC_API OpType stringToOpType(const std::string &op_type_name);

  • OpParam - onnx/pytorch/mindspore

  • REGISTER_OP_PARAM_IMPLEMENTION

op - data_type[fp32(this takes precedence),fp16,int8]/data_format[kDataFormatAuto]

  • cpu op

  • cuda op

  • ascend op

Memory management

  • kv block

llama discussion

  • onnx llama operators are very fragmented

    • selfattention, more than a dozen operators?

    • Graph optimization

  • Manual graph construction - constructing model structure through expr

    • fp32

    • fp16

    • int8

    • The weights are just safetensors