stable diffusion

1 Goal

1.1. Deploy stable diffusion series models based on DAG to form products similar to comfyui. (Can form products, directly for CV AIGC users)

1.2. Understand and implement stable diffusion from the bottom up (algorithm->framework->algorithm), forming products similar to ggml, llama.cpp, stable_diffusion.cpp

2 Why

Mainly from the following three aspects

2.1 Learning aspect

2.1.1. From the perspective of inference deployment, understand the stable diffusion series models

2.1.2. Based on stable diffusion, understand and implement the inference submodule from 0 to 1

2.1.3. High-performance op development (mainly cuda)

2.2 Research aspect

Understand and implement stable diffusion from the perspectives of algorithm, deployment, and inference, which can be used for some research, for example

2.2.1. High-performance operators - similar to flash attention operators, based on triton development

2.2.2. Inference submodule - efficient memory management, graph optimization, etc.

2.2.3. Quantization of stable diffusion models

2.3 Product aspect

2.3.1. Provide users with products similar to comfyui

2.3.2. For stable diffusion model deployment, provide another option

Summary: Through this open-source framework, understand and implement stable diffusion from the bottom up (algorithm->framework->algorithm).

3 How

3.1 Algorithm side

3.1.1 Research phase

1. Detailed analysis of classic stable diffusion models

   1. Algorithm principles - reaching a qualified level

   2. Training principles (dataset, loss) - reaching a qualified level

   3. Detailed analysis of model structure - reaching a good level

      1. There are several models

      2. How models are concatenated

      3. Pre and post-processing of models

      4. Pre and post-processing of models, inference parameters

      5. Structure of each model

3.1.2 Development phase (optional)

1. Based on mainstream frameworks, run stable diffusion inference

   1. pytorch

   2. hugging face

   3. stable_diffusion.cpp

   4. stable fast

   5. confyui

   6. sd webui

   7. …

3.1.3 Advanced phase (optional)

1. Can roughly understand the basic principles of mainstream framework implementations

3.2 Inference submodule (This part I am more familiar with, hence written more)

Through our understanding of algorithm principles, implement stable diffusion in a vertical manner. The complete inference framework includes the following submodules, we will omit most modules, focusing on the parts we consider most indispensable - implementing model inference by manually building computation graphs (comprehensive ggml+stable_diffusion.cpp+tensorrt api+mnn+tnn)

• Model conversion (convert training framework model files into weight files)

  • Brief: Convert training framework model files into custom model files for inference framework

  • Input: Training framework model files

  • Output: Custom model files

• Model interpretation (optional)

  • Brief: Read model files and interpret them into intermediate representations

  • Input: Custom model files

  • Output: Intermediate representation IR (to do)

• Manually build computation graph (to do)

  • Brief: Similar to ggml and tensorrt api, manually build computation graphs

  • Input:

    • Understanding of the model’s network structure

    • Weight files

  • Output

    • Computation graph

• Model inference initialization (to do)

  • Brief: Prepare everything for model inference

    • The most core part of the entire inference framework, model inference depends on this part, model offline quantization also depends on this part

  • Input:

    • Computation graph or intermediate representation IR (Option)

    • Inference hyperparameters

  • Output

    • An executable inference computation graph

• Model inference (to do)

  • Brief: Write input, get output

  • Input: Input tensor

  • Output: Inferred tensor

• Model post-training quantization (optional)

  • Brief: Quantize model files to make them smaller, expecting models to run faster

  • Input: fp32 model files

  • Output: int8 model files

• Model training (not for now, optional)

  • Brief: Can implement model training

  • Input:

    • Build model files through operator expressions

    • Hyperparameters

  • Output: Trainable model online

3.2.1 Research phase

1. Understand the implementation of inference frameworks

   1. ggml+stable_diffusion.cpp (preliminary understanding)

   2. tensorrt api (preliminary understanding)

   3. mnn (preliminary understanding)

   4. tnn (preliminary understanding)

   5. stable fast (optional)

   6. …

3.2.2 Development phase

1. Basic data structures - data_type\data_format\pricision etc. (nndeploy has completed)

2. Data container - already have an initial version of tensor definition, next step planned

   1. Merge mat and tensor, redesign

      1. The previous reason for considering separation was:

      2. Tensor mainly serves the input/output of the model, intermediate variables. That is, it focuses on aspects like shape, data_format, will focus on extensibility (such as service and quantization Tensor, distributed Tensor, etc., based on model’s memory pool allocation, etc.); Tensor’s properties basically do not consider shallow copy, deep copy, etc.

      3. Mat is mainly a structure proposed to replace opencv’s op in the future, will focus on operational simplicity, such as shallow copy, deep copy, etc., and there is no requirement for data_format.

      4. Now considering merging because:

         1. CV class operators can be added to the inference submodule to build graphs together;

         2. Using the inference submodule, CV class operators themselves can also build graphs.

         3. Slightly understanding CUDA, the data structure inside it is Tensor

   2. Consider the serialization of tensor’s protobuf (optional)

   3. Consider pipeline parallel (optional)

   4. Consider distribution (optional)

3. Design of IR

   1. Refer to ONNX which has preliminarily implemented a version

   2. Refer to MindIr for further optimization

   3. Consider multiple weight model files (optional)

   4. Consider protobuf serialization

4. Manual graph construction

   1. Form of interface

      1. OP

      2. Forward

   2. Consider manual graph construction (based on reasoning understanding, refer to existing DAG implementation)

   3. Consider automatic graph construction from IR later (optional)

5. Model reasoning initialization

   1. Delegate to executor

   2. Memory pre-allocation

   3. Graph optimization (optional)

3.2.3 Advanced stage (optional)

1. Distribution

2. Automatic graph construction

3.3 High-performance operators

Supported platforms

• CUDA

• OpenCL

• CPU

Four requirements

• Automatic graph construction (ready to use after encapsulation)

• Manual graph construction (ready to use after encapsulation)

• Can be exported as interfaces similar to BLAS or OpenCV for calling (ready to use after encapsulation)

• How to test alignment

Implementation method

• Library call

  • cuDNN

  • oneAPI

  • QNNPACK

  • …

• Implementation based on Triton

• Handwritten

Method of testing operators

• Compare with whom

• How to compare