Add a new device¶

Introduction¶

Device is an abstraction of nndeploy hardware devices, which abstracts the differences between various hardware device program models, nndeploy currently supports devices such as CPU, X86, ARM, CUDA, AscendCL, etc. The main functions are as follows

Unified memory allocation: Provides a unified memory allocation interface for different devices, thereby simplifying the memory allocation of data containers Buffer, Mat, Tensor
Unified memory copy: Provides a unified memory copy interface for different devices (device-to-device copy, host-to-device copy), thereby simplifying the memory copy of data containers Buffer, Mat, Tensor
Unified synchronization operations: Provides a unified synchronization operation interface for different devices, which can simplify device end model promotion, calculation, and other synchronization operations
Unified hardware device information query: Provides a unified hardware device information query interface for different devices, helping users better select model full flow deployment execution devices

Step¶

Adding a new device is mainly divided into the following three steps:

(1) Add new device type hierarchy
(2) Inherit base class Architecture, inherit base class Device
(3) Modify cmake

Step one: Add new device type hierarchy¶

(1) Modify the file \include\nndeploy\base\common.h, add the new device hierarchy in DeviceTypeCode, format kDeviceTypeCodeXxx
(2) Modify the file \source\nndeploy\base\common.cc, add the string conversion to the new device hierarchy in the stringToDeviceTypeCode(const std::string &src) function
(3) Modify the file \include\nndeploy\base\status.h, add the new error hierarchy in StatusCode, format kStatusCodeErrorDeviceXxx

Step two: Inherit base class Architecture, implement XxxArchitecture¶

2.1 Add new files¶

(1) Create a new xxx_device.h file under \include\nndeploy\device\xxx
(2) Create a new xxx_device.cc file under \source\nndeploy\device\xxx
(3) [Optional] Create a new xxx_include.h file under \source\nndeploy\device\xxx, used to include the device’s necessary header files

2.2 Inherit base class Architecture, implement XxxArchitecture¶

(1) Declare the XxxArchitecture class under \include\nndeploy\device\xxx\xxx_device.h, similar to \include\nndeploy\device\cuda\cuda_device.h\CudaArchitecture

class CudaArchitecture : public Architecture {
 public:
  /**
   * @brief Construct a new Cuda Architecture object
   *
   * @param device_type_code
   */
  explicit CudaArchitecture(base::DeviceTypeCode device_type_code);

  /**
   * @brief Destroy the Cuda Architecture object
   *
   */
  virtual ~CudaArchitecture();

  /**
   * @brief Check whether the device corresponding to the current device id
   * exists, mainly serving GPU devices
   *
   * @param device_id - device id
   * @param library_path - Mainly serving OpenCL, using the OpenCL dynamic
   * library provided by the user
   * @return base::Status
   */
  virtual base::Status checkDevice(int device_id = 0,
                                 std::string library_path = "") override;

  /**
   * @brief Enable the device corresponding to the current device id，mainly
   * serving GPU devices
   *
   * @param device_id - device id
   * @param library_path - Mainly serving OpenCL, using the OpenCL dynamic
   * library provided by the user
   * @return base::Status
   */
  virtual base::Status enableDevice(int device_id = 0,
                                  std::string library_path = "") override;

  /**
   * @brief Get the Device object
   *
   * @param device_id
   * @return Device*
   */
  virtual Device *getDevice(int device_id) override;

  /**
   * @brief Get the Device Info object
   *
   * @param library_path
   * @return std::vector<DeviceInfo>
   */
  virtual std::vector<DeviceInfo> getDeviceInfo(
      std::string library_path = "") override;
};  

(2) Register the new ArchitectureTypeArchitectureRegister xxx_architecture_register(base::kDeviceTypeCodeXxx); in \source\nndeploy\device\xxx\xxx_device.cc, similar to TypeArchitectureRegister cuda_architecture_register(base::kDeviceTypeCodeCuda); in
(3) Implement XxxArchitecture under \source\nndeploy\device\xxx\xxx_device.cc, similar to the implementation in \source\nndeploy\device\cuda\cuda_device.cc\CudaArchitecture

2.3 Inherit base class Device, implement XxxDevice¶

(1) Declare the XxxDevice class under \include\nndeploy\device\xxx\xxx_device.h, similar to \include\nndeploy\device\cuda\cuda_device.h\CudaDevice

class NNDEPLOY_CC_API CudaDevice : public Device {
  /**
   * @brief friend class
   *
   */
  friend class CudaArchitecture;  

 public:
  /**
   * @brief Convert MatDesc to BufferDesc.
   *
   * @param desc
   * @param config
   * @return BufferDesc
   */
  virtual BufferDesc toBufferDesc(const MatDesc &desc,
                                  const base::IntVector &config); 

  /**
   * @brief Convert TensorDesc to BufferDesc.
   *
   * @param desc
   * @param config
   * @return BufferDesc
   */
  virtual BufferDesc toBufferDesc(const TensorDesc &desc,
                                  const base::IntVector &config);
  /**
   * @brief Allocate Buffer
   *
   * @param size
   * @return Buffer*
   */
  virtual Buffer *allocate(size_t size);
  /**
   * @brief Allocate Buffer
   *
   * @param desc
   * @return Buffer*
   */
  virtual Buffer *allocate(const BufferDesc &desc);
  /**
   * @brief Deallocate buffer
   *
   * @param buffer
   */
  virtual void deallocate(Buffer *buffer);  

  /**
   * @brief Copy buffer
   *
   * @param src - Device's buffer.
   * @param dst - Device's buffer.
   * @return base::Status
   * @note Ensure that the memory space of dst is greater than or equal to src.
   */
  virtual base::Status copy(Buffer *src, Buffer *dst,
                          int index = 0);
  /**
   * @brief Download memory from the device to the host.
   *
   * @param src - Device's buffer.
   * @param dst - Host's buffer.
   * @return base::Status
   * @note Ensure that the memory space of dst is greater than or equal to src.
   */
  virtual base::Status download(Buffer *src, Buffer *dst,
                          int index = 0);
  /**
   * @brief Upload memory from the host to the device.
   *
   * @param src - Host's buffer.
   * @param dst - Device's buffer.
   * @return base::Status
   * @note Ensure that the memory space of dst is greater than or equal to src.
   */
  virtual base::Status upload(Buffer *src, Buffer *dst,
                          int index = 0);  

  /**
   * @brief synchronize
   *
   * @return base::Status
   */brancg
  virtual base::Status synchronize(); 

  /**
   * @brief Get the Command Queue object
   *
   * @return void*
   */
  virtual void *getNativeStream();  

 protected:
  /**
   * @brief Construct a new Cuda Device object
   *
   * @param device_type
   * @param command_queue
   * @param library_path
   */
  CudaDevice(base::DeviceType device_type, void *command_queue = nullptr,
             std::string library_path = "")
      : Device(device_type), external_command_queue_(command_queue){};
  /**
   * @brief Destroy the Cuda Device object
   *
   */
  virtual ~CudaDevice(){};  

  /**
   * @brief init
   *
   * @return base::Status
   */
  virtual base::Status init();
  /**
   * @brief deinit
   *
   * @return base::Status
   */
  virtual base::Status deinit();  

 private:
  void *external_command_queue_ = nullptr;
  cudaStream_t stream_;
};

(2) Implement XxxDevice under \source\nndeploy\device\xxx\xxx_device.cc, similar to the implementation in \source\nndeploy\device\cuda\cuda_device.cc\CudaDevice

Step three: Modify cmake¶

(1) Modify the CMakeLists.txt file under \CMakeLists.txt,
- Add device compilation option nndeploy_option(ENABLE_NNDEPLOY_DEVICE_XXX “ENABLE_NNDEPLOY_DEVICE_XXX” OFF)
- Due to the addition of new devices, source files and header files have been added, it is necessary to incorporate the source files and header files into the translation files, and it is necessary to add the code block if(ENABLE_NNDEPLOY_DEVICE) endif() such as CMake source code.
```
if (ENABLE_NNDEPLOY_DEVICE_XXX)
  file(GLOB_RECURSE DEVICE_XXX_SOURCE
    "${ROOT_PATH}/include/nndeploy/device/xxx/*.h"
    "${ROOT_PATH}/source/nndeploy/device/xxx/*.cc"
  )
  set(DEVICE_SOURCE ${DEVICE_SOURCE} ${DEVICE_XXX_SOURCE})
endif()
```
(2) [Optional] If there is a need to link the device-related three-party libraries
- It is necessary to create a new xxx.cmake under the \cmake directory, similar to \cmake\ascend_cl.cmake or \cmake\cuda.cmake
- Modify \cmake\nndeploy.cmake, add include(”${ROOT_PATH}/cmake/xxx.cmake”)
(3) Modify \build\config.cmake, add device compilation option set(ENABLE_NNDEPLOY_DEVICE_XXX ON)