推理的基类 More...

#include <inference.h>

Collaboration diagram for nndeploy::inference::Inference:

Public Member Functions
virtual bool	canOpInput ()
	是否可以操作推理框架内部分配的输入tensor More...

virtual bool	canOpOutput ()
	是否可以操作推理框架内部分配的输出tensor More...

virtual base::Status	deinit ()=0
	反初始化推理 More...

virtual base::ShapeMap	getAllInputShape ()
	Get the Output Shape object. More...

virtual std::map< std::string, device::Tensor * >	getAllInputTensorMap ()
	Get the All Input Tensor Map object. More...

virtual std::vector< std::string >	getAllInputTensorName ()
	Get the All Input Tensor Name object. More...

virtual std::vector< device::Tensor * >	getAllInputTensorVector ()
	Get the All Input Tensor Vector object. More...

virtual std::map< std::string, device::Tensor * >	getAllOutputTensorMap ()
	Get the All Output Tensor Map object. More...

virtual std::vector< std::string >	getAllOutputTensorName ()
	Get the All Output Tensor Name object. More...

virtual std::vector< device::Tensor * >	getAllOutputTensorVector ()
	Get the All Output Tensor Vector object. More...

base::DeviceType	getDeviceType ()

virtual float	getGFLOPs ()
	获得推理计算量 More...

base::InferenceType	getInferenceType ()

virtual std::string	getInputName (int i)
	Get the Input Name object. More...

virtual base::IntVector	getInputShape (const std::string &name)
	Get the Input Shape object. More...

virtual device::Tensor *	getInputTensor (const std::string &name)
	Get the Input Tensor object. More...

virtual device::TensorDesc	getInputTensorAlignDesc (const std::string &name)
	Get the Input Tensor Align Desc object. More...

virtual device::TensorDesc	getInputTensorDesc (const std::string &name)
	Get the Input Tensor Desc object. More...

base::ShapeMap	getMaxShape ()
	针对动态输入的推理，获取输入tensor的max_shape More...

virtual int64_t	getMemorySize ()
	获取推理所需的内存大小 More...

base::ShapeMap	getMinShape ()
	针对动态输入的推理，获取输入tensor的min_shape More...

virtual int	getNumOfInputTensor ()
	Get the Num Of Input Tensor object. More...

virtual int	getNumOfOutputTensor ()
	Get the Num Of Output Tensor object. More...

base::ShapeMap	getOptShape ()
	针对动态输入的推理，获取输入tensor的opt_shape More...

virtual std::string	getOutputName (int i)
	Get the Output Name object. More...

virtual device::Tensor *	getOutputTensor (const std::string &name)
	Get the Output Tensor object. More...

virtual device::Tensor *	getOutputTensorAfterRun (const std::string &name, base::DeviceType device_type, bool is_copy, base::DataFormat data_format=base::kDataFormatAuto)=0
	Get the Output Tensor object. More...

virtual device::TensorDesc	getOutputTensorAlignDesc (const std::string &name)
	Get the Output Tensor Align Desc object. More...

virtual device::TensorDesc	getOutputTensorDesc (const std::string &name)
	Get the Output Tensor Desc object. More...

base::Param *	getParam ()
	Get the Inference Param（这里使用基类指针） More...

std::shared_ptr< base::Param >	getParamSharedPtr ()

device::Stream *	getStream ()

	Inference (base::InferenceType type)

virtual base::Status	init ()=0
	初始化推理 More...

virtual bool	isBatch ()
	是否为多batch推理 More...

virtual bool	isInputDynamic ()
	是否为动态输入 More...

virtual bool	isOutputDynamic ()
	是否为动态输出 More...

virtual bool	isShareContext ()
	该推理实例是否与nndeploy共享一个context More...

virtual bool	isShareStream ()
	该推理实例是否与nndeploy共享一个stream More...

virtual base::Status	reshape (base::ShapeMap &shape_map)=0
	针对动态输入的推理，设置输入tensor的shape More...

virtual base::Status	run ()=0
	推理 More...

virtual base::Status	setInputTensor (const std::string &name, device::Tensor *input_tensor)
	Set the Input Tensor object. More...

virtual base::Status	setMemory (device::Buffer *buffer)
	设置推理所需的内存（推理内存由外部分配） More...

base::Status	setParam (base::Param *param)
	Set the Inference Param（这里使用基类指针） More...

base::Status	setParamSharedPtr (std::shared_ptr< base::Param > param)

void	setStream (device::Stream *stream)

virtual	~Inference ()

Protected Attributes
std::map< std::string, device::Tensor * >	external_input_tensors_
	外部输入tensor的map More...

std::shared_ptr< InferenceParam >	inference_param_
	推理框架的配置 More...

std::map< std::string, device::Tensor * >	input_tensors_
	输入tensor的map More...

bool	is_external_stream_ = false
	第三方推理框架是否与应用层共用一个stream More...

bool	is_share_context_ = true
	第三方推理框架是否与应用层共用一个context More...

std::map< std::string, device::Tensor * >	output_tensors_
	输出tensor的map More...

device::Stream *	stream_ = nullptr

base::InferenceType	type_
	推理框架的类型 More...

Detailed Description

推理的基类

根据InferencParam *param初始化

写入输入tensor数据

推理

得到输出tensor数据

其他

获取输入输出tensor的信息

动态输入

动态输出

是否可以操作推理框架分配的输入

是否可以操作推理框架分配的输出

获取推理初始化后的各种信息

例如内存大小

gflops等

Definition at line 40 of file inference.h.

Constructor & Destructor Documentation

◆ Inference()

nndeploy::inference::Inference::Inference ( base::InferenceType type )

◆ ~Inference()

virtual nndeploy::inference::Inference::~Inference ( )

virtual

Member Function Documentation

◆ canOpInput()

virtual bool nndeploy::inference::Inference::canOpInput ( )

virtual

是否可以操作推理框架内部分配的输入tensor

Returns: true

部分推理框架会为输入tensor分配内存，以TNN为例，其会为输入tensor分配内存，当为CPU推理时，可以操作这些内存但是当为OpenCL推理时，这些内存是在OpenCL上分配的，TNN无法与外部共享GPU上下文，故无法操作这些内存

◆ canOpOutput()

virtual bool nndeploy::inference::Inference::canOpOutput ( )

virtual

是否可以操作推理框架内部分配的输出tensor

Returns: true

部分推理框架会为输出tensor分配内存，以TNN为例，其会为输出tensor分配内存，当为CPU推理时，可以操作这些内存但是当为OpenCL推理时，这些内存是在OpenCL上分配的，TNN无法与外部共享GPU上下文，故无法操作这些内存

◆ deinit()

virtual base::Status nndeploy::inference::Inference::deinit ( )

pure virtual

反初始化推理

Returns: base::Status

◆ getAllInputShape()

virtual base::ShapeMap nndeploy::inference::Inference::getAllInputShape ( )

virtual

Get the Output Shape object.

Parameters

name

Returns: base::IntVector

◆ getAllInputTensorMap()

virtual std::map<std::string, device::Tensor *> nndeploy::inference::Inference::getAllInputTensorMap ( )

virtual

Get the All Input Tensor Map object.

Returns: std::map<std::string, device::Tensor *>

◆ getAllInputTensorName()

virtual std::vector<std::string> nndeploy::inference::Inference::getAllInputTensorName ( )

virtual

Get the All Input Tensor Name object.

Returns: std::vector<std::string>

◆ getAllInputTensorVector()

virtual std::vector<device::Tensor *> nndeploy::inference::Inference::getAllInputTensorVector ( )

virtual

Get the All Input Tensor Vector object.

Returns: std::vector<device::Tensor *>

◆ getAllOutputTensorMap()

virtual std::map<std::string, device::Tensor *> nndeploy::inference::Inference::getAllOutputTensorMap ( )

virtual

Get the All Output Tensor Map object.

Returns: std::map<std::string, device::Tensor *>

◆ getAllOutputTensorName()

virtual std::vector<std::string> nndeploy::inference::Inference::getAllOutputTensorName ( )

virtual

Get the All Output Tensor Name object.

Returns: std::vector<std::string>

◆ getAllOutputTensorVector()

virtual std::vector<device::Tensor *> nndeploy::inference::Inference::getAllOutputTensorVector ( )

virtual

Get the All Output Tensor Vector object.

Returns: std::vector<device::Tensor *>

◆ getDeviceType()

base::DeviceType nndeploy::inference::Inference::getDeviceType ( )

inline

Definition at line 64 of file inference.h.

◆ getGFLOPs()

virtual float nndeploy::inference::Inference::getGFLOPs ( )

virtual

获得推理计算量

Returns: float

◆ getInferenceType()

base::InferenceType nndeploy::inference::Inference::getInferenceType ( )

◆ getInputName()

virtual std::string nndeploy::inference::Inference::getInputName ( int i )

virtual

Get the Input Name object.

Parameters

i

Returns: std::string

◆ getInputShape()

virtual base::IntVector nndeploy::inference::Inference::getInputShape ( const std::string & name )

virtual

Get the Input Shape object.

Parameters

name

Returns: base::IntVector

◆ getInputTensor()

virtual device::Tensor* nndeploy::inference::Inference::getInputTensor ( const std::string & name )

virtual

Get the Input Tensor object.

Parameters

name

Returns: device::Tensor*

◆ getInputTensorAlignDesc()

virtual device::TensorDesc nndeploy::inference::Inference::getInputTensorAlignDesc ( const std::string & name )

virtual

Get the Input Tensor Align Desc object.

Parameters

name

Returns: device::TensorDesc

◆ getInputTensorDesc()

virtual device::TensorDesc nndeploy::inference::Inference::getInputTensorDesc ( const std::string & name )

virtual

Get the Input Tensor Desc object.

Parameters

name

Returns: device::TensorDesc

◆ getMaxShape()

base::ShapeMap nndeploy::inference::Inference::getMaxShape ( )

针对动态输入的推理，获取输入tensor的max_shape

Returns: base::ShapeMap

◆ getMemorySize()

virtual int64_t nndeploy::inference::Inference::getMemorySize ( )

virtual

获取推理所需的内存大小

Returns: int64_t

◆ getMinShape()

base::ShapeMap nndeploy::inference::Inference::getMinShape ( )

针对动态输入的推理，获取输入tensor的min_shape

Returns: base::ShapeMap

◆ getNumOfInputTensor()

virtual int nndeploy::inference::Inference::getNumOfInputTensor ( )

virtual

Get the Num Of Input Tensor object.

Returns: int

◆ getNumOfOutputTensor()

virtual int nndeploy::inference::Inference::getNumOfOutputTensor ( )

virtual

Get the Num Of Output Tensor object.

Returns: int

◆ getOptShape()

base::ShapeMap nndeploy::inference::Inference::getOptShape ( )

针对动态输入的推理，获取输入tensor的opt_shape

Returns: base::ShapeMap

◆ getOutputName()

virtual std::string nndeploy::inference::Inference::getOutputName ( int i )

virtual

Get the Output Name object.

Parameters

i

Returns: std::string

◆ getOutputTensor()

virtual device::Tensor* nndeploy::inference::Inference::getOutputTensor ( const std::string & name )

virtual

Get the Output Tensor object.

Parameters

name

Returns: device::Tensor*

◆ getOutputTensorAfterRun()

virtual device::Tensor* nndeploy::inference::Inference::getOutputTensorAfterRun	(	const std::string &	name,
		base::DeviceType	device_type,
		bool	is_copy,
		base::DataFormat	data_format = `base::kDataFormatAuto`
	)

pure virtual

Get the Output Tensor object.

Parameters

name

Returns: device::Tensor*

◆ getOutputTensorAlignDesc()

virtual device::TensorDesc nndeploy::inference::Inference::getOutputTensorAlignDesc ( const std::string & name )

virtual

Get the Output Tensor Align Desc object.

Parameters

name

Returns: device::TensorDesc

◆ getOutputTensorDesc()

virtual device::TensorDesc nndeploy::inference::Inference::getOutputTensorDesc ( const std::string & name )

virtual

Get the Output Tensor Desc object.

Parameters

name

Returns: device::TensorDesc

◆ getParam()

base::Param* nndeploy::inference::Inference::getParam ( )

Get the Inference Param（这里使用基类指针）

Returns: base::Param*

◆ getParamSharedPtr()

std::shared_ptr<base::Param> nndeploy::inference::Inference::getParamSharedPtr ( )

◆ getStream()

device::Stream* nndeploy::inference::Inference::getStream ( )

◆ init()

virtual base::Status nndeploy::inference::Inference::init ( )

pure virtual

初始化推理

Returns: base::Status

◆ isBatch()

virtual bool nndeploy::inference::Inference::isBatch ( )

virtual

是否为多batch推理

Returns: bool

◆ isInputDynamic()

virtual bool nndeploy::inference::Inference::isInputDynamic ( )

virtual

是否为动态输入

Returns: bool

◆ isOutputDynamic()

virtual bool nndeploy::inference::Inference::isOutputDynamic ( )

virtual

是否为动态输出

Returns: bool

◆ isShareContext()

virtual bool nndeploy::inference::Inference::isShareContext ( )

virtual

该推理实例是否与nndeploy共享一个context

Returns: bool

◆ isShareStream()

virtual bool nndeploy::inference::Inference::isShareStream ( )

virtual

该推理实例是否与nndeploy共享一个stream

Returns: bool

◆ reshape()

virtual base::Status nndeploy::inference::Inference::reshape ( base::ShapeMap & shape_map )

pure virtual

针对动态输入的推理，设置输入tensor的shape

Parameters

shape_map

Returns: base::Status

◆ run()

virtual base::Status nndeploy::inference::Inference::run ( )

pure virtual

推理

Returns: base::Status

◆ setInputTensor()

virtual base::Status nndeploy::inference::Inference::setInputTensor	(	const std::string &	name,
		device::Tensor *	input_tensor
	)

virtual

Set the Input Tensor object.

Parameters

name
input_tensor

Returns: base::Status

传入外部的tensor

◆ setMemory()

virtual base::Status nndeploy::inference::Inference::setMemory ( device::Buffer * buffer )

virtual

设置推理所需的内存（推理内存由外部分配）

Parameters

buffer

Returns: base::Status

◆ setParam()

base::Status nndeploy::inference::Inference::setParam ( base::Param * param )

Set the Inference Param（这里使用基类指针）

Parameters

param

Returns: base::Status

◆ setParamSharedPtr()

base::Status nndeploy::inference::Inference::setParamSharedPtr ( std::shared_ptr< base::Param > param )

◆ setStream()

void nndeploy::inference::Inference::setStream ( device::Stream * stream )

Member Data Documentation

◆ external_input_tensors_

std::map<std::string, device::Tensor *> nndeploy::inference::Inference::external_input_tensors_

protected

外部输入tensor的map

外部输入tensor的map，也可以是input_tensors_

Definition at line 377 of file inference.h.

◆ inference_param_

std::shared_ptr<InferenceParam> nndeploy::inference::Inference::inference_param_

protected

推理框架的配置

构造推理类时，也会构造一个默认的配置，可尽量减少用户需要配置的参数

Definition at line 342 of file inference.h.

◆ input_tensors_

std::map<std::string, device::Tensor *> nndeploy::inference::Inference::input_tensors_

protected

输入tensor的map

部分第三方推理框架（TNN、MNN、Openvino等）内部分配输入tensor，这是第三方推理框架内部分配的输入tensor的一个浅拷贝部分推理框架（TensorRt、onnxruntime等）内部不分配输入tensor（也或许是外部无法获得到内部分配的输入tensor），nndeploy的推理将尝试为第三方推理框架分配输入tensor

Definition at line 363 of file inference.h.

◆ is_external_stream_

bool nndeploy::inference::Inference::is_external_stream_ = false

protected

第三方推理框架是否与应用层共用一个stream

Definition at line 354 of file inference.h.

◆ is_share_context_

bool nndeploy::inference::Inference::is_share_context_ = true

protected

第三方推理框架是否与应用层共用一个context

Definition at line 348 of file inference.h.

◆ output_tensors_

std::map<std::string, device::Tensor *> nndeploy::inference::Inference::output_tensors_

protected

输出tensor的map

部分第三方推理框架（TNN、MNN、Openvino等）内部分配输出tensor，这是第三方推理框架内部分配的输出tensor的一个浅拷贝部分推理框架（TensorRt、onnxruntime等）内部不分配输出tensor（也或许是外部无法获得到内部分配的输出tensor），nndeploy的推理将尝试为第三方推理框架分配输出tensor

Definition at line 370 of file inference.h.

◆ stream_

device::Stream* nndeploy::inference::Inference::stream_ = nullptr

protected

Definition at line 355 of file inference.h.

◆ type_

base::InferenceType nndeploy::inference::Inference::type_

protected

推理框架的类型

Definition at line 336 of file inference.h.

The documentation for this class was generated from the following file:

/home/docs/checkouts/readthedocs.org/user_builds/nndeploy-zh/checkouts/stable/framework/include/nndeploy/inference/inference.h

Public Member Functions

Protected Attributes

Detailed Description

根据InferencParam *param初始化

写入输入tensor数据

推理

得到输出tensor数据

其他

获取输入输出tensor的信息

动态输入

动态输出

是否可以操作推理框架分配的输入

是否可以操作推理框架分配的输出

获取推理初始化后的各种信息

例如内存大小

gflops等

Constructor & Destructor Documentation

◆ Inference()

◆ ~Inference()

Member Function Documentation

◆ canOpInput()

◆ canOpOutput()

◆ deinit()

◆ getAllInputShape()

◆ getAllInputTensorMap()

◆ getAllInputTensorName()

◆ getAllInputTensorVector()

◆ getAllOutputTensorMap()

◆ getAllOutputTensorName()

◆ getAllOutputTensorVector()

◆ getDeviceType()

◆ getGFLOPs()

◆ getInferenceType()

◆ getInputName()

◆ getInputShape()

◆ getInputTensor()

◆ getInputTensorAlignDesc()

◆ getInputTensorDesc()

◆ getMaxShape()

◆ getMemorySize()

◆ getMinShape()

◆ getNumOfInputTensor()

◆ getNumOfOutputTensor()

◆ getOptShape()

◆ getOutputName()

◆ getOutputTensor()

◆ getOutputTensorAfterRun()

◆ getOutputTensorAlignDesc()

◆ getOutputTensorDesc()

◆ getParam()

◆ getParamSharedPtr()

◆ getStream()

◆ init()

◆ isBatch()

◆ isInputDynamic()

◆ isOutputDynamic()

◆ isShareContext()

◆ isShareStream()

◆ reshape()

◆ run()

◆ setInputTensor()

◆ setMemory()

◆ setParam()

◆ setParamSharedPtr()

◆ setStream()

Member Data Documentation

◆ external_input_tensors_

◆ inference_param_

◆ input_tensors_

◆ is_external_stream_

◆ is_share_context_

◆ output_tensors_

◆ stream_

◆ type_