nndeploy::tokenizer Namespace Reference

Classes
class	TokenizerDecode
	TokenizerDecode. More...
class	TokenizerEncode
	TokenizerEncode. More...
class	TokenizerIds
class	TokenizerPraram
class	TokenizerText

Enumerations
enum	TokenizerType : int {   kTokenizerTypeHF = 0x0000 , kTokenizerTypeBPE , kTokenizerTypeSentencePiece , kTokenizerTypeRWKVWorld ,   kTokenizerTypeNotSupport }

Functions
TokenizerType	stringToTokenizerType (const std::string &src)
std::string	tokenizerTypeToString (TokenizerType type)

Enumeration Type Documentation

TokenizerType

enum nndeploy::tokenizer::TokenizerType : int

Enumerator
kTokenizerTypeHF	Create HF tokenizer from a single in-memory json blob. Parameters json_blob The json blob. Returns The created tokenzier.
kTokenizerTypeBPE	Create BPE tokenizer. Parameters vocab_blob The blob that contains vocabs. merges_blob The blob that contains the merges. added_tokens The added tokens. Returns The created tokenizer.
kTokenizerTypeSentencePiece	Create SentencePiece. Parameters model_blob The blob that contains vocabs. Returns The created tokenizer.
kTokenizerTypeRWKVWorld	Create RWKVWorldTokenizer. Parameters model_blob The blob that contains vocabs. Returns The created tokenizer.
kTokenizerTypeNotSupport

Definition at line 31 of file tokenizer.h.

Function Documentation

stringToTokenizerType()

TokenizerType nndeploy::tokenizer::stringToTokenizerType

(

const std::string &

src

)

tokenizerTypeToString()

std::string nndeploy::tokenizer::tokenizerTypeToString

(

TokenizerType

type

)