ONEDNN Research

Supported Architectures:

Intel Architecture Processors, Intel Processor Graphics and Xe Architecture Graphics, Experimental Support for Arm* 64 (AArch64), OpenPOWER* Power ISA (PPC64)

• Intel 64 or AMD64,

• Arm 64-bit Architecture (AArch64).

• OpenPOWER / IBM Power Instruction Set.

• IBMz z/Architecture (s390x).

• RISC-V 64-bit (RV64).

CPU:

• Intel 64/AMD64 Architecture

  • Intel Atom(R) Processor (At least requires Intel SSE4.1 support)

  • Intel Core(TM) Processor (At least requires Intel SSE4.1 support)

  • Intel Ultra Core Processor (Originally Meteor Lake)

  • Intel Xeon(R) Processor E3, E5 and E7 Series (Previously known as Sandy Bridge, Ivy Bridge, Haswell and Broadwell)

  • Intel Xeon Scalable Processors (Previously known as Skylake, Cascade Lake, Cooper Lake, Ice Lake, Sapphire Rapids and Emerald Rapids)

  • Intel Xeon CPU Max Series (Previously known as Sapphire Rapids HBM)

  • Future Intel Scalable Processors (Code-named Sierra Forest and Granite Rapids)

• AArch64 Architecture

  • Arm Neoverse(TM) N1 and V1 Processors

GPU:

The library is optimized for the following GPUs:

• Intel Graphics for 11th to 14th Gen Intel Core Processors

• Intel Graphics for Intel Ultra Core Processors (Originally Meteor Lake)

• Intel Iris Xe MAX Graphics (Originally DG1)

• Intel Arc(TM) Graphics (Originally Alchemist)

• Intel Data Center GPU Flex Series (Originally Arctic Sound)

• Intel Data Center GPU Max Series (Originally Ponte Vecchio) Among them: Support for Nvidia is implemented through SYCL CUDA backend, requires:

• oneAPI DPC++ Compiler supporting CUDA or NVIDIA GPU

• NVIDIA CUDA* Driver

• cuBLAS 10.1 or higher

• cuDNN 7.6 or higher

Computational Support:

| CNN Primitives (Convolution, Inner Product, Pooling, etc.) | | ——————————– | | RNN Primitives (LSTM, Vanilla, RNN, GRU) | | Normalization (LRN, Batch Processing, Layer) | | Element-wise Operations (ReLU, Tanh, ELU, Abs, etc.) | | Softmax, Sum, Concat, Shuffle | | Reordering from Optimized Data Layout | | 8-bit Integer, 16-bit, 32-bit and bfloat16 Floating Point Data Types | | |

Core Concepts

The main concepts of oneDNN are primitives, engine, stream, and memory objects. Primitive: The primitive (dnnl::primitive) is an object that encapsulates a specific computation such as forward convolution, backward LSTM computation, or data transformation operation. A single primitive can sometimes represent more complex fused computations, such as a forward convolution operation followed immediately by a ReLU operation. Among other things, fusion is controlled through the primitive’s attribute mechanism. (The most important difference between primitives and pure functions is that primitives can store state.) 1. Part of the primitive’s state is immutable. For example, the convolution primitive stores parameters (such as tensor shapes) and can pre-compute other related parameters (such as cache blocking). This approach allows oneDNN primitives to generate specialized code tailored to the operations to be executed. The oneDNN programming model assumes that the time required for pre-computation can be amortized by reusing the same primitive multiple times for computation. 2. On this basis, microkernel extensions (low-level abstractions, implementing sequence block operations, allowing users to combine these block computations to implement custom operations) and graph extensions (high-level abstractions, which allow you to use computation graphs rather than individual primitives) can be performed.

Engine (dnnl::engine) is an abstraction of computing devices: CPU, specific GPU card in the system, etc. Most primitives are created to perform computations on a specific engine. The only exception is the reorder primitive, which can transfer data between two different engines.

Stream (dnnl::stream) encapsulates execution contexts bound to specific engines. For example, they can correspond to DPC++ command queues.

Memory object (dnnl::memory) encapsulates handles to memory allocated to specific engines, tensor dimensions, data types, and memory formats - the way tensor indices are mapped to offsets in linear memory space. Memory objects are passed to primitives during execution.

Data Layout Support

CUDA SYCL: NCDHW, NDHWC, NCHW, NHWC, NCW, NWC, NC

Data Format

Block Layout: For better vectorization and cache reuse, oneDNN introduces block layouts, splitting one or more dimensions into fixed-size blocks. The most popular oneDNN data formats are nChw16c on AVX512+ systems and nChw8c on SSE4.1+ systems. As the names suggest, the only dimension that is blocked is the channel, with block sizes of 16 and 8, respectively.

More precisely, the offset function for nChw8c is:

offset_nChw8c(n, c, h, w) = n * CHW
                          + (c / 8) * HW*8
                          + h * W*8
                          + w * 8
                          + (c % 8)

A simple example

Example code:

This C++ API example demonstrates the basic knowledge of oneDNN programming model:

· How to create a DNN memory object.

o How to put data from the user buffer into the oneDNN memory object.

o How the tensor’s logical dimensions and memory object formats are associated.

· How to create a DNN primitive.

· How to execute a primitive.

This example uses the ReLU operation, including the following steps:

1.     创建引擎和流来执行原语。

2.     执行数据准备（oneDNN 之外的代码）。

3.     将数据包装到 oneDNN 内存对象中

Creating dnln::memory involves two steps:

1.     初始化 dnnl::memory::desc 结构体（也称为内存描述符），该结构体仅描述张量数据，不包含数据本身。内存描述符用于创建 dnnl::memory 对象并初始化基元描述符（稍后在示例中显示）。

2.     基于步骤 1 中初始化的内存描述符、引擎以及可选的数据句柄创建 dnnl::memory 对象本身（也称为内存对象）。执行原语时会使用内存对象。

Memory Descriptor

auto src_md = memory::desc(
        {N, C, H, W}, // logical dims, the order is defined by a primitive
        memory::data_type::f32, // tensor's data type
        memory::format_tag::nhwc // memory format, NHWC in this case
);

Creating Memory Object

// src_mem contains a copy of image after write_to_dnnl_memory function
auto src_mem = memory(src_md, eng);
write_to_dnnl_memory(image.data(), src_mem);
// For dst_mem the library allocates buffer
auto dst_mem = memory(src_md, eng);

4.     创建 ReLU 原语。

1、创建一个操作原语描述符（此处为 dnnl::eltwise_forward::primitive_desc），它定义操作参数，并且是实现给定操作的实际算法的轻量级描述符。用户可以查询所选实现的不同特征，例如内存消耗以及下一个主题（内存格式传播）中将介绍的其他一些特征。

2、创建一个可以在内存对象上执行以计算操作的原语（此处为 dnnl::eltwise_forward）。

// ReLU primitive descriptor, which corresponds to a particular
// implementation in the library
auto relu_pd = eltwise_forward::primitive_desc(
        eng, // an engine the primitive will be created for
        prop_kind::forward_inference, algorithm::eltwise_relu,
        src_md, // source memory descriptor for an operation to work on
        src_md, // destination memory descriptor for an operation to work on
        0.f, // alpha parameter means negative slope in case of ReLU
        0.f // beta parameter is ignored in case of ReLU
);
// ReLU primitive
auto relu = eltwise_forward(relu_pd); // !!! this can take quite some time

5.     执行 ReLU 原语。

6.     获取结果并验证（检查生成的图像不包含负值）。

Memory Format Propagation

This example revolves around a CNN constructed from convolution and pooling, including the following steps:

1. Create a pooling primitive descriptor based on the memory format selected for the convolution primitive.

2. Create memory descriptors for input and output data in NCHW memory format.

// Tensor and kernel dimensions. We use the same 3x3 kernel with padding=1
// for both convolution and pooling primitives, which means that the
// activation tensor shapes do not change.
const int N = 1, H = 14, W = 14, IC = 128, OC = 256, KH = 3, KW = 3;
auto conv_src_md = memory::desc({N, IC, H, W}, memory::data_type::f32,
        memory::format_tag::any // let convolution choose memory format
);
auto conv_weights_md = memory::desc(
        {OC, IC, KH, KW}, memory::data_type::f32,
        memory::format_tag::any // let convolution choose memory format
);
auto conv_dst_md = memory::desc({N, OC, H, W}, memory::data_type::f32,
        memory::format_tag::any // let convolution choose memory format
);
const auto &pool_dst_md = conv_dst_md; // shape does not change
auto conv_pd = convolution_forward::primitive_desc(
        eng, prop_kind::forward_inference, algorithm::convolution_auto,
        conv_src_md, conv_weights_md,
        conv_dst_md, // shape information
        {1, 1}, // strides
        {1, 1}, {1, 1} // left and right padding
);
auto pool_pd
        = pooling_forward::primitive_desc(eng, prop_kind::forward_inference,
                algorithm::pooling_max, conv_pd.dst_desc(),
                pool_dst_md, // shape information
                {1, 1}, {KH, KW}, // strides and kernel
                {0, 0}, // dilation
                {1, 1}, {1, 1} // left and right padding
        );

1. Determine whether input and output data need to be reordered from the optimized memory format.

2. Create memory objects; and the necessary primitives and execute them.

auto conv_scratchpad_mem = memory(conv_pd.scratchpad_desc(), eng);

auto conv = convolution_forward(conv_pd);

conv.execute(s,

        {{DNNL_ARG_SRC, conv_src_mem}, {DNNL_ARG_WEIGHTS, conv_weights_mem},

                {DNNL_ARG_DST, conv_dst_mem}});

auto pool_scratchpad_mem = memory(pool_pd.scratchpad_desc(), eng);

auto pool = pooling_forward(pool_pd);

pool.execute(

        s, {{DNNL_ARG_SRC, conv_dst_mem}, {DNNL_ARG_DST, pool_dst_mem}});

s.wait();

Supported Primitive APIs:

• Convolution

• Inner Product

• Matrix Multiplication

• RNN

• Batch Normalization

• Binary

• Concat

• Eltwise

• Group Normalization

• Layer Normalization

• Local Response Normalization (LRN)

• Pooling

• PReLU

• Resampling

• Shuffle

• Softmax

• Sum

• Reorder

• Reduction

Multi-operator Demo (ocl)