MLGraphBuilder operations

Experimental: This is an experimental technology. Check the browser compatibility table carefully before using it in production.

The operations of the Web Neural Network (WebNN) API are methods of the MLGraphBuilder interface. Each operation takes one or more MLOperand arguments, usually together with an options dictionary, and returns a new MLOperand (or a sequence of operands) that describes the result. Operations do not compute anything when called: they add a node to the graph that build() later compiles and MLContext.dispatch() executes.

This page documents all 95 operations, grouped by category. For the builder itself, including input(), constant() and build(), see MLGraphBuilder.

Overview

Common behavior

The following rules apply to every operation on this page, so they are not repeated in each entry.

Options and labels

Every options dictionary derives from MLOperatorOptions and therefore accepts an optional label string. Implementations may use it to make error messages and debugging output more useful. The options argument is always optional, except where an entry says otherwise.

const y = builder.matmul(a, b, { label: 'decoder.proj' });

Exceptions

Operations are validated synchronously, at the point of the call. Unless an entry lists other behavior, an operation throws:

• InvalidStateError DOMException

  The builder cannot build: build() has already been called, or the builder’s context is lost.

• TypeError

  An operand was created by a different MLGraphBuilder; an operand’s data type is not one of the operation’s allowed data types; an operand’s rank or shape is not valid for the operation (for example, shapes that cannot be broadcast, or an axis outside the valid range); or an option value is out of range.

Data types and ranks

Each entry has a tensor limits table with one row per operand (the output is listed in italics):

• Allowed data types are the MLOperandDataType values the specification permits. same as input means the operand must have the same data type as another operand, and any means every data type is allowed.
• Required data types are the values every implementation must support at minimum. An implementation may support more of the allowed types, but not fewer than the required ones. Models that use only required types are portable across implementations.
• Allowed ranks and required ranks work the same way for the number of dimensions. N stands for any rank.

The data types and ranks that the current context really supports are available at run time from MLContext.opSupportLimits(). The returned dictionary has one member per operation, named after the method (for example limits.conv2d or limits.add), with the dictionary type given in each entry.

const limits = context.opSupportLimits();
const canUseFp16 = limits.softmax.input.dataTypes.includes('float16');
const maxRank = limits.add.a.rankRange.max;

Broadcasting

Several operations combine operands of different shapes using the NumPy broadcasting rule . The shapes are aligned from the trailing dimension, and for each pair of dimensions the sizes must either be equal or one of them must be 1.

• Bidirectionally broadcastable: both operands may be stretched. The output rank is the larger of the two ranks and each output dimension is the larger of the two input sizes. Used by the element-wise binary and logical operations, prelu() and where().
• Unidirectionally broadcastable: only the first operand is stretched to match a target shape. Used by expand() and by the c operand of gemm().

Element-wise binary operations

Element-wise binary operations compute a result from two input tensors, one pair of elements at a time. The inputs must be bidirectionally broadcastable.

Syntax

add(a, b)
add(a, b, options)
// Likewise for sub(), mul(), div(), max(), min() and pow()

Parameters

• a

  An MLOperand: the first input tensor.

• b

  An MLOperand: the second input tensor.

• options optional

  An MLOperatorOptions dictionary.

Return value

An MLOperand holding the result. Its rank is the larger of the ranks of a and b, and each dimension is the larger of the corresponding input dimensions.

Tensor limits

Support limits key: add, sub, mul, div, max, min, pow (MLBinarySupportLimits).

add()

Element-wise addition: a + b.

sub()

Element-wise subtraction: a - b.

mul()

Element-wise multiplication: a * b.

div()

Element-wise division: a / b.

max()

Element-wise maximum of a and b.

min()

Element-wise minimum of a and b.

pow()

Element-wise power: each element of a raised to the power of the corresponding element of b.

Example

const a = builder.input('a', { dataType: 'float32', shape: [2, 3] });
const bias = builder.constant(
  { dataType: 'float32', shape: [3] }, new Float32Array([0.1, 0.2, 0.3]));
 
// bias is broadcast across the first dimension: the output shape is [2, 3].
const y = builder.add(a, bias);
console.log(y.shape); // [2, 3]

Element-wise logical operations

Comparison and logical operations produce a uint8 tensor in which each element is 1 (true) or 0 (false). Operations with two operands require bidirectionally broadcastable inputs.

Syntax

equal(a, b)            // notEqual, greater, greaterOrEqual, lesser, lesserOrEqual
logicalAnd(a, b)       // logicalOr, logicalXor
logicalNot(a)
isNaN(a)               // isInfinite
// Each accepts an optional trailing options argument.

Parameters

• a

  An MLOperand: the first (or only) input tensor.

• b

  An MLOperand: the second input tensor. Not used by logicalNot(), isNaN() and isInfinite().

• options optional

  An MLOperatorOptions dictionary.

Return value

An MLOperand of data type uint8. For two-operand operations the rank and dimensions follow the broadcasting rule; for single-operand operations the shape equals that of a.

Tensor limits

equal(), notEqual(), greater(), greaterOrEqual(), lesser(), lesserOrEqual(): support limits key of the same name (MLBinarySupportLimits).

logicalNot(): support limits key logicalNot (MLLogicalNotSupportLimits, with members a and output).

logicalAnd(), logicalOr(), logicalXor(): support limits key of the same name (MLBinarySupportLimits).

isNaN(), isInfinite(): support limits key of the same name (MLLogicalNotSupportLimits).

equal()

1 where a equals b, otherwise 0.

notEqual()

1 where a does not equal b, otherwise 0.

greater()

1 where a is greater than b, otherwise 0.

greaterOrEqual()

1 where a is greater than or equal to b, otherwise 0.

lesser()

1 where a is less than b, otherwise 0.

lesserOrEqual()

1 where a is less than or equal to b, otherwise 0.

logicalNot()

Logical negation of a.

logicalAnd()

Logical AND of a and b.

logicalOr()

Logical OR of a and b.

logicalXor()

Logical exclusive OR of a and b.

isNaN()

1 where the element of a is not-a-number (NaN), otherwise 0.

isInfinite()

1 where the element of a is positive or negative infinity, otherwise 0.

Example

// Keep only values greater than a threshold, zero elsewhere.
const x = builder.input('x', { dataType: 'float32', shape: [4] });
const threshold = builder.constant('float32', 0.5);
const zero = builder.constant('float32', 0);
 
const mask = builder.greater(x, threshold);   // uint8: 1 or 0
const y = builder.where(mask, x, zero);       // see where()

Element-wise unary operations

Element-wise unary operations apply a function to each element of one input tensor. The output has the same shape as the input.

Syntax

abs(input)
abs(input, options)
// Likewise for ceil(), cos(), erf(), exp(), floor(), identity(), log(), neg(),
// reciprocal(), roundEven(), sign(), sin(), sqrt() and tan()

Parameters

• input

  An MLOperand: the input tensor.

• options optional

  An MLOperatorOptions dictionary.

Return value

An MLOperand with the same shape as input. Its data type equals that of input.

Tensor limits

Support limits key: the method name (MLSingleInputSupportLimits).

abs(), neg(), sign()

ceil(), cos(), erf(), exp(), floor(), log(), reciprocal(), roundEven(), sin(), sqrt(), tan()

identity()

abs()

The absolute value of each element.

ceil()

The ceiling of each element: the smallest integer greater than or equal to it.

cos()

The cosine of each element (in radians).

erf()

The error function  of each element.

exp()

The exponential, e raised to the power of each element.

floor()

The floor of each element: the largest integer less than or equal to it.

identity()

Copies each element of the input to the output unchanged. Use it to expose an input or constant operand as a graph output, or to give an operand a separate node.

log()

The natural logarithm of each element.

neg()

The numerical negative (-x) of each element.

reciprocal()

The reciprocal (1 / x) of each element.

roundEven()

Rounds each element to the nearest integer, with halves rounded to the nearest even value (“banker’s rounding”). For example, [0.1, 0.9, 1.1, 1.9, -3.5, -2.5, -1.5, 1.5, 2.5, 3.5] yields [0, 1, 1, 2, -4, -2, -2, 2, 2, 4].

sign()

The sign of each element: 1 if greater than zero, -1 if less than zero, and 0 otherwise.

sin()

The sine of each element (in radians).

sqrt()

The square root of each element.

tan()

The tangent of each element (in radians).

Example

const x = builder.input('x', { dataType: 'float32', shape: [3] });
 
// Euclidean norm of a vector built from unary and reduction operations.
const norm = builder.sqrt(builder.reduceSum(builder.mul(x, x)));
console.log(norm.shape); // []

Activation functions

Activation functions are applied element-wise (except softmax() and prelu(), which are noted below) and are typically placed after a convolution or matrix multiplication. Unless stated otherwise, each takes one input tensor and returns a tensor of the same shape and data type.

All of the following use MLSingleInputSupportLimits under the support limits key of the same name, except prelu(), which has its own dictionary.

elu()

The exponential linear unit: max(0, x) + alpha * (exp(min(0, x)) - 1).

elu(input)
elu(input, options)

• input

  An MLOperand.

• options optional

  An MLEluOptions dictionary (derived from MLOperatorOptions).

gelu()

The Gaussian error linear unit: 0.5 * x * (1 + erf(x / sqrt(2))).

gelu(input)
gelu(input, options)

hardSigmoid()

A non-smooth approximation of the sigmoid function that is cheaper to compute: max(0, min(1, alpha * x + beta)).

hardSigmoid(input)
hardSigmoid(input, options)

An MLHardSigmoidOptions dictionary accepts:

hardSwish()

The function introduced by MobileNetV3: x * max(0, min(6, x + 3)) / 6.

hardSwish(input)
hardSwish(input, options)

leakyRelu()

The leaky rectified linear function: max(0, x) + alpha * min(0, x).

leakyRelu(input)
leakyRelu(input, options)

An MLLeakyReluOptions dictionary accepts:

linear()

A linear function: alpha * x + beta.

linear(input)
linear(input, options)

An MLLinearOptions dictionary accepts:

prelu()

The parametric rectified linear function: max(0, x) + slope * min(0, x). Unlike leakyRelu(), the slope is a tensor, typically learned during training. slope must be bidirectionally broadcastable to input, and the output rank is the larger of the two ranks.

prelu(input, slope)
prelu(input, slope, options)

• input

  An MLOperand: the input tensor.

• slope

  An MLOperand: the slope tensor.

Support limits key prelu (MLPreluSupportLimits, with members input, slope and output).

relu()

The rectified linear function: max(0, x).

relu(input)
relu(input, options)

sigmoid()

The sigmoid function: 1 / (exp(-x) + 1).

sigmoid(input)
sigmoid(input, options)

softmax()

The softmax  values of the input along the given axis, so that the values along that axis are positive and sum to 1.

softmax(input, axis)
softmax(input, axis, options)

• input

  An MLOperand: the N-D input tensor.

• axis

  An unsigned long: the dimension along which softmax is computed. It must be in the range 0 to N-1, where N is the rank of input.

softplus()

The softplus function: ln(1 + exp(x)).

softplus(input)
softplus(input, options)

softsign()

The softsign function: x / (1 + abs(x)).

softsign(input)
softsign(input, options)

tanh()

The hyperbolic tangent: (exp(2 * x) - 1) / (exp(2 * x) + 1).

tanh(input)
tanh(input, options)

Example

const logits = builder.input('logits', { dataType: 'float32', shape: [1, 10] });
const probabilities = builder.softmax(logits, 1);
 
const x = builder.input('x', { dataType: 'float32', shape: [1, 64] });
const slope = builder.constant({ dataType: 'float32', shape: [64] }, slopeData);
const activated = builder.prelu(x, slope);

Convolution operations

Convolutions slide a filter (kernel) over a 4-D input tensor. The memory layout of the input, output, and filter is selected with the inputLayout and filterLayout options. The layout that the platform handles most efficiently is reported by MLOpSupportLimits.preferredInputLayout.

Both operations use MLConv2dSupportLimits under their own support limits key, with members input, filter, bias and output.

conv2d()

Computes a 2-D convolution of a 4-D input tensor with a 4-D filter tensor.

conv2d(input, filter)
conv2d(input, filter, options)

Parameters

• input

  An MLOperand: the input 4-D tensor, interpreted according to inputLayout.

• filter

  An MLOperand: the filter 4-D tensor, interpreted according to filterLayout and groups.

• options optional

  An MLConv2dOptions dictionary (derived from MLOperatorOptions):

filterLayout values:

Return value

An MLOperand: a 4-D tensor holding the convolution result, laid out according to inputLayout. Each spatial output dimension is calculated, with the division rounded down, as:

outputSize = 1 + (inputSize - (filterSize - 1) * dilation - 1 + beginningPadding + endingPadding) / stride

Tensor limits

Example

const input = builder.input('input', { dataType: 'float32', shape: [1, 3, 224, 224] });
const filter = builder.constant({ dataType: 'float32', shape: [32, 3, 3, 3] }, filterData);
const bias = builder.constant({ dataType: 'float32', shape: [32] }, biasData);
 
const y = builder.conv2d(input, filter, {
  strides: [2, 2],
  padding: [1, 1, 1, 1],
  bias,
});
console.log(y.shape); // [1, 32, 112, 112]

convTranspose2d()

Computes a 2-D transposed convolution (sometimes called deconvolution) of a 4-D input tensor with a 4-D filter tensor. It is commonly used to upsample feature maps.

convTranspose2d(input, filter)
convTranspose2d(input, filter, options)

Parameters

• input

  An MLOperand: the input 4-D tensor, interpreted according to inputLayout.

• filter

  An MLOperand: the filter 4-D tensor, interpreted according to filterLayout and groups.

• options optional

  An MLConvTranspose2dOptions dictionary. It has the members padding, strides, dilations, groups, inputLayout and bias described for conv2d(), plus:

filterLayout values:

Return value

An MLOperand: a 4-D tensor holding the result, laid out according to inputLayout. Unless outputSizes is given, each spatial output dimension (rounded down) is:

outputSize = (inputSize - 1) * stride + (filterSize - 1) * dilation + 1 - beginningPadding - endingPadding + outputPadding

Tensor limits

Pooling operations

Pooling reduces the spatial dimensions of a 4-D tensor by computing one value for each position of a sliding window. The three operations share the same options and differ only in the function applied to the elements in each window.

Syntax

averagePool2d(input)
averagePool2d(input, options)
// Likewise for l2Pool2d() and maxPool2d()

Parameters

• input

  An MLOperand: the input 4-D tensor, interpreted according to layout.

• options optional

  An MLPool2dOptions dictionary (derived from MLOperatorOptions):

Return value

An MLOperand: a 4-D tensor laid out according to layout. Each spatial dimension is calculated, unless outputSizes is given, as:

outputSize = floor(1 + (inputSize - windowSize + beginningPadding + endingPadding) / stride)

where ceil replaces floor when outputShapeRounding is "ceil".

Tensor limits

Support limits key: averagePool2d, l2Pool2d, maxPool2d (MLSingleInputSupportLimits).

averagePool2d(), l2Pool2d()

maxPool2d()

averagePool2d()

Computes the average of the values in each window.

l2Pool2d()

Applies the L2 norm to each window: the square root of the sum of the squares of its elements.

maxPool2d()

Computes the maximum of the values in each window.

Example

const x = builder.input('x', { dataType: 'float32', shape: [1, 64, 56, 56] });
 
const halved = builder.maxPool2d(x, { windowDimensions: [2, 2], strides: [2, 2] });
console.log(halved.shape); // [1, 64, 28, 28]
 
// Global average pooling: the window defaults to the full spatial size.
const pooled = builder.averagePool2d(x);
console.log(pooled.shape); // [1, 64, 1, 1]

Normalization operations

Normalization operations rescale activations to have a controlled mean and variance. They differ in which values the statistics are computed over.

batchNormalization()

Normalizes the input using batch normalization . The mean and variance of each feature, computed across the batch during training, are supplied as operands for inference: (input - mean) / sqrt(variance + epsilon) * scale + bias.

batchNormalization(input, mean, variance)
batchNormalization(input, mean, variance, options)

Parameters

• input

  An MLOperand: the input N-D tensor.

• mean

  An MLOperand: a 1-D tensor of the mean values of the input features, with size equal to the size of the input dimension denoted by axis.

• variance

  An MLOperand: a 1-D tensor of the variance values of the input features, with the same size.

• options optional

  An MLBatchNormalizationOptions dictionary:

Return value

An MLOperand: the normalized tensor, with the same shape as input.

Tensor limits

Support limits key batchNormalization (MLBatchNormalizationSupportLimits, with members input, mean, variance, scale, bias and output).

instanceNormalization()

Normalizes the input using instance normalization . Unlike batch normalization, the mean and variance are computed on the fly for each feature of each individual sample.

instanceNormalization(input)
instanceNormalization(input, options)

Parameters

• input

  An MLOperand: the input 4-D tensor.

• options optional

  An MLInstanceNormalizationOptions dictionary:

Return value

An MLOperand: the normalized 4-D tensor, with the same shape as input.

Tensor limits

Support limits key instanceNormalization (MLNormalizationSupportLimits, with members input, scale, bias and output).

layerNormalization()

Normalizes the input using layer normalization . The mean and variance are computed on the fly across the features of each individual sample. It is widely used in transformer models.

layerNormalization(input)
layerNormalization(input, options)

Parameters

• input

  An MLOperand: the input N-D tensor.

• options optional

  An MLLayerNormalizationOptions dictionary:

Return value

An MLOperand: the normalized tensor, with the same shape as input.

Tensor limits

Support limits key layerNormalization (MLNormalizationSupportLimits).

Example

const hidden = builder.input('hidden', { dataType: 'float32', shape: [1, 128, 768] });
const gamma = builder.constant({ dataType: 'float32', shape: [768] }, gammaData);
const beta = builder.constant({ dataType: 'float32', shape: [768] }, betaData);
 
// Normalize over the last dimension only.
const normalized = builder.layerNormalization(hidden, {
  axes: [2],
  scale: gamma,
  bias: beta,
  epsilon: 1e-12,
});

Matrix operations

gemm()

Computes the general matrix multiplication  from BLAS: alpha * A * B + beta * C. A is a 2-D tensor of shape [M, K] (or [K, M] when transposed), B is [K, N] (or [N, K]), and C is unidirectionally broadcastable to [M, N].

gemm(a, b)
gemm(a, b, options)

Parameters

• a

  An MLOperand: the first 2-D tensor, of shape [M, K], or [K, M] if aTranspose is true.

• b

  An MLOperand: the second 2-D tensor, of shape [K, N], or [N, K] if bTranspose is true.

• options optional

  An MLGemmOptions dictionary:

Return value

An MLOperand: a 2-D tensor of shape [M, N].

Tensor limits

Support limits key gemm (MLGemmSupportLimits, with members a, b, c and output).

Example

// A fully connected layer: y = x * W^T + bias
const x = builder.input('x', { dataType: 'float32', shape: [8, 512] });
const w = builder.constant({ dataType: 'float32', shape: [1000, 512] }, weights);
const bias = builder.constant({ dataType: 'float32', shape: [1000] }, biasData);
 
const y = builder.gemm(x, w, { c: bias, bTranspose: true });
console.log(y.shape); // [8, 1000]

matmul()

Computes the matrix product of two tensors. For 2-D inputs of shape [M, K] and [K, N] the output has shape [M, N]. If either input has more than two dimensions it is treated as a stack of matrices in its last two dimensions; the leading (batch) dimensions must be bidirectionally broadcastable, and the output rank is the larger of the two ranks.

matmul(a, b)
matmul(a, b, options)

Parameters

• a

  An MLOperand: the first tensor, at least 2-D.

• b

  An MLOperand: the second tensor, at least 2-D. The size of its second-to-last dimension must equal the size of the last dimension of a.

• options optional

  An MLOperatorOptions dictionary.

Return value

An MLOperand holding the matrix product.

Tensor limits

Support limits key matmul (MLBinarySupportLimits).

Example

// Attention scores for 12 heads: [1, 12, 128, 64] x [1, 12, 64, 128]
const q = builder.input('q', { dataType: 'float32', shape: [1, 12, 128, 64] });
const kT = builder.input('kT', { dataType: 'float32', shape: [1, 12, 64, 128] });
const scores = builder.matmul(q, kT);
console.log(scores.shape); // [1, 12, 128, 128]

triangular()

Returns the upper or lower triangular part of a matrix. If the input has more than two dimensions it is treated as a batch of matrices, and the result has the same shape. Elements outside the retained part are set to zero. It is typically used to build causal attention masks.

triangular(input)
triangular(input, options)

Parameters

• input

  An MLOperand: the input tensor, at least 2-D.

• options optional

  An MLTriangularOptions dictionary:

Return value

An MLOperand with the same shape as input.

Tensor limits

Support limits key triangular (MLSingleInputSupportLimits).

Example

// input: [[7, 1, 2],
//         [9, 4, 8],
//         [2, 6, 3]]
const input = builder.constant(
  { dataType: 'float32', shape: [3, 3] },
  new Float32Array([7, 1, 2, 9, 4, 8, 2, 6, 3]));
 
builder.triangular(input);
// upper:            [[7, 1, 2], [0, 4, 8], [0, 0, 3]]
builder.triangular(input, { diagonal: 1 });
// upper, excluding the main diagonal: [[0, 1, 2], [0, 0, 8], [0, 0, 0]]
builder.triangular(input, { upper: false });
// lower:            [[7, 0, 0], [9, 4, 0], [2, 6, 3]]
builder.triangular(input, { upper: false, diagonal: 1 });
// lower, plus one diagonal above:     [[7, 1, 0], [9, 4, 8], [2, 6, 3]]

Reduction operations

Reduction operations collapse one or more dimensions of a tensor to a single value each, for example by summing or taking the maximum.

argMax() and argMin()

argMax() and argMin() return the index of the maximum or minimum value along an axis. When values tie, which index is returned is implementation-defined.

argMax(input, axis)
argMax(input, axis, options)
argMin(input, axis)
argMin(input, axis, options)

Parameters

• input

  An MLOperand: the input N-D tensor.

• axis

  An unsigned long: the dimension to reduce, in the range 0 to N-1.

• options optional

  An MLArgMinMaxOptions dictionary:

Return value

An MLOperand of data type outputDataType whose rank equals the input rank if keepDimensions is true, or the input rank minus 1 otherwise. Values are in the range 0 to S-1, where S is the size of the input along axis. A TypeError is thrown if S does not fit in outputDataType.

Tensor limits

Support limits key argMax, argMin (MLSingleInputSupportLimits).

argMax()

Index of the maximum value along axis.

argMin()

Index of the minimum value along axis.

cumulativeSum()

Computes the running (prefix) sum of values along an axis.

cumulativeSum(input, axis)
cumulativeSum(input, axis, options)

Parameters

• input

  An MLOperand: the input tensor.

• axis

  An unsigned long: the axis to sum along, in the range 0 to N-1.

• options optional

  An MLCumulativeSumOptions dictionary:

Return value

An MLOperand with the same shape as input.

Tensor limits

Support limits key cumulativeSum (MLSingleInputSupportLimits).

Reduce functions

The ten reduce*() operations share one signature and one set of options. Each applies a reduction function to the values across the reduced dimensions.

reduceSum(input)
reduceSum(input, options)
// Likewise for reduceL1(), reduceL2(), reduceLogSum(), reduceLogSumExp(),
// reduceMax(), reduceMean(), reduceMin(), reduceProduct() and reduceSumSquare()

Parameters

• input

  An MLOperand: the input tensor.

• options optional

  An MLReduceOptions dictionary:

Return value

An MLOperand of rank 0 to the input rank, depending on axes and keepDimensions. Reducing a scalar returns a scalar.

Tensor limits

Support limits key: the method name (MLSingleInputSupportLimits).

reduceL1(), reduceSum(), reduceSumSquare()

reduceL2(), reduceLogSum(), reduceLogSumExp(), reduceMean()

reduceMax(), reduceMin()

reduceProduct()

reduceL1()

The L1 norm : the sum of the absolute values.

reduceL2()

The L2 norm : the square root of the sum of the squares.

reduceLogSum()

The natural logarithm of the sum of the values.

reduceLogSumExp()

The natural logarithm of the sum of the exponentials of the values.

reduceMax()

The maximum of the values.

reduceMean()

The average of the values.

reduceMin()

The minimum of the values.

reduceProduct()

The product of the values.

reduceSum()

The sum of the values.

reduceSumSquare()

The sum of the squares of the values.

Example

const x = builder.input('x', { dataType: 'float32', shape: [2, 3, 4] });
 
const total = builder.reduceSum(x);                                  // shape []
const perRow = builder.reduceMean(x, { axes: [2] });                 // shape [2, 3]
const kept = builder.reduceMax(x, { axes: [1, 2], keepDimensions: true }); // shape [2, 1, 1]
const best = builder.argMax(x, 2);                                   // int32, shape [2, 3]

Shape and data movement operations

These operations change the type, shape, or arrangement of tensor data without (mostly) changing the values themselves.

cast()

Converts each element of the input to another data type.

cast(input, dataType)
cast(input, dataType, options)

• input

  An MLOperand: the input tensor.

• dataType

  An MLOperandDataType: the target data type.

• options optional

  An MLOperatorOptions dictionary.

Return value: an MLOperand with the same shape as input and data type dataType.

Support limits key cast (MLSingleInputSupportLimits).

// Turn a uint8 comparison mask into float32 so it can be multiplied.
const mask = builder.greater(x, threshold);
const maskF32 = builder.cast(mask, 'float32');
const y = builder.mul(x, maskF32);

clamp()

Clamps each element of the input to the range between a minimum and a maximum value.

clamp(input)
clamp(input, options)

• input

  An MLOperand: the input tensor.

• options optional

  An MLClampOptions dictionary:

Both limits are cast to the data type of input.

Return value: an MLOperand with the same shape as input.

Support limits key clamp (MLSingleInputSupportLimits).

// ReLU6
const relu6 = builder.clamp(x, { minValue: 0, maxValue: 6 });

concat()

Concatenates a list of tensors along an axis.

concat(inputs, axis)
concat(inputs, axis, options)

• inputs

  A sequence of MLOperand objects. All must have the same shape except in the dimension being concatenated, and the same data type.

• axis

  An unsigned long: the dimension to concatenate along, in the range 0 to N-1.

• options optional

  An MLOperatorOptions dictionary.

Return value: an MLOperand with the same shape as the inputs, except that the size along axis is the sum of the input sizes along that axis.

Support limits key concat (MLConcatSupportLimits, with members inputs and output).

const a = builder.input('a', { dataType: 'float32', shape: [1, 3] });
const b = builder.input('b', { dataType: 'float32', shape: [1, 5] });
console.log(builder.concat([a, b], 1).shape); // [1, 8]

expand()

Expands every dimension of size 1 of the input to a larger size, following the NumPy broadcasting rule. The input must be unidirectionally broadcastable to newShape: each of its dimensions is either 1 or equal to the corresponding dimension in newShape.

expand(input, newShape)
expand(input, newShape, options)

• input

  An MLOperand: the input tensor.

• newShape

  A sequence of unsigned long: the shape to expand to.

• options optional

  An MLOperatorOptions dictionary.

Return value: an MLOperand with shape newShape.

Support limits key expand (MLSingleInputSupportLimits).

pad()

Inflates a tensor by adding values at the edges of each dimension.

pad(input, beginningPadding, endingPadding)
pad(input, beginningPadding, endingPadding, options)

• input

  An MLOperand: the input tensor.

• beginningPadding

  A sequence of unsigned long of length N (the rank of input): how many values to add before the content in each dimension.

• endingPadding

  A sequence of unsigned long of length N: how many values to add after the content in each dimension.

• options optional

  An MLPadOptions dictionary:

MLPaddingMode values:

Return value: an MLOperand whose size in each dimension is beginningPadding + inputSize + endingPadding.

Support limits key pad (MLSingleInputSupportLimits).

// input: [[1, 2, 3],
//         [4, 5, 6]]
const input = builder.constant(
  { dataType: 'float32', shape: [2, 3] }, new Float32Array([1, 2, 3, 4, 5, 6]));
const begin = [1, 2];
const end = [1, 2];
 
builder.pad(input, begin, end);
// "constant": [[0,0,0,0,0,0,0], [0,0,1,2,3,0,0], [0,0,4,5,6,0,0], [0,0,0,0,0,0,0]]
 
builder.pad(input, begin, end, { mode: 'edge' });
// "edge":     [[1,1,1,2,3,3,3], [1,1,1,2,3,3,3], [4,4,4,5,6,6,6], [4,4,4,5,6,6,6]]
 
builder.pad(input, begin, end, { mode: 'reflection' });
// "reflection": [[6,5,4,5,6,5,4], [3,2,1,2,3,2,1], [6,5,4,5,6,5,4], [3,2,1,2,3,2,1]]

reshape()

Changes the logical shape of a tensor without copying or changing its contents. The number of elements implied by newShape must equal the number of elements in input.

reshape(input, newShape)
reshape(input, newShape, options)

• input

  An MLOperand: the input tensor.

• newShape

  A sequence of unsigned long: the shape of the output.

• options optional

  An MLOperatorOptions dictionary.

Return value: an MLOperand with the same values as input and shape newShape.

Support limits key reshape (MLSingleInputSupportLimits).

const x = builder.input('x', { dataType: 'float32', shape: [1, 64, 7, 7] });
const flat = builder.reshape(x, [1, 64 * 7 * 7]);

reverse()

Reverses the order of elements along the given axes.

reverse(input)
reverse(input, options)

• input

  An MLOperand: the input tensor.

• options optional

  An MLReverseOptions dictionary:

Return value: an MLOperand with the same shape as input.

Support limits key reverse (MLSingleInputSupportLimits).

slice()

Extracts a sub-tensor from the input.

slice(input, starts, sizes)
slice(input, starts, sizes, options)

• input

  An MLOperand: the input tensor.

• starts

  A sequence of unsigned long of length N: the index at which to start slicing in each dimension. Each must be in the range 0 to the dimension size minus 1.

• sizes

  A sequence of unsigned long of length N: the number of elements to take in each dimension. A size must not be 0, and start + size must not exceed the dimension size.

• options optional

  An MLSliceOptions dictionary:

Return value: an MLOperand with the same rank as input, containing the selected region.

Support limits key slice (MLSingleInputSupportLimits).

const x = builder.input('x', { dataType: 'float32', shape: [4, 6] });
// Rows 1-2, columns 0-3.
const part = builder.slice(x, [1, 0], [2, 4]);
console.log(part.shape); // [2, 4]

split()

Splits a tensor into several sub-tensors along an axis.

split(input, splits)
split(input, splits, options)

• input

  An MLOperand: the input tensor.

• splits

  An unsigned long, or a sequence of unsigned long. A number is the count of equally sized outputs, and must divide the size of input along axis evenly. A sequence gives the size of each output along axis, and its sum must equal the size of input along axis.

• options optional

  An MLSplitOptions dictionary:

Return value: a sequence of MLOperand. If splits is a number, the sequence has that many elements, each with the shape of input except along axis, where the size is the original size divided by splits. If splits is a sequence, there is one output per entry, and the size of output i along axis is splits[i].

Support limits key split (MLSplitSupportLimits, with members input and outputs).

const qkv = builder.input('qkv', { dataType: 'float32', shape: [1, 128, 2304] });
const [q, k, v] = builder.split(qkv, 3, { axis: 2 });
console.log(q.shape); // [1, 128, 768]

tile()

Repeats a tensor a given number of times along each dimension.

tile(input, repetitions)
tile(input, repetitions, options)

• input

  An MLOperand: the input N-D tensor.

• repetitions

  A sequence of unsigned long: how many times to repeat each dimension. Its length must equal the rank of input; use 1 for dimensions that should keep their size.

• options optional

  An MLOperatorOptions dictionary.

Return value: an MLOperand whose size in each dimension is the input size multiplied by the repetition count.

Support limits key tile (MLSingleInputSupportLimits).

transpose()

Permutes the dimensions of a tensor.

transpose(input)
transpose(input, options)

• input

  An MLOperand: the input N-D tensor.

• options optional

  An MLTransposeOptions dictionary:

Return value: an MLOperand whose dimension i is the input dimension permutation[i].

Support limits key transpose (MLSingleInputSupportLimits).

// NHWC to NCHW
const nhwc = builder.input('x', { dataType: 'float32', shape: [1, 224, 224, 3] });
const nchw = builder.transpose(nhwc, { permutation: [0, 3, 1, 2] });
console.log(nchw.shape); // [1, 3, 224, 224]

Gather and scatter operations

These operations read values from, or write values into, a tensor at positions given by an indices tensor. Index values must be of type int32, uint32 or int64, and may be negative to count from the end of a dimension: valid values lie in the range -S (inclusive) to S (exclusive), where S is the size of the indexed dimension.

gather()

Gathers values of the input along an axis according to the indices.

gather(input, indices)
gather(input, indices, options)

• input

  An MLOperand: the N-D tensor to gather from.

• indices

  An MLOperand: the tensor of indices into the dimension axis of input.

• options optional

  An MLGatherOptions dictionary:

Return value: an MLOperand of rank rank(input) + rank(indices) - 1.

Support limits key gather (MLGatherSupportLimits, with members input, indices and output).

// Embedding lookup: 30000 x 768 table, 16 token ids.
const table = builder.constant({ dataType: 'float32', shape: [30000, 768] }, embeddings);
const ids = builder.input('ids', { dataType: 'int32', shape: [16] });
const vectors = builder.gather(table, ids, { axis: 0 });
console.log(vectors.shape); // [16, 768]

gatherElements()

Gathers individual elements of the input along an axis: each output element is read from the position in input whose index along axis is given by the corresponding element of indices.

gatherElements(input, indices)
gatherElements(input, indices, options)

• input

  An MLOperand: the N-D tensor to gather from.

• indices

  An MLOperand: the indices, with the same rank as input.

• options optional

  An MLGatherOptions dictionary (see gather()); axis defaults to 0.

Return value: an MLOperand with the same rank as input.

Support limits key gatherElements (MLGatherSupportLimits).

gatherND()

Gathers slices of the input using full coordinates. The last dimension of indices holds the coordinate: an indices tensor of shape [10, 1] holds 10 single-axis indices, and shape [4, 3] holds 4 three-dimensional coordinates.

gatherND(input, indices)
gatherND(input, indices, options)

• input

  An MLOperand: the N-D tensor to gather from.

• indices

  An MLOperand: the coordinates into input. Each value must be in the range -S to S-1 (S is the size of the corresponding input dimension).

• options optional

  An MLOperatorOptions dictionary.

Return value: an MLOperand of rank rank(input) + rank(indices) - shape(indices)[-1] - 1.

Support limits key gatherND (MLGatherSupportLimits).

scatterElements()

Writes values from updates over a copy of input, at positions along an axis given by indices.

scatterElements(input, indices, updates)
scatterElements(input, indices, updates, options)

• input

  An MLOperand: the N-D tensor used to initialize the output.

• indices

  An MLOperand: the positions along axis to write to.

• updates

  An MLOperand: the new values, with the same shape as indices.

• options optional

  An MLScatterOptions dictionary:

Return value: an MLOperand with the same shape as input.

Support limits key scatterElements (MLScatterSupportLimits, with members input, indices, updates and output).

scatterND()

Writes slices from updates over a copy of input, using full coordinates. The last dimension of indices holds the coordinate, as for gatherND().

scatterND(input, indices, updates)
scatterND(input, indices, updates, options)

• input

  An MLOperand: the N-D tensor used to initialize the output.

• indices

  An MLOperand: the coordinates into the output to write to.

• updates

  An MLOperand: the new values or slices to write.

• options optional

  An MLOperatorOptions dictionary.

Return value: an MLOperand with the same shape as input.

Support limits key scatterND (MLScatterSupportLimits).

where()

Selects values element by element from trueValue or falseValue, depending on condition (non-zero is true, zero is false). The three tensors must be bidirectionally broadcastable. The condition is usually the output of a comparison or logical operation.

where(condition, trueValue, falseValue)
where(condition, trueValue, falseValue, options)

• condition

  An MLOperand: the condition tensor.

• trueValue

  An MLOperand: the values chosen where the condition is true.

• falseValue

  An MLOperand: the values chosen where the condition is false.

• options optional

  An MLOperatorOptions dictionary.

Return value: an MLOperand whose rank is the largest of the input ranks and whose dimensions are the largest of the input dimensions.

Support limits key where (MLWhereSupportLimits, with members condition, trueValue, falseValue and output).

// Causal attention mask: keep the lower triangle, fill the rest with -inf.
const scores = builder.input('scores', { dataType: 'float32', shape: [1, 12, 128, 128] });
const ones = builder.constant({ dataType: 'uint8', shape: [128, 128] }, new Uint8Array(128 * 128).fill(1));
const lower = builder.triangular(ones, { upper: false });
const masked = builder.where(lower, scores, builder.constant('float32', -Infinity));

Quantization operations

Quantization maps floating-point values to low-precision integers and back, which reduces model size and memory bandwidth. Both operations take a scale and a zeroPoint tensor that can be smaller than the input because they are blockwise broadcastable: they must have the same rank as input, and every dimension of input must be an exact multiple of the corresponding dimension of scale. This allows per-tensor, per-channel and per-block quantization.

Scale values must be positive and nonzero; otherwise the behavior is implementation-defined (correct results, incorrect results, or a compilation failure).

dequantizeLinear()

Converts a quantized integer tensor to floating point: output = (input - zeroPoint) * scale.

dequantizeLinear(input, scale, zeroPoint)
dequantizeLinear(input, scale, zeroPoint, options)

• input

  An MLOperand: the quantized input tensor.

• scale

  An MLOperand: the scale to multiply by, blockwise broadcastable with input.

• zeroPoint

  An MLOperand: the zero point to subtract, with the same shape as scale.

• options optional

  An MLOperatorOptions dictionary.

Return value: an MLOperand holding the dequantized values, with the shape of input and the data type of scale.

Support limits key dequantizeLinear (MLQuantizeDequantizeLinearSupportLimits, with members input, scale, zeroPoint and output).

quantizeLinear()

Converts a floating-point tensor to a quantized integer tensor. For example, for uint8 output: output = clamp(roundEven(input / scale) + zeroPoint, 0, 255).

quantizeLinear(input, scale, zeroPoint)
quantizeLinear(input, scale, zeroPoint, options)

• input

  An MLOperand: the floating-point input tensor.

• scale

  An MLOperand: the scale to divide by, blockwise broadcastable with input.

• zeroPoint

  An MLOperand: the zero point to add, with the same shape as scale. Its data type selects the output type.

• options optional

  An MLOperatorOptions dictionary.

Return value: an MLOperand holding the quantized values, with the shape of input and the data type of zeroPoint.

Support limits key quantizeLinear (MLQuantizeDequantizeLinearSupportLimits).

// Per-tensor int8 quantization round trip.
const x = builder.input('x', { dataType: 'float32', shape: [1, 16] });
const scale = builder.constant({ dataType: 'float32', shape: [1, 1] }, new Float32Array([0.05]));
const zeroPoint = builder.constant({ dataType: 'int8', shape: [1, 1] }, new Int8Array([0]));
 
const q = builder.quantizeLinear(x, scale, zeroPoint);   // int8
const restored = builder.dequantizeLinear(q, scale, zeroPoint); // float32

Recurrent network operations

Recurrent operations process a sequence of inputs one time step at a time while carrying hidden state. Each network type comes in two forms: a full operation that loops over all time steps (gru(), lstm()), and a cell operation that computes a single step (gruCell(), lstmCell()).

Shared enumerations:

In the shapes below, numDirections is 1 for "forward" and "backward", and 2 for "both".

gru()

Runs a Gated Recurrent Unit  network over a sequence, using an update gate, a reset gate, and a new gate.

gru(input, weight, recurrentWeight, steps, hiddenSize)
gru(input, weight, recurrentWeight, steps, hiddenSize, options)

Parameters

• input

  An MLOperand: a 3-D tensor of shape [steps, batchSize, inputSize].

• weight

  An MLOperand: a 3-D tensor of shape [numDirections, 3 * hiddenSize, inputSize]. The gate order in the second dimension follows layout.

• recurrentWeight

  An MLOperand: a 3-D tensor of shape [numDirections, 3 * hiddenSize, hiddenSize].

• steps

  An unsigned long: the number of time steps. It must be greater than 0.

• hiddenSize

  An unsigned long: the number of features in the hidden state.

• options optional

  An MLGruOptions dictionary:

Return value

A sequence of MLOperand. The first is a 3-D tensor of shape [numDirections, batchSize, hiddenSize]: the output of the last time step. If returnSequence is true, the second is a 4-D tensor of shape [steps, numDirections, batchSize, hiddenSize] with the output of every time step.

Tensor limits

Support limits key gru (MLGruSupportLimits, with members input, weight, recurrentWeight, bias, recurrentBias, initialHiddenState, output0 and output1).

gruCell()

Computes a single time step of a GRU network.

gruCell(input, weight, recurrentWeight, hiddenState, hiddenSize)
gruCell(input, weight, recurrentWeight, hiddenState, hiddenSize, options)

Parameters

• input

  An MLOperand: a 2-D tensor of shape [batchSize, inputSize].

• weight

  An MLOperand: a 2-D tensor of shape [3 * hiddenSize, inputSize].

• recurrentWeight

  An MLOperand: a 2-D tensor of shape [3 * hiddenSize, hiddenSize].

• hiddenState

  An MLOperand: a 2-D tensor of shape [batchSize, hiddenSize].

• hiddenSize

  An unsigned long: the number of features in the hidden state.

• options optional

  An MLGruCellOptions dictionary with the members resetAfter, layout and activations described for gru(), and:

Return value

An MLOperand: a 2-D tensor of shape [batchSize, hiddenSize] with the new hidden state.

Tensor limits

Support limits key gruCell (MLGruCellSupportLimits, with members input, weight, recurrentWeight, hiddenState, bias, recurrentBias and output).

lstm()

Runs a Long Short-Term Memory  network over a sequence, using input, output, forget, and cell gates.

lstm(input, weight, recurrentWeight, steps, hiddenSize)
lstm(input, weight, recurrentWeight, steps, hiddenSize, options)

Parameters

• input

  An MLOperand: a 3-D tensor of shape [steps, batchSize, inputSize].

• weight

  An MLOperand: a 3-D tensor of shape [numDirections, 4 * hiddenSize, inputSize]. The gate order follows layout.

• recurrentWeight

  An MLOperand: a 3-D tensor of shape [numDirections, 4 * hiddenSize, hiddenSize].

• steps

  An unsigned long: the number of time steps. It must be greater than 0.

• hiddenSize

  An unsigned long: the number of features in the hidden state.

• options optional

  An MLLstmOptions dictionary: