14. Neural Nets

Read this chapter to see what a neural net is and how a small one learns, or to build one inside a MALDA program. 14.1 is the idea, with no library call. A picture reuses one small kernel (14.11.6). A larger net is a stack of smaller nets (14.10.1). The programs in Examples/AI_Theory/ train offline, with no API key. The chapter is optional. Skipping it leaves prompts, agents, and the rest of MALDA unchanged.

14.1 What a neural net is

A neural net is a function from a list of numbers to another list of numbers. It is built from units. Each unit multiplies its inputs by stored weights, adds a bias, and passes that sum through an activation. The weights and the bias are ordinary numbers you keep. Training changes those numbers. The shape of the function stays put.

You reach for a net when the rule is easier to show than to write. Four rows of AND fit in a table. A picture, a pole angle, or the next character in a word does not: there are too many inputs, and you still want an answer for an input you never listed. The weights are a compact guess at that rule, fitted to the examples you did list. The examples in this chapter are small enough to read on one page. A larger net runs the same loop.

14.1.1 One unit

One unit reads a vector x, keeps a weight for each entry, and keeps one bias. The weighted sum is z = math.dot(x, w) + b. The activation reads z and writes the unit's output.

A step activation returns 1 when z >= 0 and 0 otherwise. That unit is a perceptron. The weights below implement AND. Both inputs have to be 1 before the sum clears the bias -1.5. The four prints are the four corners, in the order [0, 0], [0, 1], [1, 0], [1, 1].

function stepUnit(x, w, b) {
    var z = math.dot(x, w) + b;
    if (z >= 0.0) {
        return 1;
    }
    return 0;
}
var w = [1.0, 1.0];
var b = -1.5;
print(stepUnit([0.0, 0.0], w, b));
print(stepUnit([0.0, 1.0], w, b));
print(stepUnit([1.0, 0.0], w, b));
print(stepUnit([1.0, 1.0], w, b));

Examples/AI_Theory/perceptron.malda searches for weights like these. AND and OR come out right. XOR stays wrong. The two outputs that should be 1 sit on opposite corners, and one straight cut leaves one of them on the 0 side. A hidden layer, with a bend between it and the output, can place both. That program is xor_neural_net.malda, a 2-4-1 net: 2 inputs, 4 units in the middle, 1 output.

14.1.2 Layers

A layer is several units that share the input and keep separate weights. Its output is one number per unit. The next layer reads that vector as its x. You write the target for the last layer. The middle layers are hidden. Their values are whatever makes the last layer match.

Between layers, an activation bends the sum. ReLU keeps the positive part and replaces the rest with 0. Sigmoid squashes the sum into a range near 0 to 1. That bend is why a later layer can combine cuts an earlier layer made. A stack of plain products, with no bend between them, collapses into one product, and one product is still one straight cut. The names and the formulas are 14.4.

14.1.3 Training, then using the net

Using the net is a forward pass. The inputs flow through each layer to the output, and the weights stay still. That pass, after training has finished, is inference.

Training is the loop that moves the weights. You pick a loss, one number that is small when the output matches the example. For a number you wanted, squared error is enough (nn.mse in 14.6). For a class label, the loss is high when the net puts its confidence on the wrong class. For one weight, the gradient is how much that loss grows when the weight grows a little. You subtract a fraction of the gradient from the weight. The fraction is the learning rate.

Each weight meets the rest of the net as two numbers: the input it multiplied, and the slope arriving from the unit after it. The chain rule writes the slope of the whole net as the product of those local slopes. Start at the output and multiply back toward the input. That product is backpropagation.

One input and two linear units. The input is 2. The hidden weight is 3 and the hidden bias is 0, so the hidden unit writes 6. The output weight is 0.5 and the output bias is 0, so the prediction is 3. The target is 1. The step uses the derivative of half the squared error, which is pred − target, here 3 − 1 = 2. 14.6 names that slope nn.mseGrad. Both units are linear, so each activation derivative is 1. The product that remains is the number the weight multiplied, times the slope arriving from the unit after it. The output weight multiplied 6, so its gradient is 6 × 2 = 12. The prediction was 0.5 times the hidden output, so the slope arriving at the hidden unit is 2 × 0.5 = 1. The hidden weight multiplied the input 2, so its gradient is 2 × 1 = 2. A learning rate times 12 comes off the output weight, and a learning rate times 2 comes off the hidden weight. The snippet prints the hidden output, the prediction, the output-weight gradient, the slope at the hidden output, and the hidden-weight gradient. The same numbers, as two library calls, are 14.5.1. Examples/AI_Theory/chain_rule.malda checks this product on a curve.

var x = 2.0;
var w1 = 3.0;
var b1 = 0.0;
var hidden = w1 * x + b1;
var w2 = 0.5;
var b2 = 0.0;
var pred = w2 * hidden + b2;
var dPred = pred - 1.0;
var dW2 = hidden * dPred;
var dHidden = w2 * dPred;
var dW1 = x * dHidden;
print(int(hidden));
print(int(pred));
print(int(dW2));
print(int(dHidden));
print(int(dW1));

One pass over the examples is an epoch. You repeat it. When you stop, a forward pass is the trained function. Examples/AI_Theory/ writes this loop in the order in 14.7. From 14.2 on, the same loop is nn.dense for the forward and nn.denseBackward for one layer's gradients. You still own the weights.

You keep the weights and the bias yourself. nn.dense is one matrix multiply, plus that bias, then an activation. nn.denseBackward returns the three gradients of that one layer and stops. It does not record an autograd tape, and it does not change your arrays. You subtract learningRate times each gradient, or you hand a fixed stack of layers to 14.10, which applies that same step for you.

A deeper net is the same pair of calls stacked. Run the output layer backward first. Its dInput is the upstream gradient of the layer below it. Both layers need the pre they returned on the way forward, because the activation derivative is evaluated there.

This chapter recomputes one forward, one backward, a two-row batch, the hand-off between two layers, both losses, one policy step, and one Sequential.fit step. The signature tables are 14.9 and 14.10. 14.7 is the order of the offline programs in Examples/AI_Theory/, which write the same products with named variables before a helper hides them.

14.2 What stays on math

Linear algebra stays on math. A weight matrix is [in, out]: one row per input unit, one column per output unit, the same layout as math.matmul(x, weights). The inner dimensions must match, so a vector of length 2 multiplies a matrix with 2 rows. The product below is the linear part of the layer in 14.3.1, before the bias [0.25, -0.5] is added. Activations and the layer live on nn. Write nn.relu, not relu: there is no flat alias.

print(math.dot([1.0, 2.0], [3.0, 1.0]));
print(int(math.matmul([1.0, 2.0], [[0.5, -1.0], [1.0, 0.0]])[0] * 10));
print(int(math.matmul([1.0, 2.0], [[0.5, -1.0], [1.0, 0.0]])[1] * 10));

14.3 One dense layer

nn.dense(x, weights, bias?, activation?) accepts one sample as a vector, or several samples as a matrix with one row per sample. The bias has one number per output. Omit it and the bias is zeros. The call returns { pre, out }.

pre is the pre-activation, the linear part before the nonlinearity: pre = xW + b (or XW + b for a batch). out is activation(pre). With no activation, or with "linear", out is a copy of pre and the local derivative is 1. Save pre. The backward step evaluates the activation derivative there, not on out.

14.3.1 Reading the matrix

Take x = [1, 2], weights [[0.5, -1], [1, 0]], and bias [0.25, -0.5]. Column 0 of the matrix is the weights into the first output: 0.5 from the first input and 1 from the second. Column 1 is the weights into the second output: -1 and 0.

The first pre-activation is 1×0.5 + 2×1 + 0.25 = 2.75. The second is 1×(-1) + 2×0 + (-0.5) = -1.5. ReLU leaves 2.75 and replaces -1.5 with 0, so out is [2.75, 0].

var layer = nn.dense([1.0, 2.0], [[0.5, -1.0], [1.0, 0.0]], [0.25, -0.5], "relu");
print(int(layer.pre[0] * 100));
print(int(layer.pre[1] * 100));
print(int(layer.out[0] * 100));
print(int(layer.out[1]));

14.3.2 A batch

Pass a matrix and each row is one sample. The same weights and the same bias apply to every row. pre and out keep that row layout, so pre[0] is the first sample. Two copies of [1, 2] each produce the pre-activation [2.75, -1.5] from 14.3.1.

The matching backward sums over the rows. Each row has dZ = [1, 0], so the bias gradient is [2, 0] and the weight gradient is [[2, 0], [4, 0]], twice the single-sample gradient. dInput stays one row per sample: the first row is still [0.5, 1]. Divide by the row count yourself when you want a mean. A learning rate tuned on one sample is large for many rows unless you scale it.

var x = [[1.0, 2.0], [1.0, 2.0]];
var w = [[0.5, -1.0], [1.0, 0.0]];
var layer = nn.dense(x, w, [0.25, -0.5], "relu");
var back = nn.denseBackward(x, w, [[1.0, 1.0], [1.0, 1.0]], "relu", layer.pre);
print(int(layer.pre[0][0] * 100));
print(int(layer.pre[0][1] * 100));
print(int(back.dBias[0]));
print(int(back.dBias[1]));
print(int(back.dWeights[0][0]));
print(int(back.dWeights[1][0]));
print(int(back.dWeights[0][1]));
print(int(back.dInput[0][0] * 10));
print(int(back.dInput[0][1]));

14.4 Activations

nn.dense accepts these names. The matching nn.d* call is the derivative with respect to the pre-activation z. At z = 0, ReLU's derivative is 0 (the test is z > 0).

The standalone calls nn.leakyRelu(x, alpha?) and nn.elu(x, alpha?) take an optional alpha. nn.dense does not: it always uses the defaults above. sigmoid, tanh, and softmax are also on math and call the same functions.

Save pre because several of these derivatives are a test on z, and out has already folded that test. ReLU's derivative at 0 is 0. At 3 it is 1. Sigmoid at 0 is 0.5, and its derivative there is 0.25. A pre-activation past about ±20 makes that derivative 0, which is why a deep stack of sigmoids needs a small initial scale (14.7.3).

print(int(nn.dRelu(0.0)));
print(int(nn.dRelu(3.0)));
print(int(nn.sigmoid(0.0) * 100));
print(int(nn.dSigmoid(0.0) * 100));

14.5 One backward step

nn.denseBackward(x, weights, upstream, activation?, pre?) returns { dInput, dWeights, dBias }. upstream is the gradient of the loss with respect to out, one number per output, in the same shape as out. The layer multiplies that vector by the activation derivative at pre and calls the result dZ. Then:

• dBias is dZ. On a batch it is the sum of the rows.
• dWeights[i][j] is input i times dZ[j]. That is the outer product of the input and dZ.
• dInput[i] is the sum over outputs of dZ[j] × weights[i][j]. That vector is what the previous layer receives as its upstream.

On the same layer as 14.3.1, ReLU's derivative is 1 at 2.75 and 0 at -1.5. An upstream of [1, 1] therefore becomes dZ = [1, 0]. The bias gradient is [1, 0]. The weight gradient is [[1, 0], [2, 0]] because the inputs are 1 and 2. The input gradient is [0.5, 1]: the dead unit drops the second column of W before that multiply. The snippet prints dInput[0] × 10, the two bias entries, then dWeights[0][0], dWeights[1][0], and dWeights[0][1].

var x = [1.0, 2.0];
var w = [[0.5, -1.0], [1.0, 0.0]];
var layer = nn.dense(x, w, [0.25, -0.5], "relu");
var back = nn.denseBackward(x, w, [1.0, 1.0], "relu", layer.pre);
print(int(back.dInput[0] * 10));
print(int(back.dBias[0]));
print(int(back.dBias[1]));
print(int(back.dWeights[0][0]));
print(int(back.dWeights[1][0]));
print(int(back.dWeights[0][1]));

14.5.1 A second layer

Run the output layer backward first. Its dInput is the hidden layer's upstream, and the hidden layer's forward out is the x that the output layer saw. Both layers need the pre they returned.

var hidden = nn.dense(x, w1, b1, "sigmoid");
var out = nn.dense(hidden.out, w2, b2, "sigmoid");
var upstream = nn.mseGrad(out.out, [target]);
var backOut = nn.denseBackward(hidden.out, w2, upstream, "sigmoid", out.pre);
var backHidden = nn.denseBackward(x, w1, backOut.dInput, "sigmoid", hidden.pre);

Examples/AI_Theory/xor_neural_net.malda writes those multiplies with named variables. Examples/AI_Theory/nn_dense.malda trains the same 2-4-1 XOR net with the two calls above.

These are the two linear units from 14.1.3. On a linear pair every local derivative is 1, so the hand-off is the product itself. The input is 2, the hidden weight is 3, and the hidden bias is 0, so the hidden output is 6. The output weight is 0.5 and the output bias is 0, so the prediction is 3. The target is 1, and nn.mseGrad is 3 − 1 = 2. The output weight gradient is 6 × 2 = 12. The output dInput is 2 × 0.5 = 1, and that 1 is the hidden layer's upstream. The hidden weight gradient is 2 × 1 = 2. The snippet prints the hidden output, the prediction, the output weight gradient, the hidden upstream, and the hidden weight gradient.

var x = [2.0];
var w1 = [[3.0]];
var b1 = [0.0];
var hidden = nn.dense(x, w1, b1, "linear");
var w2 = [[0.5]];
var b2 = [0.0];
var out = nn.dense(hidden.out, w2, b2, "linear");
var upstream = nn.mseGrad(out.out, [1.0]);
var backOut = nn.denseBackward(hidden.out, w2, upstream, "linear", out.pre);
var backHidden = nn.denseBackward(x, w1, backOut.dInput, "linear", hidden.pre);
print(int(hidden.out[0]));
print(int(out.out[0]));
print(int(backOut.dWeights[0][0]));
print(int(backOut.dInput[0]));
print(int(backHidden.dWeights[0][0]));

14.5.2 The weight step

There is no optimizer builtin. After the backward calls, subtract learningRate times dWeights from the weight matrix and learningRate times dBias from the bias, entry by entry. A batch bias gradient is a sum, so a learning rate tuned on one sample is large for many rows unless you scale it.

Take one input 2, weight 1, bias 0, and target 6, with a linear activation. The forward output is 2. nn.mseGrad is 2 − 6 = −4. The weight gradient is 2 × −4 = −8 and the bias gradient is −4. A learning rate of 0.25 moves the weight to 1 − 0.25 × (−8) = 3 and the bias to 1. The next forward is 3 × 2 + 1 = 7, closer to 6. The snippet prints the first output, the weight gradient, the updated weight, the updated bias, and the output after the step.

var w = [[1.0]];
var b = [0.0];
var layer = nn.dense([2.0], w, b, "linear");
var back = nn.denseBackward([2.0], w, nn.mseGrad(layer.out, [6.0]), "linear", layer.pre);
var lr = 0.25;
print(int(layer.out[0]));
print(int(back.dWeights[0][0]));
w[0][0] = w[0][0] - lr * back.dWeights[0][0];
b[0] = b[0] - lr * back.dBias[0];
print(int(w[0][0]));
print(int(b[0]));
var again = nn.dense([2.0], w, b, "linear");
print(int(again.out[0]));

14.6 Losses

On two numbers, nn.mse(pred, target) is the squared difference. On two vectors of the same length it is the mean of those squares. nn.mse(2, 0) is 4. nn.mse([2, 0], [0, 0]) is 2, the mean of 4 and 0.

nn.mseGrad(pred, target) is elementwise pred − target. That is the derivative of 1/2 (pred − target)², which is what nn_dense.malda accumulates as the epoch loss. The derivative of the value nn.mse itself returns is twice as large on one number, and it is also divided by the length on a vector. Pass mseGrad into denseBackward as upstream when the last layer is a regression output.

print(int(nn.mse(2.0, 0.0)));
print(int(nn.mseGrad(2.0, 0.0)));
print(int(nn.mse([2.0, 0.0], [0.0, 0.0])));

14.6.1 Classification

nn.crossEntropyFromLogits(logits, targetIndex) is logSumExp(logits) − logits[targetIndex], the same number as −log(softmax(logits)[targetIndex]). nn.softmaxGrad(logits, target) is softmax(logits) minus a one-hot vector at that class index, or minus a numeric vector of the same length. That gradient is the upstream for a classification layer whose forward activation was "linear": softmax is inside the loss, not a second nonlinearity on pre.

Two equal logits [0, 0] and target class 0 give a softmax of [0.5, 0.5]. The loss is −log(0.5), and int(loss × 1000) prints 693. The gradient is [−0.5, 0.5]. Logits [2, 0] already prefer class 0, so the same scaling prints a loss of 126 and gradient entries −11 and 11 (the truncation of about ±0.119). Leave the last layer "linear" and pass softmaxGrad in as upstream.

var flat = nn.crossEntropyFromLogits([0.0, 0.0], 0);
var flatGrad = nn.softmaxGrad([0.0, 0.0], 0);
print(int(flat * 1000));
print(int(flatGrad[0] * 10));
print(int(flatGrad[1] * 10));
var peaked = nn.crossEntropyFromLogits([2.0, 0.0], 0);
var peakedGrad = nn.softmaxGrad([2.0, 0.0], 0);
print(int(peaked * 1000));
print(int(peakedGrad[0] * 100));
print(int(peakedGrad[1] * 100));

14.6.2 A return instead of a label

A classifier is handed the right class. A policy samples an action from softmax(logits), and the environment answers with a return. That return is the teacher.

nn.softmaxGrad(logits, action) is still softmax minus a one-hot. On a labeled example the one-hot is the true class, and that gradient is the upstream. On REINFORCE the one-hot is the action you just sampled, and you multiply the gradient by the return G before denseBackward. The quantity you descend is −G × log π(action). A positive G makes that action more likely under the subtraction in 14.5.2. A negative G makes it less likely. A G of 0 leaves the weights where they are.

On one step the return is the reward. On a longer episode, G at a step is the sum of the rewards from that step through the end, discounted if you want a factor below 1. You keep that sum in a variable.

One step, in numbers. The state is 1. The weights and the bias are zero, so both logits are 0 and both actions have probability 0.5. The sampled action is 1 and the return is 1. nn.softmaxGrad at action 1 is [0.5, −0.5]. Times the return, that vector is the upstream. A learning rate of 1 moves the weight row to [−0.5, 0.5] and the bias to the same pair. The next forward on state 1 prefers action 1. int(probability × 100) is 88. The snippet prints the upstream times 10, the updated weight row times 10, and that probability.

var x = [1.0];
var w = [[0.0, 0.0]];
var b = [0.0, 0.0];
var layer = nn.dense(x, w, b, "linear");
var grad = nn.softmaxGrad(layer.out, 1);
var upstream = [grad[0] * 1.0, grad[1] * 1.0];
var back = nn.denseBackward(x, w, upstream, "linear", layer.pre);
print(int(upstream[0] * 10));
print(int(upstream[1] * 10));
w[0][0] = w[0][0] - 1.0 * back.dWeights[0][0];
w[0][1] = w[0][1] - 1.0 * back.dWeights[0][1];
b[0] = b[0] - 1.0 * back.dBias[0];
b[1] = b[1] - 1.0 * back.dBias[1];
print(int(w[0][0] * 10));
print(int(w[0][1] * 10));
var again = nn.dense(x, w, b, "linear");
var probs = math.softmax(again.out);
print(int(probs[1] * 100));

A number you subtract from G that ignores the action is a baseline. The usual choice is the mean return so far. The expected update is the same, and a long run of +1 rewards stops arriving at full scale. Examples/AI_Theory/reinforce_bandit.malda is that loop on a 1-4-2 net. The state is +1 or −1. Action 1 is the match for +1, and action 0 is the match for −1. The hidden layer is tanh, the logits are linear, and the upstream is softmaxGrad times reward − baseline.

malda play Examples/Games/cartpole_neural.malda fits a canvas net to a score of pole angle, angular rate, cart position, and cart speed. A force of ±10 N shows up many steps later, so that short demo keeps the score as the teacher. The bandit is the program where the return arrives on the same step. malda play Examples/Games/grand_prix_neural.malda fits the same dense layer to a driving coach on simplified Grand Prix centerlines. malda play Examples/Games/spoken_reward.malda fits a 4-12-1 net, on x, y, x², and y², to a reward you paint or name in a sentence. The creature climbs the net, so it keeps walking the old way for a moment after the sentence changes.

14.7 Examples

The same three moves run through every file. A loss, or a return, names an upstream. nn.denseBackward turns that upstream into dInput, dWeights, and dBias. You subtract the learning rate times each gradient, or you store a velocity, a cap, or Adam's moments and then subtract. A second layer takes the first layer's dInput. A classifier passes nn.softmaxGrad of a known class. A policy passes that same vector times a return (14.6.2). A convolution, a recurrence, and an attention head change which numbers are multiplied. You still own the step.

Read the files in the order below. Each one seeds the RNG where it draws weights, runs offline, and needs no API key. The calls in this chapter are enough to read them. What changes from file to file is the idea: a hidden layer, a held-out glyph, a velocity you store yourself, one shared kernel, one attention head.

The same folder also holds svm_linear.malda and sarsa_cliff.malda. Those are a margin classifier and a tabular cliff, not a dense stack.

14.7.1 One neuron, then a hidden layer

chain_rule.malda is one multiply, and the reason backprop multiplies local derivatives. perceptron.malda is one neuron: AND and OR are separable, XOR is not. xor_neural_net.malda adds the hidden layer as a 2-4-1 net and names every product, so dZ, dWeights, and dInput are still variables. nn_dense.malda trains that same net with nn.dense and nn.denseBackward, including the in-place learning-rate step from 14.5.2. sequential_xor.malda is the same XOR net through Sequential.fit.

14.7.2 A classifier

softmax_classifier.malda separates three classes in 2D. The loss is nn.crossEntropyFromLogits and the upstream is nn.softmaxGrad, as in 14.6.1. mnist_digits.malda uses ten 5×5 glyphs in the MNIST shape. The hidden nn.dense is "relu". The ten logits are "linear". glyph_holdout.malda trains those ten glyphs, then scores a copy with two flipped pixels. Train is perfect. Holdout is not, because those flips swap the 8 with the 9. There is no MNIST download.

reinforce_bandit.malda keeps nn.softmaxGrad and passes the sampled action as the index. The scale is the return minus a baseline (14.6.2). The state is +1 or −1, the hidden layer is tanh, and the two logits are linear.

14.7.3 Check the gradient and the scale

gradcheck_dense.malda nudges each weight of one ReLU sample by a central difference and matches nn.denseBackward. init_scale.malda is one forward through a deep stack. A wide uniform draw saturates the sigmoids, so the local derivative from 14.4 collapses. Dividing by sqrt(in) (Xavier) keeps that stack responsive. The He draw keeps a ReLU stack smaller than the same uniform draw. new Dense uses 1/sqrt(inFeatures) by default.

14.7.4 The step you still write

Momentum, a length cap, and Adam are programs. They are not builtins. Each one starts from the subtraction in 14.5.2. momentum_valley.malda keeps a velocity on a quadratic that is steep in one axis. grad_clip.malda uses a steeper quadratic: unclipped, the magnitude grows; a cap on the gradient length walks toward zero. adam_valley.malda keeps m, v, and the step counter. The bias-corrected step finishes near the origin, while plain SGD stays far.

14.7.5 Shapes that are not one dense call

The layout is the lesson. rnn_delay.malda shows a sigmoid recurrence shrinking its gradient, then a small tanh net that recalls the bit from three steps earlier. conv_stroke.malda reuses one 3×3 kernel at every window of a 5×5 dash, and each window adds the upstream times the pixel under that weight (14.11.1). conv_classifier.malda trains those same dashes with new Conv(3) and a sigmoid Dense. The 3×3 map is flattened, and Dense.backward is reshaped into Conv.backward. cnn_basic_layers.malda walks one CHW map through Conv2D, Activation, pooling, and Flatten (14.11.6). cnn_composable.malda stacks two of those blocks inside an outer Sequential and trains the classifier with nn.mseGrad (14.10.1, 14.11.7). parallel_compose.malda adds two branches and joins two vectors, with an Identity skip (14.10.2). residual_dropout.malda puts a skip next to a deep sigmoid stack, then a train-time mask that eval does not apply.

14.7.6 From one row to one decoder block

embed_row.malda writes the gradient into the selected embedding row and leaves the other rows put. next_char.malda is a bigram on a repeated word: the loss is nn.crossEntropyFromLogits, and the dense gradient scatters back into that one row. next_char_mlp.malda is the case a bigram cannot name. Both samples end on a, one successor is b, the other is c, and a ReLU hidden layer on the two embedding rows names both. layer_norm.malda subtracts the mean and divides by the standard deviation so a deep product stays near 1, then checks the scale with a central difference.

attention_step.malda is one head. Scores are QK^T / sqrt(d). Training puts the mass on a marked token, then a causal mask stops position 0 from reading position 2. positional_encoding.malda adds fixed sinusoids: position 0 is sin(0) and cos(0), and position 1 matches sin(1) and cos(1). sentence_decoder.malda is one causal block: token plus sinusoid, residual attention, one linear readout. It overfits abca to bcab, and the dense gradient is clipped before the step.

attention_is_all_you_need.malda is the step after that block. It stays on math.matmul and adds multi-head attention, the same sinusoids, and the encoder-decoder. microgpt.malda is a tiny character model on that track. embedding_2d.malda projects a hash embedding and lists VectorDB neighbors. onnx_inspect.malda loads new OnnxModel(path) on a small identity model: inspect and one forward, on the host, with no training step. malda play Examples/Games/xor_decision_boundary.malda draws the XOR field. malda play Examples/Games/softmax_decision_boundary.malda draws the three softmax regions. malda play Examples/Games/cartpole_neural.malda fits a canvas policy to a score of the pole and the cart. malda play Examples/Games/grand_prix_neural.malda fits that policy to a driving coach on eight Grand Prix circuits. malda play Examples/Games/spoken_reward.malda fits a 4-12-1 net to a painted or spoken reward, and the creature follows the net rather than the reward. The return itself is 14.6.2.

14.8 Backends

Interpreter, C# transpile, and JavaScript all run math.dot / matmul / transpose, nn.*, new Dense / new Sequential / new Parallel / new Identity, and the layer classes in 14.11. OnnxModel is host-only. There is no tensor type, no autograd tape, and no Adam or data loader in core. There is no flat alias for nn names: write nn.relu, not relu.

14.9 Signatures

Arity and return values. The layer model is the rest of this chapter. Interpreter, C# transpile, and JavaScript all run nn.*. There is no flat alias.

print(int(nn.relu(-2.0) + nn.relu(1.0)));
print(int(nn.leakyRelu(-2.0) * 100));
print(int(nn.dSigmoid(0.0) * 100));
var layer = nn.dense([1.0, 2.0], [[0.5, -1.0], [1.0, 0.0]], [0.25, -0.5], "relu");
print(int(layer.out[0] * 100));
print(int(layer.out[1]));

14.10 Sequential

new Dense(inFeatures, outFeatures, activation?, scale?) owns one weight matrix [in, out] and one bias. The default scale is 1/sqrt(inFeatures), drawn with the same generator as math.seed. Pass 0 when you want zeros and will write the entries yourself. forward(x) returns the output vector and remembers pre and the input. backward(upstream) calls nn.denseBackward and keeps dWeights and dBias. sgd(lr) subtracts those gradients in place. The input is one vector, not a batch. Call forward before backward, and backward before sgd.

new Sequential([layer, ...]) stacks any layer that has forward and backward, including a nested Sequential. That nested net is one layer of the outer net (14.10.1). sgd skips a layer that has no sgd. forward, backward, and sgd walk the stack. backward starts at the last layer, so you pass only the loss gradient. fit(inputs, targets, epochs, lr, loss?) is online SGD: each sample is a forward, a backward, and an sgd before the next sample is seen. It returns the mean loss of the last epoch, measured before that epoch's steps. loss is "mse" (the default; a target is a number or a vector) or "crossEntropy" (a class index; leave the last layer "linear"). fit expects a vector output. It is not a tape, and it is not Adam.

The snippet builds a 1-to-1 linear layer of zeros and fits one sample, input [1] and target 2, for one epoch at learning rate 0.5. The reported loss is 4, the squared error of the zero prediction. The step then sets both the weight and the bias to 1, so the next forward of [1] is 2. The prints are that loss, the weight and the bias times 100, and the forward times 100. The first three lines are a separate layer: weight 2 and bias 1 on input 3 print 7.

nn.sequential([[in, out, activation?, scale?], ...]) builds the same Sequential. Examples/AI_Theory/sequential_xor.malda trains the 2-4-1 XOR net this way. nn_dense.malda still writes each step by hand.

var layer = new Dense(1, 1, "linear", 0.0);
layer.weights[0][0] = 2.0;
layer.bias[0] = 1.0;
print(int(layer.forward([3.0])[0]));
var fitLayer = new Dense(1, 1, "linear", 0.0);
var net = new Sequential([fitLayer]);
var loss = net.fit([[1.0]], [2.0], 1, 0.5);
print(int(loss));
print(int(fitLayer.weights[0][0] * 100));
print(int(fitLayer.bias[0] * 100));
print(int(fitLayer.forward([1.0])[0] * 100));

14.10.1 Composition from smaller nets

A Sequential is a layer. Another Sequential can hold it. The outer net calls that child as one layer. forward calls forward on each child and passes that child's output to the next child. backward starts at the last child and passes the gradient that child returns to the child before it. sgd calls sgd on each child. A nested Sequential does the same walk on its own layers, so one outer sgd steps every weight in the tree. layers on the outer net is the list of children you passed.

nn.sequential([[in, out, activation?, scale?], ...]) builds a Sequential of Dense layers from those rows. A conv block, a classifier, or any mix of layer classes is new Sequential([...]). fit walks that same tree. The last layer has to return a vector, because the loss compares that vector with the target. A stack that still ends on a map has no vector for mse or crossEntropy. End the net with Flatten or a global pool, then a Dense, when you want fit to train a picture. Examples/AI_Theory/cnn_composable.malda builds two conv blocks and a classifier as three Sequential values, puts them in one outer Sequential, and trains with forward, nn.mseGrad, backward, and sgd so it can count correct labels along the way. The shape of that net is 14.11.7.

The snippet is two one-unit linear nets. The stem multiplies by 2. The head multiplies by 3 and adds 1. The input 4 becomes 8, then 25. layers.length is 2. An upstream of 1 and a learning rate of 1 move the head weight from 3 to −5, because that weight multiplied 8. The stem weight moves from 2 to −10: the head sent an upstream of 3, and the stem multiplied the input 4.

var hidden = new Dense(1, 1, "linear", 0.0);
hidden.weights[0][0] = 2.0;
hidden.bias[0] = 0.0;
var stem = new Sequential([hidden]);
var outLayer = new Dense(1, 1, "linear", 0.0);
outLayer.weights[0][0] = 3.0;
outLayer.bias[0] = 1.0;
var head = new Sequential([outLayer]);
var net = new Sequential([stem, head]);
print(net.layers.length);
print(int(net.forward([4.0])[0]));
net.backward([1.0]);
net.sgd(1.0);
print(int(outLayer.weights[0][0]));
print(int(hidden.weights[0][0]));

14.10.2 Parallel

new Parallel([branch, ...], merge) gives the same input to every branch, then merges the outputs. A branch is any layer with forward and backward, including a Sequential. There are at least two branches. merge is "add" or "concat". add needs the same shape on every branch. concat joins vectors end to end, or joins CHW maps on the channel axis, or joins NCHW maps on the channel axis inside the batch. The other axes have to match. backward on add sends that same upstream to every branch and sums the gradients those branches return. backward on concat splits the upstream along the joined axis, then sums the gradients into the input. sgd calls sgd on each branch that has it. Call forward before backward.

new Identity() returns the input from forward and the upstream from backward. It has no weights and no sgd. A residual block is new Parallel([block, new Identity()], "add"). Examples/AI_Theory/parallel_compose.malda checks the sum, the skip, the joined vector, and one channel stack. residual_dropout.malda names that sum in the program itself.

The snippet is two linear units on the input 4. The left weight is 2, so that branch emits 8. The right weight is 3, so that branch emits 12. add prints 20. An upstream of 1 and a learning rate of 1 subtract 4 from each weight, because each weight multiplied the input 4. The weights print −2 and −1. The skip weight is 0.5, so the residual prints 6: 2 from the block and 4 from Identity. A fresh pair of those same weights, joined with concat, prints 8 then 12. An upstream of [1, 10] subtracts 4 from the left weight and 40 from the right weight, so those weights print −2 and −37.

var left = new Dense(1, 1, "linear", 0.0);
left.weights[0][0] = 2.0;
left.bias[0] = 0.0;
var right = new Dense(1, 1, "linear", 0.0);
right.weights[0][0] = 3.0;
right.bias[0] = 0.0;
var added = new Parallel([left, right], "add");
print(int(added.forward([4.0])[0]));
added.backward([1.0]);
added.sgd(1.0);
print(int(left.weights[0][0]));
print(int(right.weights[0][0]));
var skip = new Dense(1, 1, "linear", 0.0);
skip.weights[0][0] = 0.5;
skip.bias[0] = 0.0;
var residual = new Parallel([skip, new Identity()], "add");
print(int(residual.forward([4.0])[0]));
var catLeft = new Dense(1, 1, "linear", 0.0);
catLeft.weights[0][0] = 2.0;
catLeft.bias[0] = 0.0;
var catRight = new Dense(1, 1, "linear", 0.0);
catRight.weights[0][0] = 3.0;
catRight.bias[0] = 0.0;
var wide = new Parallel([catLeft, catRight], "concat");
var joined = wide.forward([4.0]);
print(int(joined[0]));
print(int(joined[1]));
wide.backward([1.0, 10.0]);
wide.sgd(1.0);
print(int(catLeft.weights[0][0]));
print(int(catRight.weights[0][0]));

14.11 Other layers

These classes keep the same contract as Dense. You own the parameters. forward remembers the input it saw. backward stores the gradients. sgd(lr) subtracts them in place. They do not record a tape, and they do not replace the handwritten files in 14.7.

14.11.1 Conv

new Conv(size, scale?) owns one square kernel, one channel, valid convolution, no bias. The default scale is 1/size. forward(image) returns a matrix smaller than the image by size - 1 on each side. backward(upstream) stores one kernel gradient and returns the image gradient, a matrix the same shape as the image. The kernel gradient is not a field. A step of 1 subtracts it, so each new entry is the old entry minus that gradient. Examples/AI_Theory/conv_classifier.malda trains this layer plus a sigmoid Dense on the 5×5 dashes from conv_stroke.malda. The map is flattened, and the dense input gradient is reshaped into backward.

Each kernel cell is the weight from 14.1.3, shared by every window. At window (oy, ox) the cell (ky, kx) multiplied image[oy + ky][ox + kx], and the slope arriving from that window is upstream[oy][ox]. The kernel gradient at (ky, kx) is the sum of those products. The image gradient is the other product from 14.5: each window adds upstream[oy][ox] × kernel[ky][kx] onto the pixels it covered. Overlapping windows add. The kernel gradient uses the input and the upstream. The image gradient uses the kernel and the upstream.

On a 3×3 ramp and the kernel [[1, 0], [0, 1]], the output is [[6, 8], [12, 14]]. An upstream of ones visits four windows. Cell (0, 0) sat on 1, 2, 4, and 5, so its gradient is 12. Cell (0, 1) sat on 2, 3, 5, and 6, so its gradient is 16. Cell (1, 0) sat on 4, 5, 7, and 8, so its gradient is 24. Cell (1, 1) sat on 5, 6, 8, and 9, so its gradient is 28. A step of 1 moves the two ones to -11 and -27, and the two zeros to -16 and -24.

Those four windows also write the image gradient. The top-left window adds 1 at (0, 0) and at (1, 1), where the kernel is 1. The bottom-right window adds 1 at (1, 1) and at (2, 2). The center is 2 because both windows land there. The corner (0, 2) stays 0, because the only kernel cell that covers it is a 0. The map is [[1, 1, 0], [1, 2, 1], [0, 1, 1]]. The snippet prints the top-left output, the bottom-right output, the image gradient at (0, 0), (1, 1), and (0, 2), then the kernel after the step.

var conv = new Conv(2, 0.0);
conv.kernel[0][0] = 1.0;
conv.kernel[0][1] = 0.0;
conv.kernel[1][0] = 0.0;
conv.kernel[1][1] = 1.0;
var feat = conv.forward([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]]);
var dImage = conv.backward([[1.0, 1.0], [1.0, 1.0]]);
print(int(feat[0][0]));
print(int(feat[1][1]));
print(int(dImage[0][0]));
print(int(dImage[1][1]));
print(int(dImage[0][2]));
conv.sgd(1.0);
print(int(conv.kernel[0][0]));
print(int(conv.kernel[0][1]));
print(int(conv.kernel[1][0]));
print(int(conv.kernel[1][1]));

conv_stroke.malda writes that same product for nine weights, learningRate × (acc − target) × pixel, with pixel the image cell under the weight. It subtracts inside the window loop, so the next window sees a kernel the previous window already moved. Conv.backward sums every window first, and sgd subtracts once.

14.11.2 Embedding

new Embedding(rows, dim, scale?) owns a table. The default scale is 1/sqrt(dim). forward(index) returns a copy of that row. backward(upstream) stores a gradient for that row only. sgd writes it there and leaves the other rows put. An index is an integer. A row of 3 and an upstream of 1 at learning rate 0.5 becomes 2.5.

var emb = new Embedding(2, 1, 0.0);
emb.table[0][0] = 3.0;
emb.table[1][0] = 9.0;
print(int(emb.forward(0)[0]));
emb.backward([1.0]);
emb.sgd(0.5);
print(int(emb.table[0][0] * 10));
print(int(emb.table[1][0]));

14.11.3 Rnn

new Rnn(inputSize, hiddenSize, activation?, scale?) owns weightsXh [input, hidden], weightsHh [hidden, hidden], and a bias. The default activation is tanh. The default scale is 1/sqrt(inputSize). forward(sequence) starts the hidden state at zero and returns one hidden vector per step. backward(upstreams) needs one gradient vector per step and walks from the end, adding into the three parameter gradients. The returned matrix is one input gradient per step. With a linear cell, weight 1, and inputs 2 then 3, the hiddens are 2 and 3. An upstream of 1 on the first step and a learning rate of 0.5 moves the input weight to 0.

var rnn = new Rnn(1, 1, "linear", 0.0);
rnn.weightsXh[0][0] = 1.0;
rnn.weightsHh[0][0] = 0.0;
rnn.bias[0] = 0.0;
var hs = rnn.forward([[2.0], [3.0]]);
print(int(hs[0][0]));
print(int(hs[1][0]));
rnn.backward([[1.0], [0.0]]);
rnn.sgd(0.5);
print(int(rnn.weightsXh[0][0]));

14.11.4 LayerNorm

new LayerNorm(features) owns gamma, starting at 1, and beta, starting at 0. forward(x) subtracts the mean, divides by the standard deviation (epsilon 0.00001 inside the square root), then applies gamma and beta. backward(upstream) returns the input gradient and stores dGamma and dBeta. On [1, 3] with both betas set to 5, the outputs sum to 10. An upstream of [1, 0] and a step of 1 moves gamma[0] to about 2 and beta[0] from 5 to 4.

var ln = new LayerNorm(2);
ln.beta[0] = 5.0;
ln.beta[1] = 5.0;
var y = ln.forward([1.0, 3.0]);
print(int((y[0] + y[1]) * 10));
ln.backward([1.0, 0.0]);
ln.sgd(1.0);
print(int(ln.gamma[0] * 1000));
print(int(ln.beta[0]));

14.11.5 Attention

new Attention(length, dim, scale?) owns query and key, each [length, dim]. The score scale defaults to 1/sqrt(dim). Pass a number when you want a different multiplier. The initial draw of query and key still uses 1/sqrt(dim). forward(value, mask?) computes softmax(scale · QK^T) and multiplies by value. A mask entry below 0.5 replaces that score with a large negative before the softmax, and that position does not receive a score gradient. After forward, probs is the attention matrix. backward(upstream) stores gradients for query and key and returns the value gradient. With scale 1 and zero query and key, two values 2 and 6 are averaged to 4. The causal mask [[1, 0], [0, 1]] keeps 2 and 6.

var head = new Attention(2, 1, 1.0);
head.query[0][0] = 0.0;
head.query[1][0] = 0.0;
head.key[0][0] = 0.0;
head.key[1][0] = 0.0;
var plain = head.forward([[2.0], [6.0]]);
var causal = head.forward([[2.0], [6.0]], [[1.0, 0.0], [0.0, 1.0]]);
print(int(plain[0][0]));
print(int(plain[1][0]));
print(int(causal[0][0]));
print(int(causal[1][0]));
print(int(head.probs[0][1] * 1000));

14.11.6 Convolution

A dense layer on a flattened picture stores one weight per pixel. A convolution stores one small kernel and slides it. Every window multiplies those same numbers, so a stroke the kernel has learned at the top-left is the same stroke at the bottom-right. Conv in 14.11.1 is that slide on one channel, with no bias. Conv2D is the same slide with several input channels, several output channels, a bias, a stride, and padding. Each of those kernels keeps the two products from 14.11.1. The sums are below.

One sample is CHW: a list of channels, each channel a list of rows, each row a list of numbers. input[0][0][0] is a number, and forward returns CHW. A batch is a list of samples, NCHW, so input[0][0][0] is a row and forward returns NCHW. A list that holds one CHW map is a batch of one, and the result stays NCHW. The snippet prints the corner of a CHW picture, the number 1, and the length of that same corner once the picture sits inside a batch, which is 2.

var picture = [[[1.0, 2.0], [3.0, 4.0]]];
var batched = [picture];
print(int(picture[0][0][0]));
print(batched[0][0][0].length);

new Conv2D(inChannels, outChannels, kernelSize, stride?, padding?, scale?) owns weights [inChannels, outChannels, kernel, kernel] and a bias of length outChannels. One output cell is the bias of that output channel plus, for every input channel, the dot product of that channel's kernel with the window under it. The kernel that reads input channel ic and writes output channel oc is weights[ic][oc]. The window for output row oh and column ow starts at oh * stride - padTop and ow * stride - padLeft. A cell that lands off the image contributes 0. The default stride is 1, the default padding is "valid", and the default scale is 1/sqrt(inChannels * kernel * kernel). Pass 0 for a zero kernel and write the entries yourself. Call forward before backward.

"valid" uses no padding. The output height is (height - kernelSize) / stride + 1, with the division truncated, and the same formula for the width. A 3×3 image, kernel 2, stride 1, is 2×2, which is the map in the ramp snippet below. An image smaller than the kernel fails. "same" sets the output height to ceil(height / stride), written (height + stride - 1) / stride, and the same for the width. The kernel may hang off the edge. The pad on the top is half the extra rows, truncated, so a leftover row of zeros sits on the bottom. A leftover column sits on the right. With an odd kernel and stride 1, "same" keeps the height and the width.

On a 4×4 ramp and a 3×3 kernel that is 1 only at the center, "valid" is 2×2 and the corners are 6 and 11, the centers of the four windows. "same" with stride 1 keeps 4×4, and the top-left cell is 1, the corner of the image, because the center of that window lands on the corner. A 2×2 kernel with stride 2 and the diagonal weights [[1, 0], [0, 1]] steps without overlap. The top-left window is 1 + 6 = 7 and the bottom-right is 11 + 16 = 27. The prints are the valid height, those two corners, the same height, the same top-left, the stride-2 height, and the two stride-2 corners.

var image = [[[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0], [9.0, 10.0, 11.0, 12.0], [13.0, 14.0, 15.0, 16.0]]];
var valid = new Conv2D(1, 1, 3, 1, "valid", 0.0);
valid.weights[0][0][1][1] = 1.0;
valid.bias[0] = 0.0;
var yv = valid.forward(image);
print(yv[0].length);
print(int(yv[0][0][0]));
print(int(yv[0][1][1]));
var same = new Conv2D(1, 1, 3, 1, "same", 0.0);
same.weights[0][0][1][1] = 1.0;
same.bias[0] = 0.0;
var ys = same.forward(image);
print(ys[0].length);
print(int(ys[0][0][0]));
var stepped = new Conv2D(1, 1, 2, 2, "valid", 0.0);
stepped.weights[0][0][0][0] = 1.0;
stepped.weights[0][0][1][1] = 1.0;
stepped.bias[0] = 0.0;
var ys2 = stepped.forward(image);
print(ys2[0].length);
print(int(ys2[0][0][0]));
print(int(ys2[0][1][1]));

The 3×3 ramp from 14.11.1, as one channel, with the same diagonal 2×2 kernel. The top-left output is 1 + 5 = 6. The bottom-right is 5 + 9 = 14. MaxPool2D(2) of that 2×2 map keeps 14.

var conv = new Conv2D(1, 1, 2, 1, "valid", 0.0);
conv.weights[0][0][0][0] = 1.0;
conv.weights[0][0][0][1] = 0.0;
conv.weights[0][0][1][0] = 0.0;
conv.weights[0][0][1][1] = 1.0;
conv.bias[0] = 0.0;
var y = conv.forward([[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]]]);
var pool = new MaxPool2D(2);
var pooled = pool.forward(y);
print(int(y[0][0][0]));
print(int(y[0][1][1]));
print(int(pooled[0][0][0]));

backward(upstream) takes a map in the shape forward returned, and it returns the input gradient in the layout forward saw, CHW or NCHW. It stores a weight gradient of shape [inChannels, outChannels, kernel, kernel] and a bias gradient of length outChannels. There is no field for either one. sgd(lr) subtracts them in place, and it needs a prior backward. One output cell on channel oc, at row oh and column ow, carries an upstream number grad. The window starts where the forward window started.

• The bias gradient of channel oc gains grad, once for that output cell.
• For each input channel ic and each kernel cell (kh, kw) that lands on the image, the weight gradient at weights[ic][oc][kh][kw] gains grad times the input cell under it.
• That same pair adds grad times the weight onto the input gradient at that cell.

A cell that lands off the image adds nothing and receives nothing. Overlapping windows add, on the kernel and on the image. A batch sums the weight gradient and the bias gradient across samples. Each sample keeps its own input gradient. Divide by the sample count yourself when you want a mean, as in 14.3.2.

A 1×1 kernel is the dense product from 14.5. One input channel, weight 3, bias 0, and pixel 2 write 6. An upstream of 4 makes the weight gradient 4 × 2 = 8, the bias gradient 4, and the input gradient 4 × 3 = 12. A step of 1 moves the weight from 3 to −5 and the bias from 0 to −4. A second output channel with weight 5 and upstream 1 writes 10 on the forward pass. Its kernel receives 1 × 2 = 2, so the step leaves that weight at 3. The first channel's weight still moves to −5. The single input cell receives both channels: 4 × 3 + 1 × 5 = 17. Two input channels split the other way. Channel 0 holds 2 with weight 1, channel 1 holds 3 with weight 4, and the output is 14. An upstream of 1 gives those kernels gradients 2 and 3, so the step leaves them at −1 and 1. The input gradient is the weight that channel wrote through: 1 on channel 0 and 4 on channel 1. The snippet prints the one-channel output, its input gradient, the weight, and the bias; then the two output-channel values, the shared input gradient, both weights, and both biases; then the two-input output, both input-channel gradients, and both weights.

var dot = new Conv2D(1, 1, 1, 1, "valid", 0.0);
dot.weights[0][0][0][0] = 3.0;
dot.bias[0] = 0.0;
var y = dot.forward([[[2.0]]]);
var dIn = dot.backward([[[4.0]]]);
print(int(y[0][0][0]));
print(int(dIn[0][0][0]));
dot.sgd(1.0);
print(int(dot.weights[0][0][0][0]));
print(int(dot.bias[0]));
var outs = new Conv2D(1, 2, 1, 1, "valid", 0.0);
outs.weights[0][0][0][0] = 3.0;
outs.weights[0][1][0][0] = 5.0;
outs.bias[0] = 0.0;
outs.bias[1] = 0.0;
var yOut = outs.forward([[[2.0]]]);
var dOut = outs.backward([[[4.0]], [[1.0]]]);
print(int(yOut[0][0][0]));
print(int(yOut[1][0][0]));
print(int(dOut[0][0][0]));
outs.sgd(1.0);
print(int(outs.weights[0][0][0][0]));
print(int(outs.weights[0][1][0][0]));
print(int(outs.bias[0]));
print(int(outs.bias[1]));
var ins = new Conv2D(2, 1, 1, 1, "valid", 0.0);
ins.weights[0][0][0][0] = 1.0;
ins.weights[1][0][0][0] = 4.0;
ins.bias[0] = 0.0;
var yIn = ins.forward([[[2.0]], [[3.0]]]);
var dChan = ins.backward([[[1.0]]]);
print(int(yIn[0][0][0]));
print(int(dChan[0][0][0]));
print(int(dChan[1][0][0]));
ins.sgd(1.0);
print(int(ins.weights[0][0][0][0]));
print(int(ins.weights[1][0][0][0]));

The 3×3 ramp from 14.11.1, with the same diagonal kernel and an upstream of ones, returns the same image gradient: 1 at the top-left, 2 at the center, 0 at (0, 2). The corner weight moves from 1 to −11. Conv has no bias. Here the bias gradient is the sum of the four upstream ones, so a step of 1 moves the bias from 0 to −4. Two copies of that ramp, passed as NCHW, leave the first sample's center at 2. The corner weight moves by 24, twice 12, from 1 to −23. The bias moves by 8, twice 4, from 0 to −8.

Padding is a zero input. A 1×1 image of 7, a 3×3 kernel, and "same" keep a 1×1 output. The window hangs off on every side, and only the center kernel cell sees the 7. With that center set to 1, the forward cell is 7 and an upstream of 1 makes the center gradient 7. The eight cells that saw only padding keep gradient 0. The one image cell receives 1 × 1. A step of 1 moves the center from 1 to −6, leaves a corner at 0, and moves the bias from 0 to −1. The snippet prints the ramp's image gradient at those three cells, the corner weight, and the bias; then the batched center, corner weight, and bias; then the padded output, its input gradient, the center weight, a corner weight, and the bias.

var conv = new Conv2D(1, 1, 2, 1, "valid", 0.0);
conv.weights[0][0][0][0] = 1.0;
conv.weights[0][0][0][1] = 0.0;
conv.weights[0][0][1][0] = 0.0;
conv.weights[0][0][1][1] = 1.0;
conv.bias[0] = 0.0;
var picture = [[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]]];
var dIn = conv.forward(picture);
dIn = conv.backward([[[1.0, 1.0], [1.0, 1.0]]]);
print(int(dIn[0][0][0]));
print(int(dIn[0][1][1]));
print(int(dIn[0][0][2]));
conv.sgd(1.0);
print(int(conv.weights[0][0][0][0]));
print(int(conv.bias[0]));
var batch = new Conv2D(1, 1, 2, 1, "valid", 0.0);
batch.weights[0][0][0][0] = 1.0;
batch.weights[0][0][0][1] = 0.0;
batch.weights[0][0][1][0] = 0.0;
batch.weights[0][0][1][1] = 1.0;
batch.bias[0] = 0.0;
var up = [[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 1.0], [1.0, 1.0]]]];
var dBatch = batch.forward([picture, picture]);
dBatch = batch.backward(up);
print(int(dBatch[0][0][1][1]));
batch.sgd(1.0);
print(int(batch.weights[0][0][0][0]));
print(int(batch.bias[0]));
var pad = new Conv2D(1, 1, 3, 1, "same", 0.0);
pad.weights[0][0][1][1] = 1.0;
pad.bias[0] = 0.0;
var yPad = pad.forward([[[7.0]]]);
var dPad = pad.backward([[[1.0]]]);
print(int(yPad[0][0][0]));
print(int(dPad[0][0][0]));
pad.sgd(1.0);
print(int(pad.weights[0][0][1][1]));
print(int(pad.weights[0][0][0][0]));
print(int(pad.bias[0]));

A stride equal to the kernel size lays the windows side by side. On the 4×4 ramp, the same diagonal 2×2 kernel, and stride 2, an upstream of ones covers each image cell with one window. The top-left cell receives the kernel's 1. The cell beside it sits under a kernel 0, so its image gradient is 0. Kernel cell (0, 0) summed 1 + 3 + 9 + 11 = 24. Kernel cell (1, 1) summed 6 + 8 + 14 + 16 = 44. A step of 1 moves those ones to −23 and −43. The bias gradient is again the four upstream ones, so the bias moves to −4. The snippet prints the two image cells, the two kernel ones, and the bias.

var stepped = new Conv2D(1, 1, 2, 2, "valid", 0.0);
stepped.weights[0][0][0][0] = 1.0;
stepped.weights[0][0][0][1] = 0.0;
stepped.weights[0][0][1][0] = 0.0;
stepped.weights[0][0][1][1] = 1.0;
stepped.bias[0] = 0.0;
var image = [[[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0], [9.0, 10.0, 11.0, 12.0], [13.0, 14.0, 15.0, 16.0]]];
var dStep = stepped.forward(image);
dStep = stepped.backward([[[1.0, 1.0], [1.0, 1.0]]]);
print(int(dStep[0][0][0]));
print(int(dStep[0][0][1]));
stepped.sgd(1.0);
print(int(stepped.weights[0][0][0][0]));
print(int(stepped.weights[0][0][1][1]));
print(int(stepped.bias[0]));

14.11.7 A convolutional net

A convolutional net is a Sequential. The early layers read a map. The last layer returns a vector, so the losses in 14.6 still apply. Between them the map gets smaller and the channel count usually grows. Group those layers into smaller nets with 14.10.1.

One block in Examples/AI_Theory/cnn_composable.malda is itself a Sequential: Conv2D with "same" padding, BatchNorm2D, Activation("relu"), Dropout2D, and MaxPool2D(2). With an odd kernel and stride 1, "same" keeps the height and the width. The pool of 2, whose default stride is the pool size, then halves each side. A 5×5 map with that pool is 2×2. The size uses the "valid" formula, so a leftover row or column sits outside every window.

Activation(name) applies one name from 14.4 to every cell of a vector or a nested map. A stack of convolutions with no bend between them collapses into one linear filter, for the same reason a stack of plain products does in 14.1.2. BatchNorm2D(numFeatures, momentum?, eps?) shifts each channel to mean 0 and variance 1, over the batch and the spatial cells, then multiplies by gamma (starting at 1) and adds beta (starting at 0). The defaults are momentum 0.1 and epsilon 0.00001. train() uses the batch and updates runningMean and runningVar. Before any training forward those running values are 0 and 1. eval() uses the running values. backward needs a training forward. Dropout2D(p), with p in [0, 1), keeps each element with probability 1 - p during training and multiplies the kept elements by 1 / (1 - p). eval(), and p of 0, copy the map. It has no sgd.

Flatten walks the nested numbers in storage order, channel then row then column, and returns one vector. backward writes that vector back into the shape forward saw. GlobalAvgPool2D and GlobalMaxPool2D reduce each channel to one number: a vector when the input was CHW, and [batch, channels] when it was NCHW. MaxPool2D sends the upstream to the cell that held the maximum. A tie keeps the earlier cell, left to right, then top to bottom. AvgPool2D writes the mean of the window. Its backward adds the upstream, divided by the window area, into every cell. Overlapping windows add. Pool and flatten have no sgd. backward on a pool takes only the upstream. The spatial size is the one forward stored.

The classifier is another Sequential: Flatten, a Dense with "relu", and a Dense with "linear" scores. cnn_composable.malda uses an 8×8 one-channel image. The first block has 4 output channels, so the pool yields 4×4×4. The second block has 8 output channels, so the next pool yields 2×2×8, which is 32 numbers. The classifier is Dense(32, 8, "relu") then Dense(8, 2, "linear"). The outer Sequential is those two blocks and the classifier. layers.length on the outer net is 3. One forward walks all three. One backward starts at the classifier. One sgd steps every layer in the tree that has sgd. The training loop passes nn.mseGrad of a one-hot target. fit does that walk when the output is a vector and the loss is "mse" or "crossEntropy". The example writes the loop so it can count correct labels. cnn_basic_layers.malda is the same layers called one at a time, on one CHW map, including GlobalAvgPool2D and a Dropout2D left in eval().

The snippet is that composition on the 4×4 ramp. The block is a Sequential of a 3×3 "same" convolution, center weight 1, and MaxPool2D(2). The map stays 4×4, then the pool yields 2×2. Those four numbers are 6, 8, 14, and 16. The classifier is a Sequential of Flatten and a linear Dense whose four weights are 1. The sum is 44. The outer net's layers.length is 2. The block's layers.length is 2.

function convPool() {
    var conv = new Conv2D(1, 1, 3, 1, "same", 0.0);
    conv.weights[0][0][1][1] = 1.0;
    conv.bias[0] = 0.0;
    return new Sequential([conv, new MaxPool2D(2)]);
}
var image = [[[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0], [9.0, 10.0, 11.0, 12.0], [13.0, 14.0, 15.0, 16.0]]];
var block = convPool();
var dense = new Dense(4, 1, "linear", 0.0);
dense.weights[0][0] = 1.0;
dense.weights[1][0] = 1.0;
dense.weights[2][0] = 1.0;
dense.weights[3][0] = 1.0;
dense.bias[0] = 0.0;
var classifier = new Sequential([new Flatten(), dense]);
var cnn = new Sequential([block, classifier]);
print(cnn.layers.length);
print(block.layers.length);
print(int(cnn.forward(image)[0]));