NEURAL MATH LAB / YOUR FIRST EXPERIMENT

A tiny network.
Every calculation visible.

Two inputs, one hidden layer, and one question: are the bits different?

XOR returns 1 for (0, 1) and (1, 0), and 0 for (0, 0) and (1, 1). This report records an actual NumPy training run using manually written derivatives.

FINAL MEAN LOSS · LOWER IS BETTER
0.001075
Started at 0.704652
TRAINING ACCURACY ON FOUR EXAMPLES
100%
4000 full-batch gradient updates · 4 hidden neurons

All four XOR rows were used for training. These results show fitting this truth table, not performance on unseen data. The output p is not calibrated confidence.

XOR training loss XOR training loss image/svg+xml Matplotlib v3.9.4, https://matplotlib.org/ 0 500 1000 1500 2000 2500 3000 3500 4000 Completed gradient updates 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Mean binary cross-entropy Learning from four examples

The loss measures how well the probabilities match the targets. One epoch here is one update using all four examples.

XOR fitted model output XOR fitted model output image/svg+xml Matplotlib v3.9.4, https://matplotlib.org/ −0.25 0.00 0.25 0.50 0.75 1.00 1.25 Input x1 −0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Input x2 target 0 target 1 target 1 target 0 What the fitted network computes 0.0 0.2 0.4 0.6 0.8 1.0 Model output p

Squares mark target 0, circles target 1. The black curve, when present, is p = 0.5. Values between the four binary points illustrate the fitted function; they have no XOR ground truth.

01 / CHECK THE ANSWERS

What did the model learn?

InputsTargetOutput pClass (p ≥ 0.5)
0, 000.0001640
0, 110.9985121
1, 010.9989001
1, 100.0015440
02 / FOLLOW ONE EXAMPLE

Trace input (1, 0)

Each column of W1 describes a hidden neuron. Multiply inputs by its weights, add its bias, and apply tanh. The output neuron combines those activations, then sigmoid maps the result into (0, 1), subject to floating-point rounding.

z1 = x @ W1 + b1 = [[ 1.074338  2.184203  5.168733 -1.623115]]
h = tanh(z1) = [[ 0.79109   0.974974  0.999935 -0.925075]]
z2 = h @ W2 + b2 = [[6.811489]]
p = sigmoid(z2) = [[0.9989]]
Inspect this run’s learned weights and biases
W1 =
[[ 0.995453  2.700105  4.088063 -2.529177]
 [-2.550398  2.786716 -2.984547  3.691132]]

b1 =
[[ 0.078885 -0.515902  1.080669  0.906062]]

W2 =
[[ 2.390827]
 [ 4.81956 ]
 [-6.959536]
 [-5.033965]]

b2 =
[[2.523474]]
03 / HOW LEARNING HAPPENS

The chain rule supplies the direction.

Z1 = X @ W1 + b1       H = tanh(Z1)
Z2 = H @ W2 + b2       P = sigmoid(Z2)
L  = mean(logaddexp(0, Z2) - Y * Z2)

dZ2 = (P - Y) / N
dW2 = H.T @ dZ2        db2 = sum(dZ2, axis=0)
dZ1 = (dZ2 @ W2.T) * (1 - H**2)
dW1 = X.T @ dZ1        db1 = sum(dZ1, axis=0)

parameter = parameter - learning_rate * gradient

N is the number of examples, four here. A positive derivative means increasing that parameter locally increases the loss; subtracting the gradient takes a small step in the opposite direction. The learning rate determines the step size.

Read The network, from first principles in the repository for the full derivation, a worked numerical example, and the reason one linear classifier cannot solve XOR. Then try First principles · exercises.

04 / REPRODUCE AND EXPERIMENT

Change one setting at a time.

python -m neural_math_lab train --epochs 4000 --learning-rate 0.5 --hidden 4 --seed 7 --output runs/reproduction

Use a new output directory for each run. Compare loss curves with learning rates 0.05 and 0.5, or change the number of hidden neurons. Failed learning is useful evidence too.

Run configuration and environment
{
  "config": {
    "epochs": 4000,
    "learning_rate": 0.5,
    "hidden": 4,
    "seed": 7
  },
  "environment": {
    "python": "3.9.6",
    "numpy": "2.0.2",
    "matplotlib": "3.9.4"
  }
}

Alongside this report: model.json contains the weights; metrics.json records the results; history.csv records every epoch. Both figures are separate SVG files as well as embedded here.