LAB 02 / Circles / MANUAL NUMPY

See a network
learn the shape.

Follow the boundary as it changes. Inspect each neuron. Then ask the important question: does it work on examples it never trained on?

Held-out test accuracy97.5%120 unseen examples · selected model

Class 1 · F1 score0.975Combines precision and recall

Against linear baseline+50.0 ppBaseline accuracy: 47.5%

Selected epoch374Lowest validation BCE · stopped at 474

The model was selected using validation loss. The held-out test set was evaluated afterward. These synthetic results describe this split and seed; repeated tuning against the test score would invalidate that separation.

01 / REPLAY & EXPLORE

A boundary built from simple operations.

Selected checkpoint

ModelExamples Mark mistakes

Class 0 · square Class 1 · circle× Incorrect classp = 0p = 1

Saved training checkpoint

The map displays the selected model. Final metric cards above stay fixed while you replay training.

02 / LOOK FOR GENERALIZATION

Training improves. Validation keeps it honest.

Stopped at epoch 474

Training BCE Validation BCEDashed line: selected epoch

Read this curve.

The plotted loss excludes the L2 penalty, so the training and validation curves measure the same quantity. Training updates include the penalty on weights.

The lowest observed validation loss selects the saved weights. Patience stops training after too many epochs without a meaningful improvement; those can be different epochs.

This run stopped because: validation patience. The saved model restores epoch 374. The selected checkpoint is included in the replay.

View the exportable static figures

03 / CHECK THE EVIDENCE

More than one accuracy number.

Threshold p ≥ 0.5

Same held-out examples. Two models.

Model	Accuracy	Precision	Recall	F1	Brier ↓
Neural network	97.5%	0.983	0.967	0.975	0.0263
Linear baseline	47.5%	0.476	0.500	0.488	0.2562

The baseline has one affine score and a sigmoid, with no hidden layer. It uses the same split and training-only standardizer. Its own validation loss selects its weights.

Neural network confusion matrix

Actual ↓ / predicted →
	Class 0	Class 1
Class 0	59	1
Class 1	2	58

Rows are actual classes; columns are predicted classes. Precision and recall refer to class 1. Undefined ratios are reported as 0.

Inspect each data split

Split	Examples	Accuracy	BCE	Role
Training	360	93.3%	0.15583	Gradient updates + scaler
Validation	120	95.8%	0.11688	Choose weights and stop
Test	120	97.5%	0.10919	Final evaluation only

The split is stratified: each class is divided approximately 60/20/20. Every example ID belongs to exactly one split.

Show misclassified test examples

Row ID	x1	x2	Target	Output p	Predicted
548	-0.1569	-0.6185	0	0.962001	1
378	0.7841	0.4339	1	0.181091	0
144	-0.5623	-0.4690	1	0.384679	0

Error analysis describes this completed run. Use fresh evaluation data for future model-selection decisions.

04 / DECIDE WITH DIFFERENT COSTS

A probability becomes a decision.

Validation exploration only

A false alarm and a missed positive need not cost the same. Python selected the threshold with the lowest validation cost for each model. Explore that trade-off below without changing its saved weights or final test evaluation.

Diagnostic modelDecision threshold τExact threshold

Controls require JavaScript. Saved values remain readable below.

False positives—

False negatives—

Recall · class 1—

Precision · class 1—

Cost per example—

Mean cost = (cFP × FP + cFN × FN) / N

Prediction is class 1 when p ≥ τ. Equal scores move together. The no-positive option sets τ just above 1. Undefined precision or recall is displayed as 0, matching the original metrics.

Ranking · ROC

Retrieval · precision–recall

Probability · reliability

These curves use the selected model’s validation scores and stay fixed during threshold exploration and checkpoint replay. The dot on ROC and precision–recall marks the explored decision. A good ranking does not imply calibrated probabilities. Empty calibration bins are omitted; small bins are noisy.

FINAL TEST EVALUATION · FIXED

Evaluate the selected rules once.

These values use each model’s validation-selected threshold. The controls above never recalculate this table on test data.

Model	Selected τ	FP	FN	Mean cost	Accuracy	ROC AUC	AP	Brier
Neural network	0.4430678144727584	1	2	0.0250	0.9750	0.9933	0.9926	0.0263
Linear baseline	0.45010550557564205	47	4	0.4250	0.5750	0.4478	0.4340	0.2562

The earlier accuracy cards use threshold 0.5. This table uses the cost-selected rules, so their counts may differ. Test scores are descriptive for this synthetic split.

Read saved validation results and all curve points

Neural network · saved validation diagnostics

Selected threshold	FP cost	FN cost	Validation mean cost	Candidates
0.4430678144727584	1.0	1.0	0.0333	122

ROC AUC: 0.9950 · Average precision: 0.9949 · Brier: 0.0321 · ECE: 0.0420. Curves below use validation scores.

Reliability bins

Bin (left inclusive)	Count	Mean p	Positive fraction	Absolute gap
[0.0, 0.1)	46	0.0287	0.0000	0.0287
[0.1, 0.2)	5	0.1481	0.0000	0.1481
[0.2, 0.3)	6	0.2326	0.1667	0.0660
[0.3, 0.4)	5	0.3607	0.4000	0.0393
[0.4, 0.5)	1	0.4431	1.0000	0.5569
[0.5, 0.6)	1	0.5662	1.0000	0.4338
[0.6, 0.7)	1	0.6478	1.0000	0.3522
[0.7, 0.8)	2	0.7261	1.0000	0.2739
[0.8, 0.9)	7	0.8532	0.8571	0.0039
[0.9, 1.0]	46	0.9897	1.0000	0.0103

Empty bins have undefined means and are omitted from the plot. ECE is the count-weighted absolute bin gap.

ROC points

Threshold	False-positive rate	True-positive rate	TP	FP
Above 1 (all negative)	0.0000	0.0000	0	0
0.9995651470743017	0.0000	0.0167	1	0
0.9993636638472494	0.0000	0.0333	2	0
0.9991896325540227	0.0000	0.0500	3	0
0.9990897938148601	0.0000	0.0667	4	0
0.9990848608199032	0.0000	0.0833	5	0
0.9990628155888358	0.0000	0.1000	6	0
0.9989765961579721	0.0000	0.1167	7	0
0.9989585709483655	0.0000	0.1333	8	0
0.9989425260762993	0.0000	0.1500	9	0
0.9989251454376634	0.0000	0.1667	10	0
0.9989078468290885	0.0000	0.1833	11	0
0.9988512185922765	0.0000	0.2000	12	0
0.9985829571082949	0.0000	0.2167	13	0
0.9985148881415324	0.0000	0.2333	14	0
0.9985128001631456	0.0000	0.2500	15	0
0.9984786822445044	0.0000	0.2667	16	0
0.9983949231116146	0.0000	0.2833	17	0
0.998338781774046	0.0000	0.3000	18	0
0.9981145136462968	0.0000	0.3167	19	0
0.9980991201321183	0.0000	0.3333	20	0
0.997698326405957	0.0000	0.3500	21	0
0.9958444150606264	0.0000	0.3667	22	0
0.9952455068833476	0.0000	0.3833	23	0
0.9937261507143175	0.0000	0.4000	24	0
0.9932115416458397	0.0000	0.4167	25	0
0.9922258487367699	0.0000	0.4333	26	0
0.9914394206938558	0.0000	0.4500	27	0
0.9906398836712393	0.0000	0.4667	28	0
0.9903967297048293	0.0000	0.4833	29	0
0.9902861482342876	0.0000	0.5000	30	0
0.9900679408237653	0.0000	0.5167	31	0
0.9891063755862616	0.0000	0.5333	32	0
0.9867366140452137	0.0000	0.5500	33	0
0.9841990615019012	0.0000	0.5667	34	0
0.9833605710191803	0.0000	0.5833	35	0
0.9833415735089592	0.0000	0.6000	36	0
0.9832136647432538	0.0000	0.6167	37	0
0.9816717706456117	0.0000	0.6333	38	0
0.9810668912967623	0.0000	0.6500	39	0
0.980149123955724	0.0000	0.6667	40	0
0.9734578638884753	0.0000	0.6833	41	0
0.9677916295912803	0.0000	0.7000	42	0
0.9658105498453871	0.0000	0.7167	43	0
0.9653459390342818	0.0000	0.7333	44	0
0.9618884207699354	0.0000	0.7500	45	0
0.9404124405817048	0.0000	0.7667	46	0
0.8978763549125073	0.0000	0.7833	47	0
0.8773532198697679	0.0000	0.8000	48	0
0.8583054664297263	0.0000	0.8167	49	0
0.8580575226547608	0.0167	0.8167	49	1
0.829634862390402	0.0167	0.8333	50	1
0.8273855094308513	0.0167	0.8500	51	1
0.8238977753510867	0.0167	0.8667	52	1
0.7295305433891723	0.0167	0.8833	53	1
0.7225744277685414	0.0167	0.9000	54	1
0.6478275830592916	0.0167	0.9167	55	1
0.566175952706825	0.0167	0.9333	56	1
0.4430678144727584	0.0167	0.9500	57	1
0.3916582231109582	0.0333	0.9500	57	2
0.3816957364105995	0.0500	0.9500	57	3
0.36876642226745143	0.0500	0.9667	58	3
0.3534021533562967	0.0500	0.9833	59	3
0.3082224949351086	0.0667	0.9833	59	4
0.2615410077132776	0.0667	1.0000	60	4
0.2444055646020017	0.0833	1.0000	60	5
0.24074874812790456	0.1000	1.0000	60	6
0.22416070322357118	0.1167	1.0000	60	7
0.21573757089805132	0.1333	1.0000	60	8
0.2091375595203058	0.1500	1.0000	60	9
0.1888951932392383	0.1667	1.0000	60	10
0.1634268196813803	0.1833	1.0000	60	11
0.13822517140547647	0.2000	1.0000	60	12
0.13376574895173124	0.2167	1.0000	60	13
0.11593857783403885	0.2333	1.0000	60	14
0.09897913371606494	0.2500	1.0000	60	15
0.0954616719873313	0.2667	1.0000	60	16
0.08275827827033894	0.2833	1.0000	60	17
0.08260974238504494	0.3000	1.0000	60	18
0.0782658536866376	0.3167	1.0000	60	19
0.07551196922533823	0.3333	1.0000	60	20
0.0708822215751879	0.3500	1.0000	60	21
0.06654814574372156	0.3667	1.0000	60	22
0.05754962045994581	0.3833	1.0000	60	23
0.053185479568215975	0.4000	1.0000	60	24
0.04712942525682051	0.4167	1.0000	60	25
0.03179656418718064	0.4333	1.0000	60	26
0.03073125752250761	0.4500	1.0000	60	27
0.029868396996115043	0.4667	1.0000	60	28
0.028499409522357855	0.4833	1.0000	60	29
0.02499780610485347	0.5000	1.0000	60	30
0.021466969878031365	0.5167	1.0000	60	31
0.02055952008967827	0.5333	1.0000	60	32
0.020336600437417283	0.5500	1.0000	60	33
0.02016428435850252	0.5667	1.0000	60	34
0.020107019211602156	0.5833	1.0000	60	35
0.019906255059879814	0.6000	1.0000	60	36
0.017466264599575967	0.6167	1.0000	60	37
0.016495675508437436	0.6333	1.0000	60	38
0.016342607697102905	0.6500	1.0000	60	39
0.015959838716405984	0.6667	1.0000	60	40
0.014941734679498466	0.6833	1.0000	60	41
0.013612181198457978	0.7000	1.0000	60	42
0.013184672539258122	0.7167	1.0000	60	43
0.012807176897973234	0.7333	1.0000	60	44
0.01269488788874371	0.7500	1.0000	60	45
0.011968058918711787	0.7667	1.0000	60	46
0.01148981693262579	0.7833	1.0000	60	47
0.011247652795118883	0.8000	1.0000	60	48
0.011189529950226934	0.8167	1.0000	60	49
0.009294004908838948	0.8333	1.0000	60	50
0.008603852031541102	0.8500	1.0000	60	51
0.008550008838719474	0.8667	1.0000	60	52
0.008186173735769157	0.8833	1.0000	60	53
0.006167881580671335	0.9000	1.0000	60	54
0.00539550209872014	0.9167	1.0000	60	55
0.004872883408802006	0.9333	1.0000	60	56
0.004571779036236965	0.9500	1.0000	60	57
0.0039121157581669804	0.9667	1.0000	60	58
0.0022142473028518707	0.9833	1.0000	60	59
0.0010943992323741926	1.0000	1.0000	60	60

Precision–recall points

Threshold	Recall	Precision	TP	FP
Above 1 (all negative)	0.0000	1.0000	0	0
0.9995651470743017	0.0167	1.0000	1	0
0.9993636638472494	0.0333	1.0000	2	0
0.9991896325540227	0.0500	1.0000	3	0
0.9990897938148601	0.0667	1.0000	4	0
0.9990848608199032	0.0833	1.0000	5	0
0.9990628155888358	0.1000	1.0000	6	0
0.9989765961579721	0.1167	1.0000	7	0
0.9989585709483655	0.1333	1.0000	8	0
0.9989425260762993	0.1500	1.0000	9	0
0.9989251454376634	0.1667	1.0000	10	0
0.9989078468290885	0.1833	1.0000	11	0
0.9988512185922765	0.2000	1.0000	12	0
0.9985829571082949	0.2167	1.0000	13	0
0.9985148881415324	0.2333	1.0000	14	0
0.9985128001631456	0.2500	1.0000	15	0
0.9984786822445044	0.2667	1.0000	16	0
0.9983949231116146	0.2833	1.0000	17	0
0.998338781774046	0.3000	1.0000	18	0
0.9981145136462968	0.3167	1.0000	19	0
0.9980991201321183	0.3333	1.0000	20	0
0.997698326405957	0.3500	1.0000	21	0
0.9958444150606264	0.3667	1.0000	22	0
0.9952455068833476	0.3833	1.0000	23	0
0.9937261507143175	0.4000	1.0000	24	0
0.9932115416458397	0.4167	1.0000	25	0
0.9922258487367699	0.4333	1.0000	26	0
0.9914394206938558	0.4500	1.0000	27	0
0.9906398836712393	0.4667	1.0000	28	0
0.9903967297048293	0.4833	1.0000	29	0
0.9902861482342876	0.5000	1.0000	30	0
0.9900679408237653	0.5167	1.0000	31	0
0.9891063755862616	0.5333	1.0000	32	0
0.9867366140452137	0.5500	1.0000	33	0
0.9841990615019012	0.5667	1.0000	34	0
0.9833605710191803	0.5833	1.0000	35	0
0.9833415735089592	0.6000	1.0000	36	0
0.9832136647432538	0.6167	1.0000	37	0
0.9816717706456117	0.6333	1.0000	38	0
0.9810668912967623	0.6500	1.0000	39	0
0.980149123955724	0.6667	1.0000	40	0
0.9734578638884753	0.6833	1.0000	41	0
0.9677916295912803	0.7000	1.0000	42	0
0.9658105498453871	0.7167	1.0000	43	0
0.9653459390342818	0.7333	1.0000	44	0
0.9618884207699354	0.7500	1.0000	45	0
0.9404124405817048	0.7667	1.0000	46	0
0.8978763549125073	0.7833	1.0000	47	0
0.8773532198697679	0.8000	1.0000	48	0
0.8583054664297263	0.8167	1.0000	49	0
0.8580575226547608	0.8167	0.9800	49	1
0.829634862390402	0.8333	0.9804	50	1
0.8273855094308513	0.8500	0.9808	51	1
0.8238977753510867	0.8667	0.9811	52	1
0.7295305433891723	0.8833	0.9815	53	1
0.7225744277685414	0.9000	0.9818	54	1
0.6478275830592916	0.9167	0.9821	55	1
0.566175952706825	0.9333	0.9825	56	1
0.4430678144727584	0.9500	0.9828	57	1
0.3916582231109582	0.9500	0.9661	57	2
0.3816957364105995	0.9500	0.9500	57	3
0.36876642226745143	0.9667	0.9508	58	3
0.3534021533562967	0.9833	0.9516	59	3
0.3082224949351086	0.9833	0.9365	59	4
0.2615410077132776	1.0000	0.9375	60	4
0.2444055646020017	1.0000	0.9231	60	5
0.24074874812790456	1.0000	0.9091	60	6
0.22416070322357118	1.0000	0.8955	60	7
0.21573757089805132	1.0000	0.8824	60	8
0.2091375595203058	1.0000	0.8696	60	9
0.1888951932392383	1.0000	0.8571	60	10
0.1634268196813803	1.0000	0.8451	60	11
0.13822517140547647	1.0000	0.8333	60	12
0.13376574895173124	1.0000	0.8219	60	13
0.11593857783403885	1.0000	0.8108	60	14
0.09897913371606494	1.0000	0.8000	60	15
0.0954616719873313	1.0000	0.7895	60	16
0.08275827827033894	1.0000	0.7792	60	17
0.08260974238504494	1.0000	0.7692	60	18
0.0782658536866376	1.0000	0.7595	60	19
0.07551196922533823	1.0000	0.7500	60	20
0.0708822215751879	1.0000	0.7407	60	21
0.06654814574372156	1.0000	0.7317	60	22
0.05754962045994581	1.0000	0.7229	60	23
0.053185479568215975	1.0000	0.7143	60	24
0.04712942525682051	1.0000	0.7059	60	25
0.03179656418718064	1.0000	0.6977	60	26
0.03073125752250761	1.0000	0.6897	60	27
0.029868396996115043	1.0000	0.6818	60	28
0.028499409522357855	1.0000	0.6742	60	29
0.02499780610485347	1.0000	0.6667	60	30
0.021466969878031365	1.0000	0.6593	60	31
0.02055952008967827	1.0000	0.6522	60	32
0.020336600437417283	1.0000	0.6452	60	33
0.02016428435850252	1.0000	0.6383	60	34
0.020107019211602156	1.0000	0.6316	60	35
0.019906255059879814	1.0000	0.6250	60	36
0.017466264599575967	1.0000	0.6186	60	37
0.016495675508437436	1.0000	0.6122	60	38
0.016342607697102905	1.0000	0.6061	60	39
0.015959838716405984	1.0000	0.6000	60	40
0.014941734679498466	1.0000	0.5941	60	41
0.013612181198457978	1.0000	0.5882	60	42
0.013184672539258122	1.0000	0.5825	60	43
0.012807176897973234	1.0000	0.5769	60	44
0.01269488788874371	1.0000	0.5714	60	45
0.011968058918711787	1.0000	0.5660	60	46
0.01148981693262579	1.0000	0.5607	60	47
0.011247652795118883	1.0000	0.5556	60	48
0.011189529950226934	1.0000	0.5505	60	49
0.009294004908838948	1.0000	0.5455	60	50
0.008603852031541102	1.0000	0.5405	60	51
0.008550008838719474	1.0000	0.5357	60	52
0.008186173735769157	1.0000	0.5310	60	53
0.006167881580671335	1.0000	0.5263	60	54
0.00539550209872014	1.0000	0.5217	60	55
0.004872883408802006	1.0000	0.5172	60	56
0.004571779036236965	1.0000	0.5128	60	57
0.0039121157581669804	1.0000	0.5085	60	58
0.0022142473028518707	1.0000	0.5042	60	59
0.0010943992323741926	1.0000	0.5000	60	60

Linear baseline · saved validation diagnostics

Selected threshold	FP cost	FN cost	Validation mean cost	Candidates
0.45010550557564205	1.0	1.0	0.3000	122

ROC AUC: 0.5936 · Average precision: 0.5044 · Brier: 0.2462 · ECE: 0.0163. Curves below use validation scores.

Reliability bins

Bin (left inclusive)	Count	Mean p	Positive fraction	Absolute gap
[0.0, 0.1)	0	Undefined	Undefined	Undefined
[0.1, 0.2)	0	Undefined	Undefined	Undefined
[0.2, 0.3)	0	Undefined	Undefined	Undefined
[0.3, 0.4)	1	0.3985	0.0000	0.3985
[0.4, 0.5)	68	0.4564	0.4706	0.0142
[0.5, 0.6)	51	0.5375	0.5490	0.0115
[0.6, 0.7)	0	Undefined	Undefined	Undefined
[0.7, 0.8)	0	Undefined	Undefined	Undefined
[0.8, 0.9)	0	Undefined	Undefined	Undefined
[0.9, 1.0]	0	Undefined	Undefined	Undefined

Empty bins have undefined means and are omitted from the plot. ECE is the count-weighted absolute bin gap.

ROC points

Threshold	False-positive rate	True-positive rate	TP	FP
Above 1 (all negative)	0.0000	0.0000	0	0
0.5851089298508227	0.0167	0.0000	0	1
0.5798626689572249	0.0333	0.0000	0	2
0.5786938660457515	0.0500	0.0000	0	3
0.574657217234697	0.0667	0.0000	0	4
0.5728524867921336	0.0833	0.0000	0	5
0.5711681816417851	0.1000	0.0000	0	6
0.5697269051971212	0.1167	0.0000	0	7
0.5677875926065666	0.1333	0.0000	0	8
0.5677005076487734	0.1500	0.0000	0	9
0.5654156665192498	0.1667	0.0000	0	10
0.5630707434961463	0.1833	0.0000	0	11
0.561628445383969	0.2000	0.0000	0	12
0.5613327864144256	0.2167	0.0000	0	13
0.5607407535654877	0.2167	0.0167	1	13
0.5551147459904425	0.2333	0.0167	1	14
0.5528625222685883	0.2333	0.0333	2	14
0.5525715792295951	0.2333	0.0500	3	14
0.5520025386731356	0.2500	0.0500	3	15
0.5514090180434148	0.2667	0.0500	3	16
0.550767229649817	0.2833	0.0500	3	17
0.5474303425281031	0.2833	0.0667	4	17
0.5413676634614271	0.2833	0.0833	5	17
0.5384887344460858	0.2833	0.1000	6	17
0.5381616302353767	0.2833	0.1167	7	17
0.5361266297978856	0.2833	0.1333	8	17
0.5356608177959932	0.3000	0.1333	8	18
0.5351065625479879	0.3000	0.1500	9	18
0.5324004761681573	0.3000	0.1667	10	18
0.5316902066817418	0.3000	0.1833	11	18
0.5283586586752624	0.3000	0.2000	12	18
0.5262236718392969	0.3000	0.2167	13	18
0.5252108405987262	0.3167	0.2167	13	19
0.5238500001812392	0.3167	0.2333	14	19
0.5238107448504135	0.3167	0.2500	15	19
0.5225632112681797	0.3167	0.2667	16	19
0.520625207744978	0.3167	0.2833	17	19
0.5163669759535652	0.3167	0.3000	18	19
0.5150324141403541	0.3167	0.3167	19	19
0.511146316786026	0.3167	0.3333	20	19
0.5105189530224694	0.3167	0.3500	21	19
0.5101059569391017	0.3333	0.3500	21	20
0.5080739287643531	0.3333	0.3667	22	20
0.5070473435795869	0.3333	0.3833	23	20
0.5068350442625179	0.3500	0.3833	23	21
0.5065246857121046	0.3500	0.4000	24	21
0.5065099388482851	0.3500	0.4167	25	21
0.505361962638694	0.3500	0.4333	26	21
0.5027249368580307	0.3500	0.4500	27	21
0.5016773032731968	0.3667	0.4500	27	22
0.5015705067136768	0.3667	0.4667	28	22
0.5004545580309979	0.3833	0.4667	28	23
0.4995782893829642	0.4000	0.4667	28	24
0.4976305616322736	0.4000	0.4833	29	24
0.49622895720074345	0.4167	0.4833	29	25
0.49184065362441554	0.4167	0.5000	30	25
0.49068216980407164	0.4167	0.5167	31	25
0.48823330855793695	0.4167	0.5333	32	25
0.48778512806033375	0.4333	0.5333	32	26
0.4875756315536974	0.4333	0.5500	33	26
0.48621069202114575	0.4500	0.5500	33	27
0.4857446805145763	0.4500	0.5667	34	27
0.48547023324088145	0.4500	0.5833	35	27
0.4854526069386709	0.4500	0.6000	36	27
0.4843511996887079	0.4500	0.6167	37	27
0.48171968740110926	0.4667	0.6167	37	28
0.48080517367823394	0.4667	0.6333	38	28
0.48050441292977325	0.4667	0.6500	39	28
0.47965154810511074	0.4667	0.6667	40	28
0.4780701313132875	0.4667	0.6833	41	28
0.47713662210839813	0.4667	0.7000	42	28
0.4769493379583264	0.4667	0.7167	43	28
0.47412437307021776	0.4833	0.7167	43	29
0.4732195522418564	0.4833	0.7333	44	29
0.4726604664321759	0.4833	0.7500	45	29
0.471859143300561	0.4833	0.7667	46	29
0.4715990647242956	0.5000	0.7667	46	30
0.471148239203607	0.5000	0.7833	47	30
0.4707190641194005	0.5000	0.8000	48	30
0.467793445873459	0.5000	0.8167	49	30
0.46639844608778824	0.5167	0.8167	49	31
0.46491086084540756	0.5167	0.8333	50	31
0.46324922465737467	0.5167	0.8500	51	31
0.4608631568414421	0.5167	0.8667	52	31
0.4604049801880145	0.5167	0.8833	53	31
0.46039315106167955	0.5167	0.9000	54	31
0.4588026717432676	0.5333	0.9000	54	32
0.45811081502614676	0.5500	0.9000	54	33
0.4562065263071762	0.5667	0.9000	54	34
0.45466307539702283	0.5667	0.9167	55	34
0.452067773943587	0.5667	0.9333	56	34
0.450857469745394	0.5667	0.9500	57	34
0.45010550557564205	0.5667	0.9667	58	34
0.4480415640494636	0.5833	0.9667	58	35
0.4479901193220467	0.6000	0.9667	58	36
0.44756728122508393	0.6167	0.9667	58	37
0.44580400612414495	0.6333	0.9667	58	38
0.4388525706537992	0.6500	0.9667	58	39
0.4385657704797035	0.6667	0.9667	58	40
0.4378380922356278	0.6667	0.9833	59	40
0.43721955089783315	0.6667	1.0000	60	40
0.4363465277764172	0.6833	1.0000	60	41
0.4362799428920005	0.7000	1.0000	60	42
0.43291497057980377	0.7167	1.0000	60	43
0.43091438958006273	0.7333	1.0000	60	44
0.4303461569144904	0.7500	1.0000	60	45
0.4298872455591114	0.7667	1.0000	60	46
0.42802125434482435	0.7833	1.0000	60	47
0.4275124959693134	0.8000	1.0000	60	48
0.42666765203298607	0.8167	1.0000	60	49
0.4248724791890925	0.8333	1.0000	60	50
0.42345360330105014	0.8500	1.0000	60	51
0.4229231241108101	0.8667	1.0000	60	52
0.4225519649546344	0.8833	1.0000	60	53
0.42167867873738824	0.9000	1.0000	60	54
0.42045748201237054	0.9167	1.0000	60	55
0.4168110215906565	0.9333	1.0000	60	56
0.40675888070368127	0.9500	1.0000	60	57
0.4019645209238227	0.9667	1.0000	60	58
0.40019027908512994	0.9833	1.0000	60	59
0.3984506494478839	1.0000	1.0000	60	60

Precision–recall points

Threshold	Recall	Precision	TP	FP
Above 1 (all negative)	0.0000	1.0000	0	0
0.5851089298508227	0.0000	0.0000	0	1
0.5798626689572249	0.0000	0.0000	0	2
0.5786938660457515	0.0000	0.0000	0	3
0.574657217234697	0.0000	0.0000	0	4
0.5728524867921336	0.0000	0.0000	0	5
0.5711681816417851	0.0000	0.0000	0	6
0.5697269051971212	0.0000	0.0000	0	7
0.5677875926065666	0.0000	0.0000	0	8
0.5677005076487734	0.0000	0.0000	0	9
0.5654156665192498	0.0000	0.0000	0	10
0.5630707434961463	0.0000	0.0000	0	11
0.561628445383969	0.0000	0.0000	0	12
0.5613327864144256	0.0000	0.0000	0	13
0.5607407535654877	0.0167	0.0714	1	13
0.5551147459904425	0.0167	0.0667	1	14
0.5528625222685883	0.0333	0.1250	2	14
0.5525715792295951	0.0500	0.1765	3	14
0.5520025386731356	0.0500	0.1667	3	15
0.5514090180434148	0.0500	0.1579	3	16
0.550767229649817	0.0500	0.1500	3	17
0.5474303425281031	0.0667	0.1905	4	17
0.5413676634614271	0.0833	0.2273	5	17
0.5384887344460858	0.1000	0.2609	6	17
0.5381616302353767	0.1167	0.2917	7	17
0.5361266297978856	0.1333	0.3200	8	17
0.5356608177959932	0.1333	0.3077	8	18
0.5351065625479879	0.1500	0.3333	9	18
0.5324004761681573	0.1667	0.3571	10	18
0.5316902066817418	0.1833	0.3793	11	18
0.5283586586752624	0.2000	0.4000	12	18
0.5262236718392969	0.2167	0.4194	13	18
0.5252108405987262	0.2167	0.4062	13	19
0.5238500001812392	0.2333	0.4242	14	19
0.5238107448504135	0.2500	0.4412	15	19
0.5225632112681797	0.2667	0.4571	16	19
0.520625207744978	0.2833	0.4722	17	19
0.5163669759535652	0.3000	0.4865	18	19
0.5150324141403541	0.3167	0.5000	19	19
0.511146316786026	0.3333	0.5128	20	19
0.5105189530224694	0.3500	0.5250	21	19
0.5101059569391017	0.3500	0.5122	21	20
0.5080739287643531	0.3667	0.5238	22	20
0.5070473435795869	0.3833	0.5349	23	20
0.5068350442625179	0.3833	0.5227	23	21
0.5065246857121046	0.4000	0.5333	24	21
0.5065099388482851	0.4167	0.5435	25	21
0.505361962638694	0.4333	0.5532	26	21
0.5027249368580307	0.4500	0.5625	27	21
0.5016773032731968	0.4500	0.5510	27	22
0.5015705067136768	0.4667	0.5600	28	22
0.5004545580309979	0.4667	0.5490	28	23
0.4995782893829642	0.4667	0.5385	28	24
0.4976305616322736	0.4833	0.5472	29	24
0.49622895720074345	0.4833	0.5370	29	25
0.49184065362441554	0.5000	0.5455	30	25
0.49068216980407164	0.5167	0.5536	31	25
0.48823330855793695	0.5333	0.5614	32	25
0.48778512806033375	0.5333	0.5517	32	26
0.4875756315536974	0.5500	0.5593	33	26
0.48621069202114575	0.5500	0.5500	33	27
0.4857446805145763	0.5667	0.5574	34	27
0.48547023324088145	0.5833	0.5645	35	27
0.4854526069386709	0.6000	0.5714	36	27
0.4843511996887079	0.6167	0.5781	37	27
0.48171968740110926	0.6167	0.5692	37	28
0.48080517367823394	0.6333	0.5758	38	28
0.48050441292977325	0.6500	0.5821	39	28
0.47965154810511074	0.6667	0.5882	40	28
0.4780701313132875	0.6833	0.5942	41	28
0.47713662210839813	0.7000	0.6000	42	28
0.4769493379583264	0.7167	0.6056	43	28
0.47412437307021776	0.7167	0.5972	43	29
0.4732195522418564	0.7333	0.6027	44	29
0.4726604664321759	0.7500	0.6081	45	29
0.471859143300561	0.7667	0.6133	46	29
0.4715990647242956	0.7667	0.6053	46	30
0.471148239203607	0.7833	0.6104	47	30
0.4707190641194005	0.8000	0.6154	48	30
0.467793445873459	0.8167	0.6203	49	30
0.46639844608778824	0.8167	0.6125	49	31
0.46491086084540756	0.8333	0.6173	50	31
0.46324922465737467	0.8500	0.6220	51	31
0.4608631568414421	0.8667	0.6265	52	31
0.4604049801880145	0.8833	0.6310	53	31
0.46039315106167955	0.9000	0.6353	54	31
0.4588026717432676	0.9000	0.6279	54	32
0.45811081502614676	0.9000	0.6207	54	33
0.4562065263071762	0.9000	0.6136	54	34
0.45466307539702283	0.9167	0.6180	55	34
0.452067773943587	0.9333	0.6222	56	34
0.450857469745394	0.9500	0.6264	57	34
0.45010550557564205	0.9667	0.6304	58	34
0.4480415640494636	0.9667	0.6237	58	35
0.4479901193220467	0.9667	0.6170	58	36
0.44756728122508393	0.9667	0.6105	58	37
0.44580400612414495	0.9667	0.6042	58	38
0.4388525706537992	0.9667	0.5979	58	39
0.4385657704797035	0.9667	0.5918	58	40
0.4378380922356278	0.9833	0.5960	59	40
0.43721955089783315	1.0000	0.6000	60	40
0.4363465277764172	1.0000	0.5941	60	41
0.4362799428920005	1.0000	0.5882	60	42
0.43291497057980377	1.0000	0.5825	60	43
0.43091438958006273	1.0000	0.5769	60	44
0.4303461569144904	1.0000	0.5714	60	45
0.4298872455591114	1.0000	0.5660	60	46
0.42802125434482435	1.0000	0.5607	60	47
0.4275124959693134	1.0000	0.5556	60	48
0.42666765203298607	1.0000	0.5505	60	49
0.4248724791890925	1.0000	0.5455	60	50
0.42345360330105014	1.0000	0.5405	60	51
0.4229231241108101	1.0000	0.5357	60	52
0.4225519649546344	1.0000	0.5310	60	53
0.42167867873738824	1.0000	0.5263	60	54
0.42045748201237054	1.0000	0.5217	60	55
0.4168110215906565	1.0000	0.5172	60	56
0.40675888070368127	1.0000	0.5128	60	57
0.4019645209238227	1.0000	0.5085	60	58
0.40019027908512994	1.0000	0.5042	60	59
0.3984506494478839	1.0000	0.5000	60	60

Work through one threshold

Suppose targets are [0, 1, 0, 1] and outputs are [0.1, 0.4, 0.4, 0.9]. At τ = 0.5: TP = 1, FP = 0, FN = 1, TN = 2. If a missed positive costs 3 and a false alarm costs 1, mean cost is (0 + 3) / 4 = 0.75.

At τ = 0.4, both tied 0.4 scores enter class 1: TP = 2, FP = 1, FN = 0, TN = 1. Mean cost falls to 1 / 4 = 0.25. Moving a threshold changes decisions; probabilities and weights stay fixed.

There are four positive–negative score pairs: three wins and one tie, so ROC AUC = (3 + ½) / 4 = 7/8. Recall rises by ½ at precision 1, then by ½ at precision 2/3, giving AP = ½ + ⅓ = 5/6.

Brier = (0.1² + 0.6² + 0.4² + 0.1²) / 4 = 0.135. With two bins [0, 0.5) and [0.5, 1], ECE = (3/4) × |0.3 − 1/3| + (1/4) × |0.9 − 1| = 0.05. The recorded report uses ten bins; ECE changes with the binning choice.

Why use validation for the threshold?

The threshold is another selected setting. Choosing it to minimize test mistakes uses the test labels for model selection. This report chooses among the validation scores, 0.5 and an all-negative rule. Ties prefer the threshold closest to 0.5, then the higher threshold.

How to read calibration

In a bin with mean output 0.7 and an observed positive fraction of 0.5, the absolute calibration gap is 0.2. ECE averages these gaps weighted by bin count. Brier is the mean of (p − y)². Neither measure makes a small synthetic test into a guarantee about future data.

05 / GRADIENT LAB

Take a step. Understand why it works.

Temporary sandbox · 0 steps

Choose a weight or bias. Compare its exact backpropagation derivative with a numerical estimate, then see a proposed descent step before applying it. These calculations use only a small, fixed batch of training examples.

This sandbox changes a temporary copy in this tab. The recorded model, replay, validation-selected threshold and held-out results stay fixed. Reloading discards sandbox steps. This is plain gradient descent, even when the original run used Adam.

Starting model Training batch L2 coefficient λ

Changing the starting model, batch or λ resets the sandbox. No test or validation rows enter these calculations.

Parameter to inspectSlice radius ±

Actual objective J Tangent J + g·δAll other parameters fixed

Read numerical slice values

Parameter change δ	Actual J	Tangent J + g·δ

Analytic derivative g—

Central difference—

Absolute difference—

Finite-difference ε

Numerical derivative = [J(θ + ε) − J(θ − ε)] / (2ε). Smaller ε is not always more accurate: rounding competes with approximation error.

Inspect the sandbox step history

Step	Mean BCE	L2 penalty	Objective J	Actual change

Each row uses the same batch and λ. This is a training objective, not a new estimate of held-out performance.

Follow the derivative through the network

Standardize each raw input using the saved training mean and scale. For B training rows, the output derivative is dZ₂ = (P − Y) / B. Then dW₂ = HᵀdZ₂ + λW₂, dZ₁ = (dZ₂W₂ᵀ) ⊙ (1 − H²), and dW₁ = X̃ᵀdZ₁ + λW₁. Bias derivatives sum over rows and have no L2 penalty.

The objective is mean BCE plus ½λ(‖W₁‖² + ‖W₂‖²). Computing BCE from logits avoids taking log(0) when sigmoid rounds to 0 or 1.

Why can a descent step increase loss?

The tangent predicts ΔJ ≈ −ηg² for one parameter, or −η‖∇J‖² for all parameters. This prediction is local. Curvature matters for a finite step, so a large learning rate can overshoot.

The chart varies only the selected coordinate; an all-parameter step follows another direction. Neither a flat slice nor a small gradient proves a global minimum. Compare the predicted and actual changes, then use Undo to try a smaller η.

Work one update by hand

Take one standardized input [0, 0] with target 1, one hidden neuron, W₁ = [1, −1]ᵀ, b₁ = 0, W₂ = 2, b₂ = 0 and λ = 0.1. Then h = 0, p = 0.5, BCE = log(2), penalty = 0.3 and J ≈ 0.993147.

For b₂, the derivative is p − y = −0.5. A coordinate step with η = 0.2 makes b₂ = 0.1. The predicted change is −0.05. The actual objective becomes log(1 + exp(−0.1)) + 0.3 ≈ 0.944397: a change of −0.048751.

More derivations and exercises: Gradients & optimization and Optimization · exercises.

06 / CONNECT CODE TO MATHEMATICS

The whole model, in four lines.

01X̃ = (X − μ_train) / σ_train

02H = tanh(X̃W₁ + b₁)

03P = sigmoid(HW₂ + b₂)

04J = mean BCE + ½λ(‖W₁‖² + ‖W₂‖²)

What makes this a learning experiment?

Mini-batches estimate a gradient from a subset of training rows. Adam tracks first and second gradient moments and corrects their initial bias. L2 penalizes large weights. Early stopping chooses when to stop using validation behavior.

Python training uses explicit NumPy derivatives and optimizer updates. The gradient sandbox mirrors the network calculations in JavaScript. Neither uses an automatic differentiation framework.

Why divide by the batch size?

The data loss is a mean. Its derivative begins with dZ₂ = (P − Y) / B, where B is the current batch size. The last, shorter batch uses its own size. The L2 derivative is λW; biases receive no L2 penalty.

Why not fit the scaler on every point?

Using validation or test values to calculate the mean and standard deviation lets evaluation data influence training. This lab fits μ and σ on training inputs only, then applies those same numbers everywhere.

Why compare with a linear model?

A network can fit curved boundaries, but a more complex model is not automatically better. The baseline makes the added hidden layer's effect measurable for this experiment.

07 / YOUR NEXT EXPERIMENT

Change one idea. Explain the result.

Try SGD, a different hidden width, or stronger regularization. Choose settings using validation results. Treat the test set as the final audit, rather than a scoreboard to optimize.

python -m neural_math_lab experiment --dataset circles --samples 600 --noise 0.15 --hidden 16 --seed 7 --epochs 1200 --learning-rate 0.01 --optimizer adam --batch-size 32 --l2 0.001 --patience 100 --min-delta 1e-05 --fp-cost 1.0 --fn-cost 1.0 --output runs/new-experiment

The command runs locally in Python. Choose a new output directory each time.

Read alongside the experiment

Gradients & optimization
Gradients, loss slices, and one update worked by hand.

Optimization · exercises
Eight exercises with worked answers.

Training beyond XOR
Derivations, assumptions, and a worked Adam update.

Decisions, ranking & calibration
Decision costs, ranking, calibration, and raw-input derivatives.

Decisions · exercises
Worked threshold, ranking, calibration, and sensitivity exercises.

Training · exercises
Experiments and answers you can check yourself.

Configuration & provenance

{
  "config": {
    "dataset": "circles",
    "samples": 600,
    "noise": 0.15,
    "hidden": 16,
    "seed": 7,
    "epochs": 1200,
    "learning_rate": 0.01,
    "optimizer": "adam",
    "batch_size": 32,
    "l2": 0.001,
    "patience": 100,
    "min_delta": 1e-05,
    "fp_cost": 1.0,
    "fn_cost": 1.0
  },
  "seeds": {
    "data": 7,
    "split": 8,
    "initialization": 9,
    "network_batches": 10,
    "baseline_batches": 11
  },
  "environment": {
    "python": "3.9.6",
    "numpy": "2.0.2",
    "matplotlib": "3.9.4"
  }
}

See a networklearn the shape.

A boundary built from simple operations.

Training improves. Validation keeps it honest.

Read this curve.

More than one accuracy number.

Same held-out examples. Two models.

Neural network confusion matrix

A probability becomes a decision.

Ranking · ROC

Retrieval · precision–recall

Probability · reliability

Evaluate the selected rules once.

Neural network · saved validation diagnostics

Reliability bins

Linear baseline · saved validation diagnostics

Reliability bins

Take a step. Understand why it works.

The whole model, in four lines.

What makes this a learning experiment?

Change one idea. Explain the result.

See a network
learn the shape.