LAB 02 / Moons / MANUAL NUMPY

See a network
learn the shape.

Follow the boundary as it changes. Inspect each neuron. Then ask the important question: does it work on examples it never trained on?

Held-out test accuracy96.7%120 unseen examples · selected model

Class 1 · F1 score0.967Combines precision and recall

Against linear baseline+4.2 ppBaseline accuracy: 92.5%

Selected epoch777Lowest validation BCE · stopped at 877

The model was selected using validation loss. The held-out test set was evaluated afterward. These synthetic results describe this split and seed; repeated tuning against the test score would invalidate that separation.

01 / REPLAY & EXPLORE

A boundary built from simple operations.

Selected checkpoint

ModelExamples Mark mistakes

Class 0 · square Class 1 · circle× Incorrect classp = 0p = 1

Saved training checkpoint

The map displays the selected model. Final metric cards above stay fixed while you replay training.

02 / LOOK FOR GENERALIZATION

Training improves. Validation keeps it honest.

Stopped at epoch 877

Training BCE Validation BCEDashed line: selected epoch

Read this curve.

The plotted loss excludes the L2 penalty, so the training and validation curves measure the same quantity. Training updates include the penalty on weights.

The lowest observed validation loss selects the saved weights. Patience stops training after too many epochs without a meaningful improvement; those can be different epochs.

This run stopped because: validation patience. The saved model restores epoch 777. The selected checkpoint is included in the replay.

View the exportable static figures

03 / CHECK THE EVIDENCE

More than one accuracy number.

Threshold p ≥ 0.5

Same held-out examples. Two models.

Model	Accuracy	Precision	Recall	F1	Brier ↓
Neural network	96.7%	0.952	0.983	0.967	0.0180
Linear baseline	92.5%	0.918	0.933	0.926	0.0596

The baseline has one affine score and a sigmoid, with no hidden layer. It uses the same split and training-only standardizer. Its own validation loss selects its weights.

Neural network confusion matrix

Actual ↓ / predicted →
	Class 0	Class 1
Class 0	57	3
Class 1	1	59

Rows are actual classes; columns are predicted classes. Precision and recall refer to class 1. Undefined ratios are reported as 0.

Inspect each data split

Split	Examples	Accuracy	BCE	Role
Training	360	99.2%	0.03680	Gradient updates + scaler
Validation	120	98.3%	0.05477	Choose weights and stop
Test	120	96.7%	0.07208	Final evaluation only

The split is stratified: each class is divided approximately 60/20/20. Every example ID belongs to exactly one split.

Show misclassified test examples

Row ID	x1	x2	Target	Output p	Predicted
378	0.4679	0.2594	1	0.478665	0
320	-0.1416	0.6179	0	0.611637	1
259	1.3847	-0.1063	0	0.956249	1
315	-0.0129	0.6778	0	0.595417	1

Error analysis describes this completed run. Use fresh evaluation data for future model-selection decisions.

04 / DECIDE WITH DIFFERENT COSTS

A probability becomes a decision.

Validation exploration only

A false alarm and a missed positive need not cost the same. Python selected the threshold with the lowest validation cost for each model. Explore that trade-off below without changing its saved weights or final test evaluation.

Diagnostic modelDecision threshold τExact threshold

Controls require JavaScript. Saved values remain readable below.

False positives—

False negatives—

Recall · class 1—

Precision · class 1—

Cost per example—

Mean cost = (cFP × FP + cFN × FN) / N

Prediction is class 1 when p ≥ τ. Equal scores move together. The no-positive option sets τ just above 1. Undefined precision or recall is displayed as 0, matching the original metrics.

Ranking · ROC

Retrieval · precision–recall

Probability · reliability

These curves use the selected model’s validation scores and stay fixed during threshold exploration and checkpoint replay. The dot on ROC and precision–recall marks the explored decision. A good ranking does not imply calibrated probabilities. Empty calibration bins are omitted; small bins are noisy.

FINAL TEST EVALUATION · FIXED

Evaluate the selected rules once.

These values use each model’s validation-selected threshold. The controls above never recalculate this table on test data.

Model	Selected τ	FP	FN	Mean cost	Accuracy	ROC AUC	AP	Brier
Neural network	0.6830711918697014	1	1	0.0167	0.9833	0.9975	0.9975	0.0180
Linear baseline	0.5	5	4	0.0750	0.9250	0.9764	0.9753	0.0596

The earlier accuracy cards use threshold 0.5. This table uses the cost-selected rules, so their counts may differ. Test scores are descriptive for this synthetic split.

Read saved validation results and all curve points

Neural network · saved validation diagnostics

Selected threshold	FP cost	FN cost	Validation mean cost	Candidates
0.6830711918697014	1.0	1.0	0.0083	122

ROC AUC: 0.9992 · Average precision: 0.9992 · Brier: 0.0130 · ECE: 0.0401. Curves below use validation scores.

Reliability bins

Bin (left inclusive)	Count	Mean p	Positive fraction	Absolute gap
[0.0, 0.1)	54	0.0157	0.0000	0.0157
[0.1, 0.2)	2	0.1210	0.0000	0.1210
[0.2, 0.3)	2	0.2394	0.5000	0.2606
[0.3, 0.4)	1	0.3065	0.0000	0.3065
[0.4, 0.5)	1	0.4806	0.0000	0.4806
[0.5, 0.6)	1	0.5304	0.0000	0.5304
[0.6, 0.7)	1	0.6831	1.0000	0.3169
[0.7, 0.8)	1	0.7058	1.0000	0.2942
[0.8, 0.9)	4	0.8649	1.0000	0.1351
[0.9, 1.0]	53	0.9862	1.0000	0.0138

Empty bins have undefined means and are omitted from the plot. ECE is the count-weighted absolute bin gap.

ROC points

Threshold	False-positive rate	True-positive rate	TP	FP
Above 1 (all negative)	0.0000	0.0000	0	0
0.9998570990155635	0.0000	0.0167	1	0
0.9994912032497504	0.0000	0.0333	2	0
0.999198475776245	0.0000	0.0500	3	0
0.9987067596374886	0.0000	0.0667	4	0
0.9982762142724881	0.0000	0.0833	5	0
0.9982391677480764	0.0000	0.1000	6	0
0.9980509534361596	0.0000	0.1167	7	0
0.9976079219466578	0.0000	0.1333	8	0
0.9975259935046923	0.0000	0.1500	9	0
0.997215235256537	0.0000	0.1667	10	0
0.9969814313650801	0.0000	0.1833	11	0
0.9968438698770516	0.0000	0.2000	12	0
0.996739277732754	0.0000	0.2167	13	0
0.9962990838955852	0.0000	0.2333	14	0
0.9959105821630361	0.0000	0.2500	15	0
0.9950392573263098	0.0000	0.2667	16	0
0.9942947293735998	0.0000	0.2833	17	0
0.9941568085634186	0.0000	0.3000	18	0
0.994106445499155	0.0000	0.3167	19	0
0.9938113564316734	0.0000	0.3333	20	0
0.9936755090252735	0.0000	0.3500	21	0
0.9930804799196731	0.0000	0.3667	22	0
0.9921902828142423	0.0000	0.3833	23	0
0.9918936546009135	0.0000	0.4000	24	0
0.9918667176762911	0.0000	0.4167	25	0
0.9918357883975253	0.0000	0.4333	26	0
0.9917126773887373	0.0000	0.4500	27	0
0.9908920305534059	0.0000	0.4667	28	0
0.9899991053536784	0.0000	0.4833	29	0
0.9899922136271722	0.0000	0.5000	30	0
0.9899209478688084	0.0000	0.5167	31	0
0.9896347858952167	0.0000	0.5333	32	0
0.9884062608778107	0.0000	0.5500	33	0
0.988290020776004	0.0000	0.5667	34	0
0.9873433148395171	0.0000	0.5833	35	0
0.9868801559592073	0.0000	0.6000	36	0
0.9847549683629734	0.0000	0.6167	37	0
0.9843265329034574	0.0000	0.6333	38	0
0.9841154806466724	0.0000	0.6500	39	0
0.9832919113328149	0.0000	0.6667	40	0
0.9830579633230805	0.0000	0.6833	41	0
0.9822171463517679	0.0000	0.7000	42	0
0.9804197911173849	0.0000	0.7167	43	0
0.9782673119670493	0.0000	0.7333	44	0
0.9771697662917732	0.0000	0.7500	45	0
0.974012899767483	0.0000	0.7667	46	0
0.9729976984856966	0.0000	0.7833	47	0
0.9688341938984124	0.0000	0.8000	48	0
0.9583767722515181	0.0000	0.8167	49	0
0.9528171605961397	0.0000	0.8333	50	0
0.9514699439972591	0.0000	0.8500	51	0
0.9473657074104096	0.0000	0.8667	52	0
0.9176491536133309	0.0000	0.8833	53	0
0.8998832734630943	0.0000	0.9000	54	0
0.8804987450850631	0.0000	0.9167	55	0
0.874408565165438	0.0000	0.9333	56	0
0.8046954014013981	0.0000	0.9500	57	0
0.7057805311403742	0.0000	0.9667	58	0
0.6830711918697014	0.0000	0.9833	59	0
0.5304050535298334	0.0167	0.9833	59	1
0.4806268435573459	0.0333	0.9833	59	2
0.30648808338404493	0.0500	0.9833	59	3
0.2557656490118017	0.0500	1.0000	60	3
0.22295276551652915	0.0667	1.0000	60	4
0.1394695018552716	0.0833	1.0000	60	5
0.10247524499915803	0.1000	1.0000	60	6
0.07580306961871337	0.1167	1.0000	60	7
0.07339832458573817	0.1333	1.0000	60	8
0.060015503432675546	0.1500	1.0000	60	9
0.05241808172332978	0.1667	1.0000	60	10
0.04554937083039838	0.1833	1.0000	60	11
0.03888432372616596	0.2000	1.0000	60	12
0.033349812711742435	0.2167	1.0000	60	13
0.02610165668666878	0.2333	1.0000	60	14
0.022425708236070933	0.2500	1.0000	60	15
0.0221956308256229	0.2667	1.0000	60	16
0.021905841092016667	0.2833	1.0000	60	17
0.021717716048752292	0.3000	1.0000	60	18
0.02108263979005544	0.3167	1.0000	60	19
0.021066393939637433	0.3333	1.0000	60	20
0.019574737778340498	0.3500	1.0000	60	21
0.01814916967244225	0.3667	1.0000	60	22
0.017010331122431308	0.3833	1.0000	60	23
0.016835690919281908	0.4000	1.0000	60	24
0.01637893990042715	0.4167	1.0000	60	25
0.01572994934866739	0.4333	1.0000	60	26
0.01554860482956017	0.4500	1.0000	60	27
0.015411350978936814	0.4667	1.0000	60	28
0.013618186966289577	0.4833	1.0000	60	29
0.01157590349482914	0.5000	1.0000	60	30
0.011489166165248733	0.5167	1.0000	60	31
0.01029765294785929	0.5333	1.0000	60	32
0.009944087892008338	0.5500	1.0000	60	33
0.009543293117895263	0.5667	1.0000	60	34
0.008644216181028482	0.5833	1.0000	60	35
0.007124093976583278	0.6000	1.0000	60	36
0.006720924501836727	0.6167	1.0000	60	37
0.006709253554662497	0.6333	1.0000	60	38
0.006491754298348289	0.6500	1.0000	60	39
0.006245354599996432	0.6667	1.0000	60	40
0.005840826162231964	0.6833	1.0000	60	41
0.005152986122367993	0.7000	1.0000	60	42
0.004847394492104486	0.7167	1.0000	60	43
0.004522442613295763	0.7333	1.0000	60	44
0.004302435909602914	0.7500	1.0000	60	45
0.004176379028796375	0.7667	1.0000	60	46
0.003985676344105112	0.7833	1.0000	60	47
0.0038226572786859525	0.8000	1.0000	60	48
0.0037028946523144654	0.8167	1.0000	60	49
0.0035174409121381243	0.8333	1.0000	60	50
0.003430946818825408	0.8500	1.0000	60	51
0.0033102690502964057	0.8667	1.0000	60	52
0.0031056636701987223	0.8833	1.0000	60	53
0.0029362372352052073	0.9000	1.0000	60	54
0.002926337838199849	0.9167	1.0000	60	55
0.0028889164291862977	0.9333	1.0000	60	56
0.0028057259424077116	0.9500	1.0000	60	57
0.002299449720473628	0.9667	1.0000	60	58
0.0005892358918267316	0.9833	1.0000	60	59
0.0003414243366668133	1.0000	1.0000	60	60

Precision–recall points

Threshold	Recall	Precision	TP	FP
Above 1 (all negative)	0.0000	1.0000	0	0
0.9998570990155635	0.0167	1.0000	1	0
0.9994912032497504	0.0333	1.0000	2	0
0.999198475776245	0.0500	1.0000	3	0
0.9987067596374886	0.0667	1.0000	4	0
0.9982762142724881	0.0833	1.0000	5	0
0.9982391677480764	0.1000	1.0000	6	0
0.9980509534361596	0.1167	1.0000	7	0
0.9976079219466578	0.1333	1.0000	8	0
0.9975259935046923	0.1500	1.0000	9	0
0.997215235256537	0.1667	1.0000	10	0
0.9969814313650801	0.1833	1.0000	11	0
0.9968438698770516	0.2000	1.0000	12	0
0.996739277732754	0.2167	1.0000	13	0
0.9962990838955852	0.2333	1.0000	14	0
0.9959105821630361	0.2500	1.0000	15	0
0.9950392573263098	0.2667	1.0000	16	0
0.9942947293735998	0.2833	1.0000	17	0
0.9941568085634186	0.3000	1.0000	18	0
0.994106445499155	0.3167	1.0000	19	0
0.9938113564316734	0.3333	1.0000	20	0
0.9936755090252735	0.3500	1.0000	21	0
0.9930804799196731	0.3667	1.0000	22	0
0.9921902828142423	0.3833	1.0000	23	0
0.9918936546009135	0.4000	1.0000	24	0
0.9918667176762911	0.4167	1.0000	25	0
0.9918357883975253	0.4333	1.0000	26	0
0.9917126773887373	0.4500	1.0000	27	0
0.9908920305534059	0.4667	1.0000	28	0
0.9899991053536784	0.4833	1.0000	29	0
0.9899922136271722	0.5000	1.0000	30	0
0.9899209478688084	0.5167	1.0000	31	0
0.9896347858952167	0.5333	1.0000	32	0
0.9884062608778107	0.5500	1.0000	33	0
0.988290020776004	0.5667	1.0000	34	0
0.9873433148395171	0.5833	1.0000	35	0
0.9868801559592073	0.6000	1.0000	36	0
0.9847549683629734	0.6167	1.0000	37	0
0.9843265329034574	0.6333	1.0000	38	0
0.9841154806466724	0.6500	1.0000	39	0
0.9832919113328149	0.6667	1.0000	40	0
0.9830579633230805	0.6833	1.0000	41	0
0.9822171463517679	0.7000	1.0000	42	0
0.9804197911173849	0.7167	1.0000	43	0
0.9782673119670493	0.7333	1.0000	44	0
0.9771697662917732	0.7500	1.0000	45	0
0.974012899767483	0.7667	1.0000	46	0
0.9729976984856966	0.7833	1.0000	47	0
0.9688341938984124	0.8000	1.0000	48	0
0.9583767722515181	0.8167	1.0000	49	0
0.9528171605961397	0.8333	1.0000	50	0
0.9514699439972591	0.8500	1.0000	51	0
0.9473657074104096	0.8667	1.0000	52	0
0.9176491536133309	0.8833	1.0000	53	0
0.8998832734630943	0.9000	1.0000	54	0
0.8804987450850631	0.9167	1.0000	55	0
0.874408565165438	0.9333	1.0000	56	0
0.8046954014013981	0.9500	1.0000	57	0
0.7057805311403742	0.9667	1.0000	58	0
0.6830711918697014	0.9833	1.0000	59	0
0.5304050535298334	0.9833	0.9833	59	1
0.4806268435573459	0.9833	0.9672	59	2
0.30648808338404493	0.9833	0.9516	59	3
0.2557656490118017	1.0000	0.9524	60	3
0.22295276551652915	1.0000	0.9375	60	4
0.1394695018552716	1.0000	0.9231	60	5
0.10247524499915803	1.0000	0.9091	60	6
0.07580306961871337	1.0000	0.8955	60	7
0.07339832458573817	1.0000	0.8824	60	8
0.060015503432675546	1.0000	0.8696	60	9
0.05241808172332978	1.0000	0.8571	60	10
0.04554937083039838	1.0000	0.8451	60	11
0.03888432372616596	1.0000	0.8333	60	12
0.033349812711742435	1.0000	0.8219	60	13
0.02610165668666878	1.0000	0.8108	60	14
0.022425708236070933	1.0000	0.8000	60	15
0.0221956308256229	1.0000	0.7895	60	16
0.021905841092016667	1.0000	0.7792	60	17
0.021717716048752292	1.0000	0.7692	60	18
0.02108263979005544	1.0000	0.7595	60	19
0.021066393939637433	1.0000	0.7500	60	20
0.019574737778340498	1.0000	0.7407	60	21
0.01814916967244225	1.0000	0.7317	60	22
0.017010331122431308	1.0000	0.7229	60	23
0.016835690919281908	1.0000	0.7143	60	24
0.01637893990042715	1.0000	0.7059	60	25
0.01572994934866739	1.0000	0.6977	60	26
0.01554860482956017	1.0000	0.6897	60	27
0.015411350978936814	1.0000	0.6818	60	28
0.013618186966289577	1.0000	0.6742	60	29
0.01157590349482914	1.0000	0.6667	60	30
0.011489166165248733	1.0000	0.6593	60	31
0.01029765294785929	1.0000	0.6522	60	32
0.009944087892008338	1.0000	0.6452	60	33
0.009543293117895263	1.0000	0.6383	60	34
0.008644216181028482	1.0000	0.6316	60	35
0.007124093976583278	1.0000	0.6250	60	36
0.006720924501836727	1.0000	0.6186	60	37
0.006709253554662497	1.0000	0.6122	60	38
0.006491754298348289	1.0000	0.6061	60	39
0.006245354599996432	1.0000	0.6000	60	40
0.005840826162231964	1.0000	0.5941	60	41
0.005152986122367993	1.0000	0.5882	60	42
0.004847394492104486	1.0000	0.5825	60	43
0.004522442613295763	1.0000	0.5769	60	44
0.004302435909602914	1.0000	0.5714	60	45
0.004176379028796375	1.0000	0.5660	60	46
0.003985676344105112	1.0000	0.5607	60	47
0.0038226572786859525	1.0000	0.5556	60	48
0.0037028946523144654	1.0000	0.5505	60	49
0.0035174409121381243	1.0000	0.5455	60	50
0.003430946818825408	1.0000	0.5405	60	51
0.0033102690502964057	1.0000	0.5357	60	52
0.0031056636701987223	1.0000	0.5310	60	53
0.0029362372352052073	1.0000	0.5263	60	54
0.002926337838199849	1.0000	0.5217	60	55
0.0028889164291862977	1.0000	0.5172	60	56
0.0028057259424077116	1.0000	0.5128	60	57
0.002299449720473628	1.0000	0.5085	60	58
0.0005892358918267316	1.0000	0.5042	60	59
0.0003414243366668133	1.0000	0.5000	60	60

Linear baseline · saved validation diagnostics

Selected threshold	FP cost	FN cost	Validation mean cost	Candidates
0.5	1.0	1.0	0.1333	122

ROC AUC: 0.9419 · Average precision: 0.9475 · Brier: 0.0992 · ECE: 0.0778. Curves below use validation scores.

Reliability bins

Bin (left inclusive)	Count	Mean p	Positive fraction	Absolute gap
[0.0, 0.1)	35	0.0325	0.0286	0.0039
[0.1, 0.2)	9	0.1502	0.2222	0.0720
[0.2, 0.3)	5	0.2543	0.0000	0.2543
[0.3, 0.4)	7	0.3577	0.7143	0.3566
[0.4, 0.5)	4	0.4576	0.0000	0.4576
[0.5, 0.6)	4	0.5891	0.7500	0.1609
[0.6, 0.7)	6	0.6426	0.6667	0.0240
[0.7, 0.8)	7	0.7452	0.5714	0.1738
[0.8, 0.9)	3	0.8742	0.6667	0.2075
[0.9, 1.0]	40	0.9669	0.9750	0.0081

Empty bins have undefined means and are omitted from the plot. ECE is the count-weighted absolute bin gap.

ROC points

Threshold	False-positive rate	True-positive rate	TP	FP
Above 1 (all negative)	0.0000	0.0000	0	0
0.9970893778022378	0.0000	0.0167	1	0
0.9968987778837627	0.0000	0.0333	2	0
0.9962179163948189	0.0000	0.0500	3	0
0.9952448098315636	0.0000	0.0667	4	0
0.9946046374483011	0.0000	0.0833	5	0
0.9935855738285388	0.0000	0.1000	6	0
0.9926493786858706	0.0000	0.1167	7	0
0.9899993309972164	0.0000	0.1333	8	0
0.9892875157233764	0.0000	0.1500	9	0
0.9872181734520095	0.0000	0.1667	10	0
0.9828753223462144	0.0000	0.1833	11	0
0.9806849675726343	0.0000	0.2000	12	0
0.9805658105705933	0.0000	0.2167	13	0
0.9801247572254904	0.0000	0.2333	14	0
0.9796173347037235	0.0000	0.2500	15	0
0.9788308759855273	0.0000	0.2667	16	0
0.9772559921986599	0.0000	0.2833	17	0
0.9752522184841321	0.0000	0.3000	18	0
0.9725744737406108	0.0000	0.3167	19	0
0.972317532926128	0.0000	0.3333	20	0
0.971220425571667	0.0000	0.3500	21	0
0.9704162110929353	0.0000	0.3667	22	0
0.9703468275226623	0.0000	0.3833	23	0
0.9689402085630805	0.0000	0.4000	24	0
0.9683246734716652	0.0000	0.4167	25	0
0.9657512545244449	0.0000	0.4333	26	0
0.960130694920936	0.0000	0.4500	27	0
0.9589463880796237	0.0000	0.4667	28	0
0.9532317568908516	0.0000	0.4833	29	0
0.9529388596101162	0.0000	0.5000	30	0
0.9473141104533185	0.0000	0.5167	31	0
0.9449634911251589	0.0000	0.5333	32	0
0.9422924490501764	0.0000	0.5500	33	0
0.9406764773509962	0.0000	0.5667	34	0
0.9401545634993372	0.0000	0.5833	35	0
0.9364347838624589	0.0000	0.6000	36	0
0.9269762316305228	0.0000	0.6167	37	0
0.9266656966158918	0.0167	0.6167	37	1
0.9142282154668511	0.0167	0.6333	38	1
0.9024497679543143	0.0167	0.6500	39	1
0.8846596485921231	0.0333	0.6500	39	2
0.8827160392568102	0.0333	0.6667	40	2
0.8550943219363668	0.0333	0.6833	41	2
0.7975495838365965	0.0500	0.6833	41	3
0.7875335779427142	0.0500	0.7000	42	3
0.7425966105705569	0.0667	0.7000	42	4
0.7396711594927651	0.0667	0.7167	43	4
0.7306899435243107	0.0667	0.7333	44	4
0.7133375411360531	0.0667	0.7500	45	4
0.70516394833072	0.0833	0.7500	45	5
0.6892130108514639	0.0833	0.7667	46	5
0.6764060119523394	0.1000	0.7667	46	6
0.6371414184795541	0.1167	0.7667	46	7
0.6303997388439126	0.1167	0.7833	47	7
0.6206797214949554	0.1167	0.8000	48	7
0.6020427185773436	0.1167	0.8167	49	7
0.5925107128652473	0.1167	0.8333	50	7
0.5893615498211943	0.1333	0.8333	50	8
0.5876629017933523	0.1333	0.8500	51	8
0.5868653685118453	0.1333	0.8667	52	8
0.499948274022567	0.1500	0.8667	52	9
0.454047889362915	0.1667	0.8667	52	10
0.4488733512793822	0.1833	0.8667	52	11
0.427641063910166	0.2000	0.8667	52	12
0.38551063221977444	0.2000	0.8833	53	12
0.36919318339846635	0.2000	0.9000	54	12
0.3639608427090342	0.2000	0.9167	55	12
0.36366044873052633	0.2000	0.9333	56	12
0.3595790062646877	0.2167	0.9333	56	13
0.3346481862025413	0.2333	0.9333	56	14
0.32743436342794185	0.2333	0.9500	57	14
0.29508749366410275	0.2500	0.9500	57	15
0.2840498629761975	0.2667	0.9500	57	16
0.24932090731712234	0.2833	0.9500	57	17
0.2345862178011957	0.3000	0.9500	57	18
0.20829657210398672	0.3167	0.9500	57	19
0.19819617862680639	0.3333	0.9500	57	20
0.19132071555328292	0.3500	0.9500	57	21
0.18137458155262107	0.3500	0.9667	58	21
0.16417363129683363	0.3667	0.9667	58	22
0.1530997854238631	0.3667	0.9833	59	22
0.13149265641520635	0.3833	0.9833	59	23
0.11555176688594955	0.4000	0.9833	59	24
0.11068296688258278	0.4167	0.9833	59	25
0.10587682212019951	0.4333	0.9833	59	26
0.09194744279194193	0.4500	0.9833	59	27
0.09009697978979764	0.4667	0.9833	59	28
0.08018191043033328	0.4833	0.9833	59	29
0.06790861155343411	0.5000	0.9833	59	30
0.061966456163520235	0.5167	0.9833	59	31
0.058711668180110047	0.5333	0.9833	59	32
0.053198862947682536	0.5500	0.9833	59	33
0.0463643240438053	0.5667	0.9833	59	34
0.046244322607210946	0.5667	1.0000	60	34
0.0443454408235394	0.5833	1.0000	60	35
0.04426803515815382	0.6000	1.0000	60	36
0.04418246528452364	0.6167	1.0000	60	37
0.04302683250785025	0.6333	1.0000	60	38
0.04167261393566796	0.6500	1.0000	60	39
0.040683873984391446	0.6667	1.0000	60	40
0.03860676777239255	0.6833	1.0000	60	41
0.02866491307273212	0.7000	1.0000	60	42
0.024166577560929815	0.7167	1.0000	60	43
0.019604627265827228	0.7333	1.0000	60	44
0.018055353376382196	0.7500	1.0000	60	45
0.01668493891327012	0.7667	1.0000	60	46
0.014590109896884437	0.7833	1.0000	60	47
0.014445427130075865	0.8000	1.0000	60	48
0.013717755129914806	0.8167	1.0000	60	49
0.013691158685263771	0.8333	1.0000	60	50
0.013645036296893306	0.8500	1.0000	60	51
0.013440583986278292	0.8667	1.0000	60	52
0.01269390633473383	0.8833	1.0000	60	53
0.011230252426244768	0.9000	1.0000	60	54
0.007421712558881444	0.9167	1.0000	60	55
0.00709368230880905	0.9333	1.0000	60	56
0.005430773014975837	0.9500	1.0000	60	57
0.004862917103091628	0.9667	1.0000	60	58
0.002805774960892993	0.9833	1.0000	60	59
0.0025351142101912596	1.0000	1.0000	60	60

Precision–recall points

Threshold	Recall	Precision	TP	FP
Above 1 (all negative)	0.0000	1.0000	0	0
0.9970893778022378	0.0167	1.0000	1	0
0.9968987778837627	0.0333	1.0000	2	0
0.9962179163948189	0.0500	1.0000	3	0
0.9952448098315636	0.0667	1.0000	4	0
0.9946046374483011	0.0833	1.0000	5	0
0.9935855738285388	0.1000	1.0000	6	0
0.9926493786858706	0.1167	1.0000	7	0
0.9899993309972164	0.1333	1.0000	8	0
0.9892875157233764	0.1500	1.0000	9	0
0.9872181734520095	0.1667	1.0000	10	0
0.9828753223462144	0.1833	1.0000	11	0
0.9806849675726343	0.2000	1.0000	12	0
0.9805658105705933	0.2167	1.0000	13	0
0.9801247572254904	0.2333	1.0000	14	0
0.9796173347037235	0.2500	1.0000	15	0
0.9788308759855273	0.2667	1.0000	16	0
0.9772559921986599	0.2833	1.0000	17	0
0.9752522184841321	0.3000	1.0000	18	0
0.9725744737406108	0.3167	1.0000	19	0
0.972317532926128	0.3333	1.0000	20	0
0.971220425571667	0.3500	1.0000	21	0
0.9704162110929353	0.3667	1.0000	22	0
0.9703468275226623	0.3833	1.0000	23	0
0.9689402085630805	0.4000	1.0000	24	0
0.9683246734716652	0.4167	1.0000	25	0
0.9657512545244449	0.4333	1.0000	26	0
0.960130694920936	0.4500	1.0000	27	0
0.9589463880796237	0.4667	1.0000	28	0
0.9532317568908516	0.4833	1.0000	29	0
0.9529388596101162	0.5000	1.0000	30	0
0.9473141104533185	0.5167	1.0000	31	0
0.9449634911251589	0.5333	1.0000	32	0
0.9422924490501764	0.5500	1.0000	33	0
0.9406764773509962	0.5667	1.0000	34	0
0.9401545634993372	0.5833	1.0000	35	0
0.9364347838624589	0.6000	1.0000	36	0
0.9269762316305228	0.6167	1.0000	37	0
0.9266656966158918	0.6167	0.9737	37	1
0.9142282154668511	0.6333	0.9744	38	1
0.9024497679543143	0.6500	0.9750	39	1
0.8846596485921231	0.6500	0.9512	39	2
0.8827160392568102	0.6667	0.9524	40	2
0.8550943219363668	0.6833	0.9535	41	2
0.7975495838365965	0.6833	0.9318	41	3
0.7875335779427142	0.7000	0.9333	42	3
0.7425966105705569	0.7000	0.9130	42	4
0.7396711594927651	0.7167	0.9149	43	4
0.7306899435243107	0.7333	0.9167	44	4
0.7133375411360531	0.7500	0.9184	45	4
0.70516394833072	0.7500	0.9000	45	5
0.6892130108514639	0.7667	0.9020	46	5
0.6764060119523394	0.7667	0.8846	46	6
0.6371414184795541	0.7667	0.8679	46	7
0.6303997388439126	0.7833	0.8704	47	7
0.6206797214949554	0.8000	0.8727	48	7
0.6020427185773436	0.8167	0.8750	49	7
0.5925107128652473	0.8333	0.8772	50	7
0.5893615498211943	0.8333	0.8621	50	8
0.5876629017933523	0.8500	0.8644	51	8
0.5868653685118453	0.8667	0.8667	52	8
0.499948274022567	0.8667	0.8525	52	9
0.454047889362915	0.8667	0.8387	52	10
0.4488733512793822	0.8667	0.8254	52	11
0.427641063910166	0.8667	0.8125	52	12
0.38551063221977444	0.8833	0.8154	53	12
0.36919318339846635	0.9000	0.8182	54	12
0.3639608427090342	0.9167	0.8209	55	12
0.36366044873052633	0.9333	0.8235	56	12
0.3595790062646877	0.9333	0.8116	56	13
0.3346481862025413	0.9333	0.8000	56	14
0.32743436342794185	0.9500	0.8028	57	14
0.29508749366410275	0.9500	0.7917	57	15
0.2840498629761975	0.9500	0.7808	57	16
0.24932090731712234	0.9500	0.7703	57	17
0.2345862178011957	0.9500	0.7600	57	18
0.20829657210398672	0.9500	0.7500	57	19
0.19819617862680639	0.9500	0.7403	57	20
0.19132071555328292	0.9500	0.7308	57	21
0.18137458155262107	0.9667	0.7342	58	21
0.16417363129683363	0.9667	0.7250	58	22
0.1530997854238631	0.9833	0.7284	59	22
0.13149265641520635	0.9833	0.7195	59	23
0.11555176688594955	0.9833	0.7108	59	24
0.11068296688258278	0.9833	0.7024	59	25
0.10587682212019951	0.9833	0.6941	59	26
0.09194744279194193	0.9833	0.6860	59	27
0.09009697978979764	0.9833	0.6782	59	28
0.08018191043033328	0.9833	0.6705	59	29
0.06790861155343411	0.9833	0.6629	59	30
0.061966456163520235	0.9833	0.6556	59	31
0.058711668180110047	0.9833	0.6484	59	32
0.053198862947682536	0.9833	0.6413	59	33
0.0463643240438053	0.9833	0.6344	59	34
0.046244322607210946	1.0000	0.6383	60	34
0.0443454408235394	1.0000	0.6316	60	35
0.04426803515815382	1.0000	0.6250	60	36
0.04418246528452364	1.0000	0.6186	60	37
0.04302683250785025	1.0000	0.6122	60	38
0.04167261393566796	1.0000	0.6061	60	39
0.040683873984391446	1.0000	0.6000	60	40
0.03860676777239255	1.0000	0.5941	60	41
0.02866491307273212	1.0000	0.5882	60	42
0.024166577560929815	1.0000	0.5825	60	43
0.019604627265827228	1.0000	0.5769	60	44
0.018055353376382196	1.0000	0.5714	60	45
0.01668493891327012	1.0000	0.5660	60	46
0.014590109896884437	1.0000	0.5607	60	47
0.014445427130075865	1.0000	0.5556	60	48
0.013717755129914806	1.0000	0.5505	60	49
0.013691158685263771	1.0000	0.5455	60	50
0.013645036296893306	1.0000	0.5405	60	51
0.013440583986278292	1.0000	0.5357	60	52
0.01269390633473383	1.0000	0.5310	60	53
0.011230252426244768	1.0000	0.5263	60	54
0.007421712558881444	1.0000	0.5217	60	55
0.00709368230880905	1.0000	0.5172	60	56
0.005430773014975837	1.0000	0.5128	60	57
0.004862917103091628	1.0000	0.5085	60	58
0.002805774960892993	1.0000	0.5042	60	59
0.0025351142101912596	1.0000	0.5000	60	60

Work through one threshold

Suppose targets are [0, 1, 0, 1] and outputs are [0.1, 0.4, 0.4, 0.9]. At τ = 0.5: TP = 1, FP = 0, FN = 1, TN = 2. If a missed positive costs 3 and a false alarm costs 1, mean cost is (0 + 3) / 4 = 0.75.

At τ = 0.4, both tied 0.4 scores enter class 1: TP = 2, FP = 1, FN = 0, TN = 1. Mean cost falls to 1 / 4 = 0.25. Moving a threshold changes decisions; probabilities and weights stay fixed.

There are four positive–negative score pairs: three wins and one tie, so ROC AUC = (3 + ½) / 4 = 7/8. Recall rises by ½ at precision 1, then by ½ at precision 2/3, giving AP = ½ + ⅓ = 5/6.

Brier = (0.1² + 0.6² + 0.4² + 0.1²) / 4 = 0.135. With two bins [0, 0.5) and [0.5, 1], ECE = (3/4) × |0.3 − 1/3| + (1/4) × |0.9 − 1| = 0.05. The recorded report uses ten bins; ECE changes with the binning choice.

Why use validation for the threshold?

The threshold is another selected setting. Choosing it to minimize test mistakes uses the test labels for model selection. This report chooses among the validation scores, 0.5 and an all-negative rule. Ties prefer the threshold closest to 0.5, then the higher threshold.

How to read calibration

In a bin with mean output 0.7 and an observed positive fraction of 0.5, the absolute calibration gap is 0.2. ECE averages these gaps weighted by bin count. Brier is the mean of (p − y)². Neither measure makes a small synthetic test into a guarantee about future data.

05 / GRADIENT LAB

Take a step. Understand why it works.

Temporary sandbox · 0 steps

Choose a weight or bias. Compare its exact backpropagation derivative with a numerical estimate, then see a proposed descent step before applying it. These calculations use only a small, fixed batch of training examples.

This sandbox changes a temporary copy in this tab. The recorded model, replay, validation-selected threshold and held-out results stay fixed. Reloading discards sandbox steps. This is plain gradient descent, even when the original run used Adam.

Starting model Training batch L2 coefficient λ

Changing the starting model, batch or λ resets the sandbox. No test or validation rows enter these calculations.

Parameter to inspectSlice radius ±

Actual objective J Tangent J + g·δAll other parameters fixed

Read numerical slice values

Parameter change δ	Actual J	Tangent J + g·δ

Analytic derivative g—

Central difference—

Absolute difference—

Finite-difference ε

Numerical derivative = [J(θ + ε) − J(θ − ε)] / (2ε). Smaller ε is not always more accurate: rounding competes with approximation error.

Inspect the sandbox step history

Step	Mean BCE	L2 penalty	Objective J	Actual change

Each row uses the same batch and λ. This is a training objective, not a new estimate of held-out performance.

Follow the derivative through the network

Standardize each raw input using the saved training mean and scale. For B training rows, the output derivative is dZ₂ = (P − Y) / B. Then dW₂ = HᵀdZ₂ + λW₂, dZ₁ = (dZ₂W₂ᵀ) ⊙ (1 − H²), and dW₁ = X̃ᵀdZ₁ + λW₁. Bias derivatives sum over rows and have no L2 penalty.

The objective is mean BCE plus ½λ(‖W₁‖² + ‖W₂‖²). Computing BCE from logits avoids taking log(0) when sigmoid rounds to 0 or 1.

Why can a descent step increase loss?

The tangent predicts ΔJ ≈ −ηg² for one parameter, or −η‖∇J‖² for all parameters. This prediction is local. Curvature matters for a finite step, so a large learning rate can overshoot.

The chart varies only the selected coordinate; an all-parameter step follows another direction. Neither a flat slice nor a small gradient proves a global minimum. Compare the predicted and actual changes, then use Undo to try a smaller η.

Work one update by hand

Take one standardized input [0, 0] with target 1, one hidden neuron, W₁ = [1, −1]ᵀ, b₁ = 0, W₂ = 2, b₂ = 0 and λ = 0.1. Then h = 0, p = 0.5, BCE = log(2), penalty = 0.3 and J ≈ 0.993147.

For b₂, the derivative is p − y = −0.5. A coordinate step with η = 0.2 makes b₂ = 0.1. The predicted change is −0.05. The actual objective becomes log(1 + exp(−0.1)) + 0.3 ≈ 0.944397: a change of −0.048751.

More derivations and exercises: Gradients & optimization and Optimization · exercises.

06 / CONNECT CODE TO MATHEMATICS

The whole model, in four lines.

01X̃ = (X − μ_train) / σ_train

02H = tanh(X̃W₁ + b₁)

03P = sigmoid(HW₂ + b₂)

04J = mean BCE + ½λ(‖W₁‖² + ‖W₂‖²)

What makes this a learning experiment?

Mini-batches estimate a gradient from a subset of training rows. Adam tracks first and second gradient moments and corrects their initial bias. L2 penalizes large weights. Early stopping chooses when to stop using validation behavior.

Python training uses explicit NumPy derivatives and optimizer updates. The gradient sandbox mirrors the network calculations in JavaScript. Neither uses an automatic differentiation framework.

Why divide by the batch size?

The data loss is a mean. Its derivative begins with dZ₂ = (P − Y) / B, where B is the current batch size. The last, shorter batch uses its own size. The L2 derivative is λW; biases receive no L2 penalty.

Why not fit the scaler on every point?

Using validation or test values to calculate the mean and standard deviation lets evaluation data influence training. This lab fits μ and σ on training inputs only, then applies those same numbers everywhere.

Why compare with a linear model?

A network can fit curved boundaries, but a more complex model is not automatically better. The baseline makes the added hidden layer's effect measurable for this experiment.

07 / YOUR NEXT EXPERIMENT

Change one idea. Explain the result.

Try SGD, a different hidden width, or stronger regularization. Choose settings using validation results. Treat the test set as the final audit, rather than a scoreboard to optimize.

python -m neural_math_lab experiment --dataset moons --samples 600 --noise 0.15 --hidden 16 --seed 7 --epochs 1200 --learning-rate 0.01 --optimizer adam --batch-size 32 --l2 0.001 --patience 100 --min-delta 1e-05 --fp-cost 1.0 --fn-cost 1.0 --output runs/new-experiment

The command runs locally in Python. Choose a new output directory each time.

Read alongside the experiment

Gradients & optimization
Gradients, loss slices, and one update worked by hand.

Optimization · exercises
Eight exercises with worked answers.

Training beyond XOR
Derivations, assumptions, and a worked Adam update.

Decisions, ranking & calibration
Decision costs, ranking, calibration, and raw-input derivatives.

Decisions · exercises
Worked threshold, ranking, calibration, and sensitivity exercises.

Training · exercises
Experiments and answers you can check yourself.

Configuration & provenance

{
  "config": {
    "dataset": "moons",
    "samples": 600,
    "noise": 0.15,
    "hidden": 16,
    "seed": 7,
    "epochs": 1200,
    "learning_rate": 0.01,
    "optimizer": "adam",
    "batch_size": 32,
    "l2": 0.001,
    "patience": 100,
    "min_delta": 1e-05,
    "fp_cost": 1.0,
    "fn_cost": 1.0
  },
  "seeds": {
    "data": 7,
    "split": 8,
    "initialization": 9,
    "network_batches": 10,
    "baseline_batches": 11
  },
  "environment": {
    "python": "3.9.6",
    "numpy": "2.0.2",
    "matplotlib": "3.9.4"
  }
}

See a networklearn the shape.

A boundary built from simple operations.

Training improves. Validation keeps it honest.

Read this curve.

More than one accuracy number.

Same held-out examples. Two models.

Neural network confusion matrix

A probability becomes a decision.

Ranking · ROC

Retrieval · precision–recall

Probability · reliability

Evaluate the selected rules once.

Neural network · saved validation diagnostics

Reliability bins

Linear baseline · saved validation diagnostics

Reliability bins

Take a step. Understand why it works.

The whole model, in four lines.

What makes this a learning experiment?

Change one idea. Explain the result.

See a network
learn the shape.