Nonlinear Decision Boundaries with a Two-Layer Neural Network

Implementation of a regularized 2-layer multiclass classifier on nonlinear 2D benchmarks (flower, spiral) in NumPy (explicit backprop) and PyTorch (module-based baseline).

A single-hidden-layer network:

with activation $\phi \in {\mathrm{ReLU}, \sigma}$. This is the simplest nonlinear hypothesis class that can represent nonlinearly separable decision boundaries (e.g., spiral), illustrating feature learning via learned hidden representations.

Empirical risk minimization with L2 regularization (weight decay):

Cross-entropy corresponds to the negative log-likelihood under a categorical model $p(y\mid x)$.

Full-batch gradient descent / SGD using analytically derived gradients; demonstrates the chain rule through softmax + cross-entropy (yielding $p-y$) and through the hidden nonlinearity.

Softmax is computed with log-sum-exp shifting to prevent overflow.

Finite-difference gradient checking (using sigmoid to ensure differentiability everywhere) validates the NumPy backprop by comparing:

against a small tolerance.

python -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt

NumPy (Flower):

python scripts/train_flower_numpy.py

NumPy (Spiral):

python scripts/train_spiral_numpy.py

PyTorch (Flower):

python scripts/train_flower_torch.py

PyTorch (Spiral):

python scripts/train_spiral_torch.py

Plots are saved to:

• outputs/figures/flower-boundary.jpg
• outputs/figures/spiral-boundary.jpg

Gradient checking uses a sigmoid hidden activation so the network is differentiable everywhere (ReLU is not differentiable at 0, which makes numerical gradients disagree near 0).

Run:

python scripts/gradient_check.py

The printed output includes the relative difference:

A typical pass condition is diff < 1e-6.