ngzip — LLM Adaptive Pruning & MLP Compression (Llama-3.2-1B)

Validation Perplexity (ΔPPL)

10.0940 (-0.1249)

Quality improved vs. unpruned baseline (10.2189)

Layer-0 MLP Sparsity (Step 31)

19.11% (Up: 19.72%)

5.26M coordinates zeroed with zero PPL loss

Peak Single-Step Pruning Yield

136,691 axes

Step 48 record on Llama-3.2-1B (ΔPPL -0.52)

Emax=0.5 Extreme Compression

92.18% Sparse

Avg K_gate reduced from 2048 → 160.1 channels

LECAP Sparsity Growth vs. Full Validation Perplexity (FineWeb-Edu)

8,960 Gate + 8,960 Up Neurons · 2-Pass PAFT L1 Regularization

Total Sparsity (%) Up-Channel Sparsity (%) Validation Perplexity (Full Eval) Baseline Unpruned PPL (10.2189)

penalized_rank = raw_rank + lock_count × 5.0

Lock Level	Active Gate Axes	Processed (*)	Promoted (^)	Zeroed (x)
Level 0 (Free)	11,126,472	63,219	15,746	47,473
Level 1 (+5 rank)	21,223	11,646	11,646	0
Level 2 (+10 rank)	20,997	7,192	7,190	2
Level 3 (+15 rank)	29,771	4,478	4,451	27
Levels 4–8 (Core)	16,771	3,065	3,065	0

Total MLP Neurons Compiled

12,288 (100%)

Full-network per-neuron ΔPPL sensitivity sweep

High-Savings Prosthesis Share

81.52% (10,017)

Replaced with zero/negative ΔPPL degradation

Zero-Math Decision Trees (D=3)

61.52% (7,559)

512-dim dot-product replaced by 3 comparisons

Perplexity Improved Neurons

89.16% (10,956)

Prosthesis acts as structural regularizer (ΔPPL < 0)

8,192 Gate & Up Neurons · Avg Active Input Channels K (2048 → 160.1)

Avg Active Gate Channels K_gate (2048 → 160.1) Iteration Step Time (0.31s → 0.05s, 6.2× speedup)

Decision Tree (D=3) vs. Top-K Pruning (K=5) vs. Spline-Full

Prosthesis Method	Assigned Neurons	Share (%)	Compute Savings
Decision Tree (Depth 3)	7,559	61.52%	~100% (Zero-MUL)
Sparse Top-K Pruning (K=5)	2,458	20.00%	99.02% MAC reduction
Spline-Full (Core Sensitive)	2,271	18.48%	Preserves exact PPL
Total High-Savings	10,017 / 12,288	81.52%	ΔPPL ≤ 0.0000