PC-ALM brings layer-local learning close to backpropagation
Sakana AI researchers Jeffrey Seely and Julian Gould have introduced Augmented Lagrangian Predictive Coding, or PC-ALM, a training method designed as a layer-local alternative to backpropagation. The approach adds a Lagrange multiplier to each layer’s constraint while retaining predictive coding’s local updates. The researchers say these multipliers converge to exact backpropagation adjoints in linear networks under a stability condition.
PC-ALM alternates local activation updates with dual updates that accumulate prediction errors. In the reported experiments, residual MLPs with widths and depths from 8 to 128 were trained for one epoch on MNIST and Fashion-MNIST. With an inference budget of T = 2L, PC-ALM matched backpropagation across the tested widths, depths and identity, tanh and ReLU activations, while standard predictive coding degraded in deep, narrow networks.
A reference configuration on Fashion-MNIST achieved 77.75% test accuracy for PC-ALM, compared with 78.66% for backpropagation and 68.13% for predictive coding. The method also trained 1000-layer residual MLPs on MNIST to within roughly 2 percentage points of backpropagation. An MIT-licensed JAX implementation runs on CPU, but the method has so far been evaluated only on small image benchmarks, including ResNet-18 tests on CIFAR-10 and Tiny ImageNet.
Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.