Parameter symmetries determine representational geometry in overparameterized nonlinear networks

We analytically show that parameter symmetries can dissociate function and representation in nonlinear networks. At the same time, normative minimum-norm implementation rules can select unique representational geometries that meaningfully link representation to computation.

Representations are routinely used in machine learning, psychology, and neuroscience to probe the computations of biological and artificial systems. Yet it remains unclear to what extent computation constrains representation in artificial neural networks. One key obstacle is that these networks admit parameter symmetries: transformations of the parameters that preserve function exactly while reshaping representational geometry. Here we show that known parameter symmetries act on representations through just three primitives: addition, duplication, and scaling. This yields a closed-form descriptor of representational geometry as a sum of task-linked features and symmetry-induced noise. This decomposition further provides analytic bounds on representational similarity under parameter symmetries, revealing when functionally equivalent networks can become arbitrarily dissimilar. Finally, we identify privileged representational geometries, which weight features by their computational importance and recover a stable link between representation and computation. Overall, our results delineate when representation can support inferences about computation, and when it cannot.