Paper · 2026

Hyperparameter Transfer in Graph Neural Networks

G. DeZoort and B. Hanin · To appear in the proceedings of the Learning on Graphs Conference (LoG 2026) · arXiv:2607.05017

Good hyperparameters usually shift as a model grows, which makes large graph neural networks expensive to tune. This paper gives a parameterization under which the learning rate, initialization scale, and weight decay stay put as width and depth increase, so you can tune a small model once and reuse its settings on a much larger one.

  • Covers training with SGD, Adam, and AdamW, with performance improving as models get wider and deeper.
  • For SGD, a graph-dependent correction to the first layer speeds up early training on sparse bag-of-words inputs.
  • For Adam, how message passing is normalized decides whether settings transfer; one extra hyperparameter controls it.
  • For AdamW, learning rate and weight decay transfer together.
QM9 results showing learning-rate transfer across GNN sizes

Presented at