Paper · 2025

Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activations

G. DeZoort and B. Hanin · SIAM Journal on Mathematics of Data Science 7, 1 (2025)

Deep graph neural networks often fail before training even starts. This paper identifies three ways that happens at initialization — outputs that explode or vanish with depth, oversmoothing, and correlation collapse — and derives the initialization and residual-architecture choices that avoid each one, keeping GNNs trainable at large depth.

  • A graph-aware version of He initialization for ReLU GNNs keeps the forward and backward passes stable.
  • Oversmoothing is shown to set in exponentially with depth for a fixed message-passing operator, whatever the initialization.
  • Residual aggregation and residual connections control oversmoothing and correlation collapse, and improve time-to-train.
Test accuracy versus depth for vanilla GNNs and GNNs with residual aggregation and connections

Presented at