Brains come in many shapes and sizes. Nature has endowed big-brained primate species like humans with a proportionally large cerebral cortex. Comparative studies have suggested, however, that the total volume allocated to white matter connectivity—the brain’s infrastructure for long-range interregional communication—does not keep pace with the cortex. We investigated the consequences of this allometric scaling on brain connectivity and network organization. We collated structural and diffusion magnetic resonance imaging data across 14 primate species, describing a comprehensive 350-fold range in brain size across species. We show volumetric scaling relationships that indeed point toward a restriction of macroscale connectivity in bigger brains. We report cortical surface area to outpace white matter volume, with larger brains showing lower levels of overall connectedness particularly through sparser long-range connectivity. We show that these constraints on white matter connectivity are associated with longer communication paths, higher local network clustering, and higher levels of asymmetry in connectivity patterns between homologous areas across the left and right hemispheres. Our findings reveal conserved scaling relationships of major brain components and show consequences for macroscale brain circuitry, providing insights into the connectome architecture that could be expected in larger brains such as the human brain.