Methods: We analyzed multi-shell diffusion tensor images from 103 healthy young and older adults using tract-based spatial statistics (TBSS) to examine relationships between white matter microstructure and RA rhythm stability. RA measures were computed using both cosinor and non-parametric methods derived from 7 days of actigraphy data. Fractional anisotropy (FA) and mean diffusivity (MD) were examined in this analysis. Because prior studies have suggested that the corpus callosum (CC) is sensitive to sleep physiology and RA rhythms, we also conducted a focused region of interest analysis on the CC. Results: Greater rest-activity rhythm stability was associated with greater FA across both young and older adults, primarily in the CC and anterior corona radiata. This effect was not moderated by age group. While RA measures were associated with sleep metrics, RA rhythm measures uniquely accounted for the variance in white matter integrity. Conclusions: This study strengthens existing evidence for a relationship between brain white matter structure and RA rhythm stability in the absence of health risk factors. While there are differences in RA stability between age groups, the relationship with brain white matter was present across both young and older adults. RA rhythms may be a useful biomarker of brain health across both periods of adult development.