Aging is associated with a decrease in body and brain function and with a decline in insulin-like growth factor 1 levels. The observed associations between alterations in insulin-like growth factor 1 levels and cognitive functioning and Mild Cognitive Impairment suggest that altered insulin-like growth factor 1 signaling may accompany Alzheimer’s disease or is involved in the pathogenesis of the disease. Recent animal research has suggested a possible association between insulin-like growth factor 1 levels and the Apolipoprotein E ε4 allele, a genetic predisposition to Alzheimer’s disease. It is therefore hypothesized that a reduction in insulin-like growth factor 1 signaling may moderate the vulnerability to Alzheimer’s disease of human Apolipoprotein E ε4 carriers. We address the impact of age-related decline of insulin-like growth factor 1 levels on physical and brain function in healthy aging and Alzheimer’s disease and discuss the links between insulin-like growth factor 1 and the Apolipoprotein E ε4 polymorphism. Furthermore, we discuss lifestyle interventions that may increase insulin-like growth factor 1 serum levels, including physical activity and adherence to a protein rich diet and the possible benefits to the physical fitness and cognitive functioning of the aging population.