Previous studies have demonstrated ethnic/racial differences in cardiorespiratory fitness (CRF). However, the relative contributions of body mass index (BMI), lifestyle behaviors, socioeconomic status (SES), cardiovascular (CV) risk factors, and cardiac function to these differences in CRF are unclear. In this study, we included 2,617 Dallas Heart Study participants (58.6% women, 48.6% black; 15.7% Hispanic) without CV disease who underwent estimation of CRF using a submaximal exercise test. We constructed multivariable-adjusted linear regression models to determine the association between race/ethnicity and CRF, which was defined as peak oxygen uptake (ml/kg/min). Black participants had the lowest CRF (blacks: 26.3 ± 10.2; whites: 29.0 ± 9.8; Hispanics: 29.1 ± 10.0 ml/kg/min). In multivariate analysis, both black and Hispanic participants had lower CRF after adjustment for age and gender (blacks: Std β = −0.15; p value ≤0.0001, Hispanics: Std β = −0.05, p value = 0.01; ref group: whites). However, this association was considerably attenuated for black (Std β = −0.04, p value = 0.03) and no longer significant for Hispanic ethnicity (p value = 0.56) after additional adjustment for BMI, lifestyle factors, SES, and CV risk factors. Additional adjustment for stroke volume did not substantially change the association between black race/ethnicity and CRF (Std β = −0.06, p value = 0.01). In conclusion, BMI, lifestyle, SES, and traditional risk factor burden are important determinants of ethnicity-based differences in CRF.