Cross-sectional studies have shown that pericardial fat is associated with atherosclerotic burden above and beyond generalized and central adiposity. Whether pericardial fat is longitudinally associated with coronary artery calcium (CAC) has not been firmly established. We examined the associations between cardiac ectopic fat including pericardial and intrathoracic fat with CAC progression and incidence in a community-based study setting. Study participants were from the Framingham Heart Study Offspring and Third Generation Cohorts who underwent multidetector computed tomography at 2 consecutive examinations (2002 to 2005 and 2008 to 2011) for the assessment of CAC. Multivariable-adjusted regression models were used to evaluate the associations between cardiac ectopic fat with CAC. Nonlinear associations were also examined. We included 1,732 participants (49.6% women, mean age 49.9 years). Of the 1,024 participants with a CAC score = 0 at baseline, 197 individuals developed a CAC score > 0 (19.2%) during 6.1 years of follow-up. The remaining 708 participants with a CAC score > 0 at baseline were eligible for CAC progression analysis. We identified nonlinear association between pericardial fat and CAC progression. Higher pericardial fat was associated with higher CAC progression only for those participants with pericardial fat higher than the median value (β = 56.0, p = 0.04). Intrathoracic fat was linearly associated with CAC progression (β = 23.0, p = 0.02). However, all of these associations did not persist after additional adjustment for body mass index, abdominal visceral adipose tissue, or waist circumference (all p ≥ 0.14). Neither pericardial nor intrathoracic fat were associated with CAC incidence (all p ≥ 0.33). Overall, both of the cardiac ectopic fat measures were longitudinally associated with CAC progression.