Objective: To examine the impact of physical activity on cartilage thickness loss in knee osteoarthritis (OA). Methods: A total of 689 participants with radiographic knee OA at baseline (Kellgren/Lawrence grade ≥2) from the Osteoarthritis Initiative completed the Physical Activity Scale for the Elderly (PASE) questionnaires at annual intervals over 4 years. Magnetic resonance imaging–based cartilage thickness change in the medial femorotibial compartment (MFTC) over 4 years was the main outcome. The impact of PASE tertiles (low, moderate, or high) on changes in MFTC cartilage thickness was estimated using a mixed-effects model adjusted for baseline characteristics. Furthermore, stratification by sex was performed for secondary analyses. Results: Structural progression of MFTC cartilage loss of −0.20 mm (95% confidence interval [95% CI] −0.22, −0.17) was observed in the entire cohort, with no significant difference between physical activity levels after adjustment for baseline characteristics. An interaction between sex and physical activity was observed in the adjusted analysis (P = 0.02). Stratification by sex showed that women with low physical activity had a statistically greater cartilage loss than women with moderate physical activity (adjusted between-group difference –0.09 mm [95% CI –0.16, 0.02]), whereas no significant differences were observed in men. Conclusion: While physical activity was not associated with cartilage thickness loss in the whole cohort, this relationship significantly differed between sexes. In women, but not in men, moderate physical activity may slow down structural disease progression compared to low physical activity levels. For both men and women, high physical activity levels do not appear to be more detrimental than lower physical activity levels for cartilage thickness loss.