Introduction: Subjective memory complaints (SMC) are often considered a clinical marker of mild cognitive impairment and dementia and could manifest as shortening of activity bouts throughout the day causing daily activity to accumulate in a more fragmented pattern. In the current study we explored the association between activity fragmentation and SMC in middle-aged and older adults. Methods: We used data from 3820 participants in the National Health and Nutrition Examination Survey (2003–2006) with valid accelerometer SMC data. The analyses were conducted in 2020. To calculate activity fragmentation, an active-to-sedentary transition probability was calculated as the number of physical activity bouts (i.e., consecutive minutes registering 100+ counts/min) divided by the total sum of minutes spent in physical activity. A multivariable ordinal logistic regression model was conducted to examine the association between activity fragmentation (scaled) and SMC (yes/no). Results: Higher activity fragmentation was associated with an increased likelihood of self-reported SMC in the study population (Odd Ratio [95% Confidence Interval] = 1.335 [1.067, 1.669]; Average Marginal Effect [95%CI] = 0.029 [0.006, 0.052]; p-value = 0.021). This association was independent of total physical activity volume. Conclusions: The findings provide support that studying fragmented activity patterns can be useful in identifying those at risk for SMC, over and above total volume of physical activity. Future longitudinal studies are required to establish causality and the temporal order of the observed association. Nevertheless, activity fragmentation in middle-aged and older adults may reflect pre-clinical signs of future neurodegenerative processes indicating potential targets for modification through intervention.