Background/objective Alzheimer’s disease (AD) is mainly characterized by decline of cognitive functions such as memory and learning, which has a high prevalence and poor drug efficacy in treatment regimes. A systematic review and meta-analysis of randomized controlled trials (RCTs) were conducted to evaluate the effectiveness of exercise on cognitive function in patients diagnosed with AD. Methods The bibliographic databases (PubMed, Cochrane Library and Embase, and Web of Science) and four Chinese databases (Wanfang data, CBM, CNKI, and VIP) were searched to identify RCTs published in any language between January 1, 1960, and January 1, 2018. Only peer-reviewed articles and RCTs were included. The collected data were analyzed by Review Manager (5.3). Results Overall, 869 patients diagnosed with AD were included from 13 RCTs. Patients in the intervention group received pure exercise interventions and a cognitive test. Although there was heterogeneity in intervention methods and cognitive measures among studies, meta-analysis (seven studies) supports positive effects of physical activity on cognitive function of patients with AD (mean difference [MD] =2.53, the 95% CI=0.84 to 4.22, test for overall effect: Z=2.93 [P=0.003]). Eight studies demonstrated that exercise improves cognitive function for individuals with AD. However, the remaining five studies did not display a beneficial effect of exercise on cognitive function in patients with AD. Conclusion This meta-analysis and systematic review indicated that exercise intervention might improve the cognitive function of AD or slow down the decline of cognition; however, this relationship was not always true across studies. RCTs with clear intervention criteria, large samples, and long-term follow-up are needed in the future to demonstrate the benefits of exercise for cognitive function in AD patients.