Introduction: It has been recommended that physical activity be a part of treatment and management regimens of type 2 diabetes mellitus (T2DM), and research has shown that regular physical exercise facilitates glycemic control in these patients. In this analysis, our aim was to systematically show the therapeutic effects of mild to moderate intensity aerobic exercise on glycemic control in patients with T2DM. Methods: From February to April 2021, we searched the https://www.clinicaltrials.gov, EMBASE, MEDLINE, Cochrane Central Register of Controlled Trials (CENTRAL), Web of Science and Google Scholar databases for trials that showed the effects of aerobic exercise on glycemic control in patients with T2DM. Glycated hemoglobin (HbA1c) was the endpoint in the analysis. The RevMan version 5.4 statistical program was used for statistical analysis, and the mean difference (MD) and 95% confidence intervals (CI) used to represent the data following analysis. Results: Eighteen trials involving 972 participants with T2DM were included in this meta-analysis, of whom 523 were assigned to an exercise group and 449 were assigned to a control group. A comparison pre- versus post-aerobic exercise showed that aerobic exercise significantly improved glycemic control (HbA1c) (MD 0.35, 95% CI 0.23–0.48; P = 0.00001) in these patients with T2DM. A second comparison, T2DM participants in the experimental group post-exercise versus T2DM participants from the control group at the end of the follow-up, also showed that aerobic exercise significantly improved glycemic control (MD − 0.46, 95% CI − 0.69 to − 0.22; P = 0.0001). However, a comparison of HbA1c of T2DM participants in the control group at the beginning of the study compared to those at the end of follow-up did not show any significant improvement in glycemic control (MD 0.08, 95% CI − 0.05 to 0.21; P = 0.21). Conclusion: The current analysis showed that mild to moderate intensity aerobic exercise significantly improved glycemic control in patients with T2DM. Patients with T2DM who regularly participated in aerobic exercise activities had a better control of their disease than those who were not on a regular aerobic exercise regimen. These results lead to the recommendation that at least mild to moderate intensity aerobic exercise should be included in the treatment and management regimens of patients with T2DM.