Purpose Diabetic patients tend to have a poor quality of life. A sedentary lifestyle is considered to be a modifiable risk factor for type 2 diabetes and an independent predictor of poor quality of life. Exercise is a key treatment for people living with diabetes. The purpose of this study was to conduct a systematic review to assess the effect of exercise on the quality of life of people with type 2 diabetes. Methods We conducted a systematic review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. PubMed, Web of Science, Embase, Cochrane Library, CINAHL and three Chinese databases were searched for studies published until January 2016. The review included all clinical trials that evaluated the effect of exercise on quality of life compared with that of usual care for people with type 2 diabetes. Two reviewers independently assessed the quality of all the included studies, by using the Downs and Black Quality Index (QI). Results Thirty studies met inclusion criteria, with 2785 participants. We divided the exercise into four modes: aerobic, resistance, a combination of aerobic and resistance and yoga. Aerobic exercise showed a significant effect between groups. Resistance and combined exercise showed mixed results. Yoga also showed good intervention effects on quality of life. Conclusions The effect of aerobic exercise on the quality of life in people with type 2 diabetes was safe and effective. Then, most of the studies on aerobic exercise were of good methodological quality. The effects of resistance exercise and combined exercise on the quality of life in people with type 2 diabetes were mixed, and the effect of yoga on quality of life still need more research.