Background This systematic and meta-analytic review aimed to investigate the effects of physical exercise on the working memory of older adults, and to identify the moderators of these effects. Methods We searched six electronic databases for randomized controlled trials on the effects of physical exercise on working memory that were published before or on May 15, 2020. The PEDro scale was used to evaluate the methodological quality of the included studies. Stata 14.0 software was used to perform the meta-analysis, subgroup analysis, and publication bias testing. Results A total of 28 studies and 2156 participants were included. The methodological quality of the included studies was fair to excellent, and there was no publication bias. Overall, we found that physical exercise had a significant effect on working memory in older adults (standardized mean difference = 0.30, p < 0.0001). The effects of physical exercise on working memory were moderated by exercise frequency, intensity, type, duration, cognitive status, and control subgroup (active/passive), but not by intervention period or age of participant. Conclusion Physical exercise can effectively improve the working memory of older adults. The recommended physical exercise is multi-component exercise or mind-body exercise of moderate intensity for 45-60 min 3 times a week, for more than 6 months.