Although trials have shown that exercise has positive effects on bone mineral density (BMD), the majority of exercise trials have been conducted in older women. The aim of this study was to systematically review trials examining the effect of weight-bearing and resistance-based exercise modalities on the BMD of hip and lumbar spine of middle-aged and older men. Eight electronic databases were searched in August 2012. Randomised controlled or controlled trials that assessed the effect of weight-bearing and resistance-based exercise interventions on BMD measured by dual-energy x-ray absorptiometry, and reported effects in middle-aged and older men were included. Eight trials detailed in nine papers were included. The interventions included walking (n = 2), resistance training (n = 3), walking + resistance training (n = 1), resistance training + impact-loading activities (n = 1) and resistance training + Tai Chi (n = 1). Five of the eight trials achieved a score of less than 50% on the modified Delphi quality rating scale. Further, there was heterogeneity in the type, intensity, frequency and duration of the exercise regimens. Effects of exercise varied greatly among studies, with six interventions having a positive effect on BMD and two interventions having no significant effect. It appears that resistance training alone or in combination with impact-loading activities are most osteogenic for this population, whereas the walking trials had limited effect on BMD. Therefore, regular resistance training and impact-loading activities should be considered as a strategy to prevent osteoporosis in middle-aged and older men. High quality randomised controlled trials are needed to establish the optimal exercise prescription.