Background: Sedentary behavior has been considered an independent risk factor to health. The aim of this systematic review and meta-analysis was to examine associations between objectively measured sedentary time and physical fitness components in healthy adults. Methods: Four electronic databases (Web of Science, Scopus, Pubmed and Sport Discus) were searched (up to 20 September 2020) to retrieve studies on healthy adults which used observational, cohort and cross-sectional designs. Studies were included if sedentary time was measured objectively and examined associations with the health- or skill-related attributes of physical fitness (e.g., muscular strength, cardiorespiratory fitness, balance). After applying additional search criteria, 21 papers (11,101 participants) were selected from an initial pool of 5192 identified papers. Results: Significant negative associations were found between total sedentary time with cardiorespiratory fitness (r = -0.164, 95%CI: -0.240, -0.086, p < 0.001), muscular strength (r = -0.147, 95%CI: -0.266, -0.024, p = 0.020) and balance (r = -0.133, 95%CI: -0.255, -0.006, p = 0.040). Conclusions: The evidence found suggests that sedentary time can be associated with poor physical fitness in adults (i.e., muscular strength, cardiorespiratory fitness and balance), so strategies should be created to encourage behavioral changes.