Governments have enforced measures to limit the spread of COVID-19 with varying degrees of success, which could affect people’s physical activity (PA) and sedentary behavior. This study aimed to examine changes in PA levels, types of PA, and sedentary behavior in the Swedish population before and during the COVID-19 pandemic. Associations between changed PA levels and demographical and behavioral determinants were also investigated. In December 2020, 1035 individuals (18–79 years old) completed a survey about their PA and sedentary behavior before and during the pandemic. Factors influencing their PA were also explored. Fifty-one percent of the sample reported reduced total PA, 18% had no change, and 31% increased their PA. Overall, organized PA decreased the most and sedentary behavior increased. The youngest and oldest age groups reported the greatest reduction in PA, while middle-aged groups reported the most increased PA. Men reported a larger increase in sedentary behavior than women. Mental and physical capability was associated with change in PA. In conclusion, this study indicates that, during the COVID-19 pandemic, the majority of the Swedish population have decreased PA levels with a concurrent increase in sedentary behavior, which may have negative health consequences. Interventions are recommended to address both PA and sedentary behavior, specifically to strengthen people’s ability to perform PA and focusing on the youngest and oldest age groups.