Physical activity (PA) and sedentary behavior (SB) changed during the COVID-19 pandemic; hence, this study examined PA and SB at four time points between December 2019 and December 2022. The participants’ PA decreased during the pandemic and did not recover afterwards. Among women, PA increased slightly in 2022 but not at all in men. From 2019 to 2020, SB increased and then decreased to near the pre-pandemic level in both sexes. Regarding age, PA decreased in the oldest age group (65–79 years) across all time points, while SB increased in all age groups during 2019–2020 and then returned close to pre-pandemic levels among the two middle age groups (30–64 years), but not among the youngest and oldest groups. Considering occupation, PA decreased from 2020 to December 2022 among retired and “other” participants, while SB decreased among nonmanual workers and retired participants. The regression models associated better self-reported health, male sex, and those born overseas with higher PA. Higher age, better self-reported health, poor education, and later survey time points were associated with lower SB. These findings highlight the need to return PA and SB to at least pre-pandemic levels and that subgroups may need different interventions.