Background: The inconstant effects and consequences of many pandemics, as well as the variety of public health emergencies of international concerns that occurred throughout history, affect all aspects of the population, especially the most vulnerable groups like the elderly. In this study, we highlighted the psychological effects in this special population and health care impacts during the COVID-19 pandemic and its preventive measures. Subjects and Methods: Through a cross-sectional study design, an interview-based questionnaire of a translated and culturally modified version of the Questionnaire for Assessing the Impact of COVID-19 Pandemic and Accompanying Mitigation Efforts on Older Adults (QAICPOA) was used on a sample of 394 participants who were recruited using a multistage stratified random sampling technique of older adults registered at Primary Health Care Centers (PHCC) in Jeddah, Kingdom of Saudi Arabia, who attended between April and July 2021. Since Clinical diseases of the elderly population vary by their age, we classified them based on their age to three main groups, the youngest-old, the middle-aged old, and the oldest old, to investigate the effect of the COVID-19 pandemic on their psychosocial health at various age groups to better understand their challenges and needs. Results: The mean age of participants was 74 years (SD ± 9.27), and 54% were males. Approximately two-thirds of the participants were concerned about the COVID-19 pandemic. Hence, 83% avoided public places and crowds, 66% ceased from visiting family and friends; ultimately 25% isolated themselves from family members of the same household. The majority, 75.4% (n=297) had extreme difficulties or were unable to obtain regular medical care and medication, in addition to facing other challenges related to communication, and life necessities. In response, some were resilient and even volunteered to help. Conclusion and recommendations: The majority of the study participants experienced social and psychological difficulties related to the COVID-19 pandemic; therefore, new comprehensive and sustainable preventative measures and health services are needed in the future to ensure healthy aging of this vulnerable group during emergencies.