Cognitive function can be affected by cancer and/or its treatment, and older patients are at a particular risk. In a prospective observational study including patients ≥65 years referred for radiotherapy (RT), we aimed to investigate the association between patient- and cancer-related factors and cognitive function, as evaluated by the Montreal Cognitive Assessment (MoCA), and sought to identify groups with distinct MoCA trajectories. The MoCA was performed at baseline (T0), RT completion (T1), and 8 (T2) and 16 (T3) weeks later, with scores ranging between 0 and 30 and higher scores indicating better function. Linear regression and growth mixture models were estimated to assess associations and to identify groups with distinct MoCA trajectories, respectively. Among 298 patients with a mean age of 73.6 years (SD 6.3), the baseline mean MoCA score was 24.0 (SD 3.7). Compared to Norwegian norm data, 37.9% had cognitive impairment. Compromised cognition was independently associated with older age, lower education, and physical impairments. Four groups with distinct trajectories were identified: the very poor (6.4%), poor (8.1%), fair (37.9%), and good (47.7%) groups. The MoCA trajectories were mainly stable. We conclude that cognitive impairment was frequent but, for most patients, was not affected by RT. For older patients with cancer, and in particular for those with physical impairments, we recommend an assessment of cognitive function.