The calendar panel has to fetch the data from your calendar as you scroll*, depending on how many calendar events you have per day, and how many reminders, this can mean that if you have either a lot of changes occurring in your calendars, or if you have very few calendar events in general, or scroll for the first time further in the future or past, it might be refreshing a lot. Once fetched it should scroll smoothly.
*) Unfortunately Apple’s APIs really weren’t designed to make the kind of view that Agenda shows. It doesn’t allow us to get changes only, and worse only allows to fetch a data range. If you have a lot of events in your calendar one or two days is enough to fill the view, but if you have very few in your calendar or say scroll further into the future, suddenly we sometimes have to fetch the events of an entire month to get the few events that fill the view. It’s technically very challenging.