I would like to garner some ideas on how y’all manage updating static, flat file datasets on a regular basis.

Most of ours pull from the source via API. But a few require us every week/month/year to download a flat file from the source to then re-upload to our platform.

At first I thought a simple calendar with what needs doing when would be a good idea. But, some datasets dont drop on a regular schedule or can be delayed. So if I have a calendar entry saying “Download X data today” and its not there. It doesnt get done and I forget that ive not actually done that. So I need some way to check it off.

How do you all efficiently manage your flat file updates? What tools or methods do you use?