1. I set up a pipeline that downloads a remote file that contains atomic data information. I set the pipeline to check if the hash code of the file has changed, and to update the stored hash code in a text file, in addition to extracting the new data. Atomic Data Pipeline. I also read more of the tardis paper, and a supporting one to understand the general code better.
2. After making this pipeline, I realized that the html link is static to a specific date, and that it would never download a newer date. One possible solution is getting access authentication to the folder storing the files, and downloading the most recent one. It does not seem to matter in this particular case though, because the file has not been updated since 2016. Afterwards, I hope to focus on the Carsus environment, to make sure it is fully stable in accordance with the needs of the people using it. Carsus Repo
3. A problem I faced that helped me understand html code better was my misconception of how the html link works, that it is static and non-changing.