1. I set up a pipeline that downloads a remote file that contains atomic data information. I set the pipeline to check if the hash code of the file has changed, and to update the stored hash code in a text file, in addition to extracting the new data. Atomic Data Pipeline. I also read more of the tardis paper, and a supporting one to understand the general code better. 

2. After making this pipeline, I realized that the html link is static to a specific date, and that it would never download a newer date. One possible solution is getting access authentication to the folder storing the files, and downloading the most recent one. It does not seem to matter in this particular case though, because the file has not been updated since 2016. Afterwards, I hope to focus on the Carsus environment, to make sure it is fully stable in accordance with the needs of the people using it. Carsus Repo

3. A problem I faced that helped me understand html code better was my misconception of how the html link works, that it is static and non-changing. 

This session, we set up azure testing pipelines for "Carsus ... a TARDIS support package for creating and working with atomic datasets" (Carsus Documentation). This repository is intended to complement github.com/tardis-sn/tardis. The pipeline is much more advanced, as it runs 5 parallel tests, a simple basic pytest, one that uses remote data, one that runs slowly, one that fetches database info, and one that adds coverage results: Coverage results. (Hopefully we will get the covered files up to above 80 percent). We use pytest for these tests, and run the commands usings the Carsus setup module's arguments. I learned three new things about azure making this new pipeline. The first is how to efficiently use key value pairs to activate different environments for each test:

Environment 1:

variable = test

Environment 2:

variable = test --remote-data

script:

python setup.py variable

... Running two tests for Environment 1 and 2

 

With updated documentation here: (Most Up To Date Tardis Documentation). I also learned how to use azure's conditioning statements, which includes verifications on environmental variables. This can thus be used to assign script in the pipeline to run for specific environments, and we have to use this, if, condition for the remote data environment. I also figured out how to use (Azure's publishing Task) to generate an appealing coverage report that is readily viewable on the Azure website after the test succeeds. The main issue was generating the compatible xml file. 

I was able to solve the recursion problem I mentioned last week that seemed to be caused by astropy_helpers that is very simliar to this Python recursion error. The problem was that the Carsus repository did not have an updated configuration file (ah_bootsrap.py) for the newer version of astropy_helpers, which caused an infitie loop between the abc and typing modules for sphinx versions above 1.5.6. It seems that there is still a lot to do for the Carsus repository to fully make it python3 compatible. Since I am learning how to make everything python3 compatible, this gives me a chance to learn general overall python coding, instead of specific implementations. 

Since the building documentation pipeline requires sphinx to be above 1.6 in a python 3 enviornment, we will withold on merging the Build documentation pipeline pr, which changes the sphinx version and adds the necessary content to setup a pipeline on azure. 

Learning Highlights:

• Effecient use of key value paris in different parralel jobs
• Azure's built-in conditioning statements
• Publishing coverage results
• Astropy_helpers 
• General Python

1. I setup up testing and build documentation pipelines for a different neighboring repository on my own forked repository: https://github.com/Youssef15015/carsus. This repository is intended to complement github.com/tardis-sn/tardis, as it stores the atomic data information of elements. The pipeline is much more advanced, as it runs 5 parallel tests, a simple basic pytest, one that uses remote data, one that runs slowly, one that fetches database info, and one that adds coverage results: https://dev.azure.com/joey070/joey070/_build/results?buildId=419&view=codecoverage-tab . I learned three new things while making this pipeline: how to effectively use key value pairs to assign different outputs for the same variable for each parralel job and how to use azure's conditioning statements, and how to publish coverage results. 

2. After this, the goal is to focus on pipelines that run on self hosted agents. I beleive the idea is to enable the Dask framework for parallel execution for TARDIS. So, we can run many iterations of our simulations with different inputs automatically.

3. A problem I faced, which I condisder to be the most difficult thus far in the program, is after unpinning sphinx in the carsus environment for making the documenation for python 3, the tests were no longer running. As I receive this error: RecursionError: maximum recursion depth exceeded while calling a Python object. I can simply increase the maximum recursion depth, but that is frown upon. The two main dependecies to investegate are sphinx and astropy_helpers. The recursion happens within astropy_helpers, but it is related to sphinx, and I cannot do anything about sphinx, so evidently I will try to either find the right astropy_helpers version or change something about the git sub module we use.

Last week, I had completely cleaned the script I had for deploying the documentation, with help from Jaladh Singhal. As I had a misconception of what exactly is published to the gh-pages and how. I learned that the root contents in the gh-pages branch are what is published directly. 

This week, I finished publishing the documentation on gh-pages. The version on my forked repository can be seen here: https://youssef15015.github.io/tardis/workflow/continuous_integration.html. As stated last week, I learned the required language to edit Sphinx files to publish the documentation. The selected language was restructured text reST. I editied *.rst files so sphinx can convert them into *.html files. 

After that, I installed and tested a self hosted agent on my computer, then later did the same on a remote server, which will later be used to test more intensive builds. Instead of using the supplied azure hosted agents to test and build TARDIS on a virtual machine, we willl also use a self hosted agent for more computational duties. The documentation has already been added to the link above. This also contains a pre-existing environment, which could save time in the future for other purposes. I also added a simple pipeline file that does not trigger to test the self hosted vm.

I had one unwelcomed incident from github, as I was attempting to add author properties to my commit, and ended somehow diverging a bunch of my commits for more than one branch into different new named commits. I solved this using by rebasing and reseting onto the main tardis branches. 

There was some difficulty learning reST, as how to reference links or how to make visual diagrams, but through looking at our organization's already made documentation, I was able to decipher that I also needed to include the file name I had made, containing value-pairs of names to links, in the main rst index file. In addition to realizing that github does not show the diagrams made in rst file, and that I would have to see the update in the html links.

The biggest challenge I faced was a minor problem with the self hosted agent. One must intuitively add an agent to an agent pool. I could simply create an agent pool, and add agents to it after adopting the proper permissions, but for some reason it did not work for the already existing pool called "default", which is owned by azure pipelines, but is meant to host self hosted agents. I later discovered that one requires even more permissions, and added in the documentation how to give someone all the permissions, that the creater of the project [tardis, my mentor] on azure, has.

Highlights of what I learned:

-Restructured text

-Publishing documentation on gh-pages

-Installing and running self hosted agents for pipeline builds