Articles on WangJL's Blog

Week 7 Blog Post

Hazard15020@gmail.com (WangJL) — Sat, 29 Aug 2020 11:32:48 +0000

This week I summarized my work during GSoC 2020. This is the last week on GSoC and I have spent a great summer.

I will continue working on the Tern project. I am now focusing on the multistage dockerfile analysis. We have now passed the first step.

However, I came up with a issue about building the multistage dockerfile. This is a tricky issue and I am working on this.

Thanks for my mentors and the Python Foundation!

GSoC 2020: Final Code Submission

Hazard15020@gmail.com (WangJL) — Mon, 24 Aug 2020 15:37:45 +0000

My proposal

Organization: Python Software Foundation

Tag: Tern

Repositorie link: https://github.com/tern-tools/tern.

Tiltle: Use shlex to parse Dockerfile RUN instruction commands.

Details: Use shlex to parse Dockerfile RUN instruction commands. On looking at the type of parsing needed for full shell scripts embedded in the run command, we may need to develop a shell script parser to catch all places where software could have been installed.

Previous work on the project

Works towards docs:

1. Document YAML data output that Tern produces. https://github.com/tern-tools/tern/pull/561.

Works towards dockerfile analysis:

1. Record git project name and sha. https://github.com/tern-tools/tern/pull/571.

2. Parsing ARG varibales. https://github.com/tern-tools/tern/pull/580.

3. Find Git Project URL. https://github.com/tern-tools/tern/pull/606.

Works towards bug fixing:

1. Fix linting error for helper.py. https://github.com/tern-tools/tern/pull/650.

2. Fix linting error for generator.py. https://github.com/tern-tools/tern/pull/651.

Work during GSoC

Works towards shell script parser:

1. Using Regex to split shell script. https://github.com/tern-tools/tern/pull/717.

2. Add test dockerfiles for split shell script. https://github.com/tern-tools/tern/pull/718.

3. Update functions to use the shell script parser. https://github.com/tern-tools/tern/pull/756.

4. Add report for branch statement. https://github.com/tern-tools/tern/pull/764.

Works towards analysis on multistage dockerfile:

1. Split multistage dockerfile by stage. https://github.com/tern-tools/tern/pull/774.

2. Analyze multistage. https://github.com/tern-tools/tern/pull/786. (Still work in progress).

Works towards bug fixing:

1. Bug fix with Dockerfile RUN parsing. https://github.com/tern-tools/tern/pull/773.

Progress on my GSoC 2020

During the preparation period on the GSoC 2020, I got familiar with the basic operations on Github (filing a issue, how to commit) by working on the docs(PR#561). Then I picked up the issue on the dockerfile analysis. I quickly got through the code and tried to make changes on it. My mentors were very helpful and helped me a lot on the coding style and logic.

I chose the proposal on the shell script parser. It seemed tricky at first, but if I took the plan step by step, it should work at end. The first step is to seperate the commands, and then pick out the key words, and do analysis on the command at last. We had a weekly meeting over Zoom to keep track on my progress and resolve some problems. Finally I finished my proposal by the beginning of August.

Next, I began to work on multistage dockerfile analysis. This was kind of relevant to my previous work. My plan is spliting the multistage dockerfile, building the image and analyzing on the image. Now I have finished the first step. I will keep working on the following steps.

Thanks to my mentors, your help was greatly appreciated. Tern is awesome, I like it! This is the first time that I have participated in open source project, and I have spent a great summer in GSoC!

Week 7 Check-in

Hazard15020@gmail.com (WangJL) — Sun, 23 Aug 2020 03:37:48 +0000

What I have done this week

1. Made changes on the PR(Split the multistage dockerfile for build) and after discussion with my mentor, my PR was merged.

2. Rebased the PR(Analyze multistage dockerfile) to the latest branch. This draft PR is used to test the functionality and if it works, i will split it into small steps to implement it. So i will keep working on this issue after GSoC.

3. Discussed with my mentor on the final evaluation and made preparation on it. I will write a blog here to present my work this summer.

Next week

Finish my final evaluation.

Week 6 Blog Post

Hazard15020@gmail.com (WangJL) — Fri, 14 Aug 2020 01:39:18 +0000

What I have done this week

1. Modified the PR on split multistage dockerfile. The function works fine so far.

2. Filed a draft PR on building and analyzing the multistage dockerfile. This PR is used to test the feasibility and needs modifications. So far we can get the report on each stage. More tests will be run on this function.

Plan on next week

1. More tests on the draft PR and send feedbacks to mentors.

2. Try dockerfile lock.

Thoughts

I am not sure if this is the best way to implement analysis on multistage dockerfile. But at least this should work.

Week 6 Check-in

Hazard15020@gmail.com (WangJL) — Sat, 08 Aug 2020 15:02:56 +0000

What I have done this week

Works towards analyzing multistage dockerfile. I combined the draft PR and the review from my mentors, the new commit is the first step of my plan. We split the multistage dockerfile into seperate dockefiles for build. Here are the changes in the new commit.

1. Modified function check_multistage_dockerfile() to return.

2. Remove function split_multistage_dockerfile() since we are working on the building stage. split_multistage_dockerfile() can be improved on analyze stage.

To Do

1. Improve readability for function check_multistage_dockerfile().

2. Try build images and analyze on them.

Did I get stuck somewhere?

Not yet.

Week 5 Blog Post

Hazard15020@gmail.com (WangJL) — Fri, 31 Jul 2020 12:06:30 +0000

I am not feeling well this week and have asked for leave this week with my mentors. I will catch up with my plan on this weekend or next week.

Week 5 Check-in

Hazard15020@gmail.com (WangJL) — Sat, 25 Jul 2020 15:14:20 +0000

What I have done this week

1. Fixed the bug. The shell script parser exits while parsing complex RUN command. After debugging, I found that function consolidate_commands() will have a dead loop in this case. So I updated the logic in this function to fix it.

2. Build multistage dockerfile. I splited the multistage dockerfile into seperate dockerfiles for building. My mentor has reviewed on my draft code, the logic is right but the implementation should be better.

To Do

1. Finishing the part of multistage dockerfile building part. Improve the draft code.

2. Recognize install snippets that follow environment variable declarations. Issue 770.

Did I get stuck somewhere?

To perform the analysis on multistage dockerfile, i need to learn about how Tern deals with a simple Dockerfile. This is relative to my function designing of the analysis on multistage dockerfile. For now, i have a basic understanding, i need to go deeper while working on analysis on multistage dockerfile.

Week 4 Blog Post

Hazard15020@gmail.com (WangJL) — Sat, 18 Jul 2020 15:29:12 +0000

What I have done this week

1. Give a detailed plan on analyzing multistage dockerfile and work on step1 which is spliting a multistage dockerfile by 'FROM'.

2. Add report on branching command.

TO DO

1. A bug comes with issue #772, tern exists with error while parsing a complicated RUN command. Fix this issue.

2. Work on analyzing multistage dockerfile.

Week 4 Check-in

Hazard15020@gmail.com (WangJL) — Sun, 12 Jul 2020 14:51:10 +0000

What i have done this week

Updating functions to implement a shell script parser for the RUN command.

1. split_command(): Splite a shell script into statements: variable, command, branch and loop.

2. get_shell_commands(): Traverse the statements to pick out command and command in the loop.

3. clean_command(): Remove the tab, line inditation and long whitespaces in the command.

This PR has been merged, and at the end of week I add a report for branch statement which is previously skipped during parse.

With discussion with my mentor, we are now done with the main part of shell script parser. And next week we will move on to another issue.

To do

Perform analysis on multistage Dockerfiles, issue#612.

Here are the steps:

1. Split the Dockerfile by stage, making a single Dockerfile for each stage

2. Build and analyze each stage

3. For reporting, perhaps organize each stage as a different section and indicate that each is a build stage of the next. Pinning a multistage Dockerfile is straightforward.

Implement the first step this week.

Did i get stuck somewhere?

No.

Week 3 Blog Post

Hazard15020@gmail.com (WangJL) — Thu, 09 Jul 2020 00:23:43 +0000

Sorry for the late post.

What i have done this week

During the test for the parse command function, i find that there are long whitespaces that will be parse as package name. So i use shlex to remove long whitespaces.

First use shlex to split into seperate words, and then use space to join them( ' '.join(shlex.split(command)) ), this will remove long whitespaces, tab and line inditations.

TO DO

After discussion with my mentor, we are now at the final step of the shell script parser which is updating the functions to enable this parser in the tern.

Did i get stuck somewhere?

No.

Week 3 Check-in

Hazard15020@gmail.com (WangJL) — Sun, 28 Jun 2020 09:07:45 +0000

What i have done this week

Since we can parse a shell script into statements now. We need to fiter the install command and extact what will be installed in the command.

I use filter_install_command() to get installed packages in the commands. It works well with the package-manager commands like apt-get.

But for commands like wget, these are downloading or file-manager commands, we need to figure out how to deal with them.

TO DO

Modify the draft PR to be able to merged. Give a try on parsing commands like wget, tar. Probably adding these commands to snippest.yml.

Did i get stuck somewhere?

Not yet.

Week 2 Blog Post

Hazard15020@gmail.com (WangJL) — Fri, 19 Jun 2020 12:25:50 +0000

What i have done this week

1. Use clean_command() to remove tabs and line indentations.

2. Add code to extract statements for a loop in parse_shell_loop_and_branch().

3. After discussion with my mentor, i remove the pipe symbol '|' from the seperators and add command 'export' as a variable assignment.

This should finish the first part of my proposal, with the splited concatenated commands we can parse them in the next step.

Here this an issue that we can replace variables in our command, i put this aside and will hanlde this after we finish the whole proposal.

TO DO

Make a plan on how to parse the concatenated commands. I will give a try on the current filter_install_command() function, and give some feedback on this.

Did i get stuck somewhere?

How to handle commands like `wget curl`, they are download commands, but how can we tell they are installing something. This remains to be discussed.

Week 2 Check-in

Hazard15020@gmail.com (WangJL) — Sun, 14 Jun 2020 13:41:54 +0000

What I have done this week

This week, I have finished the spliting part of the concatenated into variable, command, branch and loop, and preparing for futher review.

In the spliting part, the concept is matching the key words:

1. loop: (a) for ...... done; (b) while ...... done; (c) util ...... done;

2. branch: (a)case ...... esac; (b) if ...... fi;

3. varibale: find '=' in one line command.

4. command: the rest are commands.

This is done by using python string function startswith() and endwith(), and find '=' can be done by regex.

During the test, I have met some corner cases where the command contains '\'. '\' stands for continue line in a shell script, our parser can not remove it so it may cause error when finding the possible installed software.

After the discussion with my mentor, we will put this aside and move on to parse the command. Most commands do not contain '\', and the command containing '\' is not an installing command so far in our test file.

We will consider this corner case after we finished the main function.

TO DO

Make necessary modifications on the code with mentor, and move on to parse the command.

Hopefully we can use the existing funciton to finish the parse.

Thoughts

GSOC is a very good opportunity for me to be a part of open source project, this project is very cool and i am enjoying it.

Week 1 Blog Post

Hazard15020@gmail.com (WangJL) — Sun, 07 Jun 2020 05:47:38 +0000

What have done this week

To split a shell script into concatenated commands, i need to find the seperators which are

1. ;("A ; B" Run A and then B, regardless of success of A)

2. &&("A && B" Run B if A succeeded)

3. ||("A || B" Run B if A failed)

Besides, i added more seperators while spliting:

1. :;(stands for TRUE, can be skipped while spliting)

2. | (pipe)

During the parse, i have met a situation where the seperator is in a quoted string. In this case, the seperator should not be splited, so we need to skip single and double quote in the script.

I use Backtracking Control Verbs to solve this issue. It can be used to skip certain pattern while matching. Here is the templete:

unwanted_pattern(*SKIP)(*F)|wanted_pattern

The regex engine will first match the unwanted part, after matched, it will have a bracktrack. During this period, the engine come to (*SKIP)(*F) which indicated abandoning matchinng result and jumping to the rest of text to continue match. So we can use this to skip single and double quote and combined to split shell script.

TO DO

1. Split the concatenated into variable, command, branch and loop.

2. Extract info from loop.

3. Parse the command.

Week 1 Check-in

Hazard15020@gmail.com (WangJL) — Fri, 29 May 2020 05:02:51 +0000

Completed tasks

During the community bonding period, i am working on the first step of my proposal. I have used shlex to split the shell script into tokens, and then find the seperator(&&|;) to concatenate the commands. After the review from my mentor, we find that we can improve the code. We do not need to split into tokens at first. Instead, we can directly find the seperator(&&|;) to seperate the commands. This will save a lot of time, since we are not going through every word in the shell script.

To do

Use split_command function and add function to split branch(if and case), and loop(for and while). We will leave the branch not to be parsed since we do not know which branch to be executed. For loop, we will futher develop a function to extract info from it. Here are 2 small steps i am going to do.

1. split command(using seperator(&&|;)), split branch and loop(finding keywords).

2. Extract loop. This is a simple version, we will just extract the commands without considering the loop.

Notes

1. Parsing command has already been implemented so this part is a key point. After the following 2 small steps finished, we should be able to parse the shell script in a Dockerfile RUN command to find what software maybe installed.

2. Try to break big issue into small independent ones.