Articles on imsahil007's Bloghttps://blogs.python-gsoc.orgUpdates on different articles published on imsahil007's BlogenSat, 21 Aug 2021 19:21:05 +0000Week 11: Final Check Inhttps://blogs.python-gsoc.org/en/imsahil007s-blog/week-11-final-check-in/<div style="font-family: Verdana,Geneva,sans-serif;">
<span style="color: #263c5a; font-size: 20px;"></span> <br>
GSoC period has come to an end. <br>
I have learnt a lot about how to write well-structured, clean code. I did asynchronous development, documentation, and test-driven development.<br>
I am extremely grateful to my mentors, Terri Oda, Anthony Harrison, John Andersen and Harmandeep Singh for their guidance.
<br><br>
Read my final project report here:<br>
<a href="https://medium.com/@imsahil007/google-summer-of-code-21-final-report-fc964256d71b">https://medium.com/@imsahil007/google-summer-of-code-21-final-report-fc964256d71b</a> <br>
It has been an incredible journey with Python Software Foundation and Intel. Thank you!<br>
</div>sahils.1997.s@gmail.com (imsahil007)Sat, 21 Aug 2021 19:21:05 +0000https://blogs.python-gsoc.org/en/imsahil007s-blog/week-11-final-check-in/Week 10: NVD API & Filter Intermediate Reportshttps://blogs.python-gsoc.org/en/imsahil007s-blog/week-10-nvd-api-filter-intermediate-reports/<div style="font-family: Verdana,Geneva,sans-serif;">
<span style="color: #263c5a; font-size: 20px;"><b>What did you do this week?</b></span> <br>
I am done refactoring NVD API. I have also added documentation for NVD API and incremental updates.
I have also added a filter argument for filtering intermediate reports based on tag string.
<br><br><span style="color: #263c5a; font-size: 20px;"><b>What is coming up next?</b></span> <br>
I will be writing some more tests. <br>
<br>
<span style="color: #263c5a; font-size: 20px;"><b>Did you get stuck anywhere?</b></span> <br>
I was getting a lot of 503 requests while refreshing the database multiple times using NVD API. Also asynchronous halting tasks didn't help much. So, I had to add an all over halt using time.sleep() to the task list while fetching the entries.
I have also added a limit of 19 hosts in the NVD API (Similar to what I used in CVE DB before GSoC)
<br><br>
</div>sahils.1997.s@gmail.com (imsahil007)Thu, 12 Aug 2021 16:39:25 +0000https://blogs.python-gsoc.org/en/imsahil007s-blog/week-10-nvd-api-filter-intermediate-reports/Week 9: Refactor NVD APIhttps://blogs.python-gsoc.org/en/imsahil007s-blog/week-9-refactor-nvd-api/<div style="font-family: Verdana,Geneva,sans-serif;">
<span style="color: #263c5a; font-size: 20px;"><b>What did you do this week?</b></span> <br>
I have been refactoring NVD API this past week.
I have removed the JSON cache in case we fetch feeds using API which was unnecessarily taking a long time.
I limited the connections per session to prevent failures.
<br><br><span style="color: #263c5a; font-size: 20px;"><b>What is coming up next?</b></span> <br>
I will be working on changing the above said limit and the interval in case of failed requests. <br>
<br>
<span style="color: #263c5a; font-size: 20px;"><b>Did you get stuck anywhere?</b></span> <br>
Yes, I have been trying to fetch NVD feed counts from NVD dashboard which is not returning contiguous dicitonary - Resulting in tests failure. I will be working on this as well.
<br><br>
</div>sahils.1997.s@gmail.com (imsahil007)Wed, 04 Aug 2021 20:48:45 +0000https://blogs.python-gsoc.org/en/imsahil007s-blog/week-9-refactor-nvd-api/Week 8: NVD API Tests and Docshttps://blogs.python-gsoc.org/en/imsahil007s-blog/week-8-nvd-api-tests-and-docs/<div style="font-family: Verdana,Geneva,sans-serif;">
<span style="color: #263c5a; font-size: 20px;"><b>What did you do this week?</b></span> <br>
I was working on NVD API. I improved the way I was using asyncio.
Wrote some tests and documentation as well.
<br><br><span style="color: #263c5a; font-size: 20px;"><b>What is coming up next?</b></span> <br>
I will be working on the optimzation of NVD API code so that we can use it as the default CVE Retrieval.<br>
<br>
<span style="color: #263c5a; font-size: 20px;"><b>Did you get stuck anywhere?</b></span> <br>
No
<br><br>
</div>sahils.1997.s@gmail.com (imsahil007)Wed, 28 Jul 2021 20:02:06 +0000https://blogs.python-gsoc.org/en/imsahil007s-blog/week-8-nvd-api-tests-and-docs/Week 7: NVD CVE Retrieval APIhttps://blogs.python-gsoc.org/en/imsahil007s-blog/week-7-nvd-cve-retrieval-api/<div style="font-family: Verdana,Geneva,sans-serif;">
<span style="color: #263c5a; font-size: 20px;"><b>What did you do this week?</b></span> <br>
I have been working on the new NVD API to retrieve CVEs and load the cache.
This will act as a backup for our current CVE retrieval and may also allow us to use incremental updates.<br>
You can read more about this <a href="https://csrc.nist.gov/CSRC/media/Projects/National-Vulnerability-Database/documents/web%20service%20documentation/Automation%20Support%20for%20CVE%20Retrieval.pdf">here</a>
<br>
I worked on the asyncio library this past week.
<br><br><span style="color: #263c5a; font-size: 20px;"><b>What is coming up next?</b></span> <br>
I will be working on the incremental updates part of the code. I will also add the related documenation and tests.<br>
<br>
<span style="color: #263c5a; font-size: 20px;"><b>Did you get stuck anywhere?</b></span> <br>
Yes, I wasn't using the asyncio library properly which led to long retreival time. It was taking me more than 30+ minutes to fetch the CVEs single time.
<br><br>
</div>sahils.1997.s@gmail.com (imsahil007)Wed, 21 Jul 2021 17:35:17 +0000https://blogs.python-gsoc.org/en/imsahil007s-blog/week-7-nvd-cve-retrieval-api/Week 6: Intermediate Severity Trace Table in PDF Reportshttps://blogs.python-gsoc.org/en/imsahil007s-blog/week-6-intermediate-severity-trace-table-in-pdf-reports/<div style="font-family: Verdana,Geneva,sans-serif;">
<span style="color: #263c5a; font-size: 20px;"><b>What did you do this week?</b></span> <br>
I have added a severity trace table in PDF reports when output is generated using intermediate scans.<br>
<img alt="Severity Trace" src="https://user-images.githubusercontent.com/35963992/125514329-623b1923-d38d-435e-93eb-cb7b287b383e.png" style="text-align: center; width: 100%;">
You can check an example <a href="https://imsahil007.github.io/severity_output.pdf">here</a>
<br>
<br><br><span style="color: #263c5a; font-size: 20px;"><b>What is coming up next?</b></span> <br>
I will be working on the NVD API to retreive incremental updates. This means, that you user won't have to update the whole database everytime. This will also act as backup in case the current CVE Retreival stops working. I am going to dive deep into the documentation this week.<br>
<br>
<span style="color: #263c5a; font-size: 20px;"><b>Did you get stuck anywhere?</b></span> <br>
No
<br><br>
</div>sahils.1997.s@gmail.com (imsahil007)Wed, 14 Jul 2021 15:54:08 +0000https://blogs.python-gsoc.org/en/imsahil007s-blog/week-6-intermediate-severity-trace-table-in-pdf-reports/Week 5: Update HTML Reports Triage based on Multiple Intermediate Reportshttps://blogs.python-gsoc.org/en/imsahil007s-blog/week-5-update-html-reports-triage-based-on-multiple-intermediate-reports/<div style="font-family: Verdana,Geneva,sans-serif;">
<span style="color: #263c5a; font-size: 20px;"><b>What did you do this week?</b></span> <br>
I am done with improving the HTML Reports when output is generated using intermediate reports.<br>
I have added 4 Ploltly based graphs in the report. Including a severity trace for different severity types i.e. (High, Low, Medium, Critical, Unknown)<br>
<img alt="Severity Trace" src="https://user-images.githubusercontent.com/35963992/124364173-5a4d0b80-dc5d-11eb-8aab-3ddbc63d75ad.png" style="text-align: center; width: 100%;">
You can check an example <a href="https://imsahil007.github.io/intermediate_output.html">here</a>
I have also updated the print_mode in the same reports with a new intermediate based table showing some necessary metadata which will be useful while comparing the triage trends.<br>
<img alt="Intermediate Table" src="https://user-images.githubusercontent.com/35963992/124364209-92ece500-dc5d-11eb-87fc-421f3d8a2ba8.png" style="text-align: center; width: 100%;">
<br><br><span style="color: #263c5a; font-size: 20px;"><b>What is coming up next?</b></span> <br>
I am going to add a similar table representation in the PDF reports. And then I will start working on updating the documentation and some html report related tests.<br>
<br>
<span style="color: #263c5a; font-size: 20px;"><b>Did you get stuck anywhere?</b></span> <br>
No
<br><br>
</div>sahils.1997.s@gmail.com (imsahil007)Tue, 06 Jul 2021 13:40:33 +0000https://blogs.python-gsoc.org/en/imsahil007s-blog/week-5-update-html-reports-triage-based-on-multiple-intermediate-reports/Week 4:Improve HTML Reports based on Multiple Intermediate Reportshttps://blogs.python-gsoc.org/en/imsahil007s-blog/week-4-improve-html-reports-based-on-multiple-intermediate-reports-1/<div style="font-family: Verdana,Geneva,sans-serif;">
<span style="color: #263c5a; font-size: 20px;"><b>What did you do this week?</b></span> <br>
I improved the MergeReport by adding new fields in `merge.py`. Now, we do not require to save a temporary file while merging multiple intermediate reports.<br>
I have also started working on improving the reports when generated using the Intermediate report's utility. As a result, I was able to add a graph to the HTML report.<br>
<img alt="HTML Report" src="https://user-images.githubusercontent.com/35963992/123557697-000d0000-d7b0-11eb-840b-63f26a89e14a.png" style="text-align: center; width: 100%;">
<br><br><span style="color: #263c5a; font-size: 20px;"><b>What is coming up next?</b></span> <br>
I am going to improving the HTML and PDF utility. And add some other data to the HTML reports to represent the triage in a meaninful way.<br>
<span style="font-size: 15px;">Note: Most of the merging will happen via Javascript rather than Python.</span><br>
<br>
<span style="color: #263c5a; font-size: 20px;"><b>Did you get stuck anywhere?</b></span> <br>
Yes. I got stuck while running the Long tests. They were failing for some reason. It turned out that NVD downloads wasn't working for some hours.
<br><br>
</div>sahils.1997.s@gmail.com (imsahil007)Mon, 28 Jun 2021 04:14:20 +0000https://blogs.python-gsoc.org/en/imsahil007s-blog/week-4-improve-html-reports-based-on-multiple-intermediate-reports-1/Week 4:Improve HTML Reports based on Multiple Intermediate Reportshttps://blogs.python-gsoc.org/en/imsahil007s-blog/week-4-improve-html-reports-based-on-multiple-intermediate-reports/<div style="font-family: Verdana,Geneva,sans-serif;">
<span style="color: #263c5a; font-size: 20px;"><b>What did you do this week?</b></span> <br>
I improved the MergeReport by adding new fields in `merge.py`. Now, we do not require to save a temporary file while merging multiple intermediate reports.<br>
I have also started working on improving the reports when generated using the Intermediate report's utility. As a result, I was able to add a graph to the HTML report.<br>
<img alt="HTML Report" src="https://user-images.githubusercontent.com/35963992/123557697-000d0000-d7b0-11eb-840b-63f26a89e14a.png" style="text-align: center; width: 100%;">
<br><br><span style="color: #263c5a; font-size: 20px;"><b>What is coming up next?</b></span> <br>
I am going to improving the HTML and PDF utility. And add some other data to the HTML reports to represent the triage in a meaninful way.<br>
<span style="font-size: 15px;">Note: Most of the merging will happen via Javascript rather than Python.</span><br>
<br>
<span style="color: #263c5a; font-size: 20px;"><b>Did you get stuck anywhere?</b></span> <br>
Yes. I got stuck while running the Long tests. They were failing for some reason. It turned out that NVD downloads wasn't working for some hours.
<br><br>
</div>sahils.1997.s@gmail.com (imsahil007)Sun, 27 Jun 2021 20:28:16 +0000https://blogs.python-gsoc.org/en/imsahil007s-blog/week-4-improve-html-reports-based-on-multiple-intermediate-reports/Week 3: Tests and Documentation for MergeReportshttps://blogs.python-gsoc.org/en/imsahil007s-blog/week-3-tests-and-documentation-for-mergereports/<div style="font-family: Verdana,Geneva,sans-serif;">
<span style="color: #263c5a; font-size: 20px;"><b>What did you do this week?</b></span> <br>
I worked on the documentation and added one <a href="https://github.com/imsahil007/cve-bin-tool/blob/append_doc/doc/how_to_guides/use_intermediate_reports.md">how-to guide</a> which contains a step-wise guide on how to use intermediate reports.
I tweaked the code related to merging reports and also added some tests related to MergeReports in <b style="background: black; color: white; font-size: 20px;">`test_merge.py`</b><br><br>
<span style="color: #263c5a; font-size: 20px;"><b>What is coming up next?</b></span> <br>
I am going to work on the webpage based merging tool utility which will help users while handling multiple intermediate reports. This can take a little more than a week depending on what features/filters we might need to add. And deciding the UI/ CSS part.<br>
<br>
<span style="color: #263c5a; font-size: 20px;"><b>Did you get stuck anywhere?</b></span> <br>
Not yet. But I am not sure what features we might need in the above mentioned HTML utility. We have to also decide the tech-stack related to above tool.
<br><br>
</div>sahils.1997.s@gmail.com (imsahil007)Mon, 21 Jun 2021 15:59:15 +0000https://blogs.python-gsoc.org/en/imsahil007s-blog/week-3-tests-and-documentation-for-mergereports/Week 2: Merging Intermediate Reports using clihttps://blogs.python-gsoc.org/en/imsahil007s-blog/week-2-merging-intermediate-reports-using-cli/<div style="font-family: Verdana,Geneva,sans-serif;">
<span style="color: #263c5a; font-size: 20px;"><b>What did you do this week?</b></span> <br>
I worked on saving intermediate reports along with some metadata and later merge them. This will improve the triage and tracking of reports from different end-users.
The strucutre of these new intermediate reports look like this: <br>
<pre style="background: black; color: white; font-size: 9px;">{
"metadata": {
"timestamp": "2021-06-17.00-00-30",
"tag": "backend",
"scanned_dir": "/home/path/",
"products_with_cve": 139,
"products_without_cve": 2,
"total_files": 49
},
"report": [
{
"vendor": "gnu",
"product": "gcc",
"version": "9.0.1",
"cve_number": "CVE-2019-15847",
"severity": "HIGH",
"score": "7.5",
"cvss_version": "3",
"paths": "/home/path/glib.tar.gz,/home/path/gcc.tar.gz",
"remarks": "NewFound",
"comments": ""
},
]
}
</pre>
I have added 3 parameters in cve-bin-tool cli for same:
<li><span style="background-color: #6f6e6e; color: white; font-family: monospace;"> -a INTERMEDIATE_PATH, --append INTERMEDIATE_PATH </span> :
This will create intermediate reports on current scans and save them in `INTERMEDIATE_PATH`<br>
</li>
<li> <span style="background-color: #6f6e6e; color: white; font-family: monospace;"> -t TAG, --tag TAG </span> :
This will add a unique tag in intermediate reports so that users can differentiate between mulitple intermediate reports.<br></li>
<li><span style="background-color: #6f6e6e; color: white; font-family: monospace;"> -m INTERMEDIATE_REPORTS, --merge INTERMEDIATE_REPORTS </span> :
This will take a list of comma-separated paths and merge them. Users can use this along with `-f --format` and `-o --output-file` to get output in other formats<br></li>
<br> <br>
<span style="color: #263c5a; font-size: 20px;"><b>What is coming up next?</b></span> <br>
I am going to work on the documentation and testing of above-added features in the upcoming weeks.<br>
Possible addition of some filters while using `-m --merge` argument.<br>
A wepage based utitlity to merge these intermediate reports rather than using `-m --merge`.<br> <br>
<span style="color: #263c5a; font-size: 20px;"><b>Did you get stuck anywhere?</b></span> <br>
No
<br><br>
</div>sahils.1997.s@gmail.com (imsahil007)Thu, 17 Jun 2021 04:35:22 +0000https://blogs.python-gsoc.org/en/imsahil007s-blog/week-2-merging-intermediate-reports-using-cli/Week 1: Intermediate JSON Reportshttps://blogs.python-gsoc.org/en/imsahil007s-blog/week-1-intermediate-json-reports/<div style="font-family: Verdana,Geneva,sans-serif;">
Hey there!
I am Sahil, a second-year MCA student at the University of Hyderabad - India. I will be working with CVE Binary Tool under the umbrella of Python Software Foundation. <br> <br>
<span style="color: #263c5a; font-size: 20px;"><b>About the project</b></span> <br>
The CVE Binary Tool scans for a number of common, vulnerable open source components such as openssl, libpng, libxml2, and expat to let you know if a given directory or binary file includes common libraries with known vulnerabilities., known as CVE<br> <br>
<span style="color: #263c5a; font-size: 20px;"><b>What did I do in Community Bonding Period?</b></span> <br>
I have been contributing to CVE Binary Tool for quite some time now. We are planning to use the new NVD API for fetching updated CVE entries. More about this <a href="https://github.com/intel/cve-bin-tool/issues/1125">here</a>. So, I researched about it. I went through the whole documentation. This will help me in the later weeks if we plan to migrate to this API. <br> <br>
<span style="color: #263c5a; font-size: 20px;"><b>What am I doing this week?</b></span> <br>
I am going to work on a medium to save intermediate reports in JSON format. This will improve the triage and tracking of reports from different end-users. I am planning to add a <b>--append</b> argument which will keep separate copies of these intermediate JSON(s). I have to decide the structure of these intermediate reports and verify if there can be an alternate(better) way to saving them.<br> <br>
<span style="color: #263c5a; font-size: 20px;"><b>Have I got stuck anywhere?</b></span> <br>
Not yet.<br><br>
Looking forward to this summer! xD
</div>sahils.1997.s@gmail.com (imsahil007)Mon, 07 Jun 2021 18:54:23 +0000https://blogs.python-gsoc.org/en/imsahil007s-blog/week-1-intermediate-json-reports/