Articles on iifawzi_'s Blog

Weekly Blog - The END!

iifawzie@gmail.com (iifawzi_) — Tue, 24 Aug 2021 01:10:13 +0000

Hi everyone! Time flew so fast.. we've reached the end...

What did you do this week?

I've worked on completing the missing parts, finishing the documentations, and writing the Work Conduct document.. We do really reached the end, couldn't think that I would write this post that fast. Thank you all for being part of this journey. You can find more details about the project @ GSOC-2021-Project-Report

That's it.. I'd love to say that I've passed the best 10 weeks in my life, being able to fix bugs, introduce new features and contribute daily to open source projects under GSoC, was a lifetime experience for me. Thank you all the mentors, python foundation, and program organizers for giving me such an opportunity. Thank you, couldn't be done without you.

Week (10) - Weekly check-in

iifawzie@gmail.com (iifawzi_) — Tue, 17 Aug 2021 10:27:14 +0000

Hi everyone! Time flew so fast.. we've reached the end.
Today, I will share with you what I've been working on, on the last week.

What did you do this week?

This week I've worked on testing all the version 2 APIs, I've used Mocha and Chai for this.. and guess what! everything worked as expected and all the tests passed from the first time.. Yay! Then, I've worked on creating the docker image that should be responsible for building and run all the components we need, out of the box, for easier deployment and development process.

What is coming up next?

Unfortunately, we've reached the end.. there's no any next directly with GSoC! but I will always have an eye on the project, to keep fixing, adding, and contributing.

Did you get stuck anywhere?

Yes, I've faced an issue because we're currently depending on heroku to serve our backend applications, I needed to modify the docker image a little bit to be able to use heroku correctly, since heroku doesn't allow using other services, such as redis, databases and so on. I've gone over it by using heroku.yml file, a file similar to docker-compose, to have some special configuration specifically for heroku. That's it.. I'd love to say that I've passed the best 10 weeks in my life, being able to fix bugs, introduce new features and contributing daily to open source projects under GSoC was a lifetime experience for me. Thank you all the mentors, python foundation, and program organizers for giving me such an opportunity. Thank you, couldn't be done without you.

Week 9 - Last but not least!

iifawzie@gmail.com (iifawzi_) — Tue, 10 Aug 2021 11:48:54 +0000

Hi, I hope that you're all doing great!
This week I've been working on adding the searching layer configurations, as I've mentioned before we decided to go with Algolia services, they have a great plans, super easy and efficient control panel to configure everything related to your index, using algolia necessitates adding some sort of synchronization to always keep our Algolia index updated, therefore I've created a method that will always be triggered when ever the main database got updated, to update the Algolia's index.
2- We've also noticed the importance of adding a validation layer in front of our APIs to ensure that they are going to be used as expected, and to respond with errors as soon as possible, we've used JOI to achieve this and it worked like a charm, Anyway, it's only added to V2 APIs. We've also added an isAdmin middleware to check whether the request is authenticated or not, this will only be used in the suggestions management APIs.

What's next?

Days fly! the next week is the last week, I will work on adding the unit tests to ensure that everything is working as expected, and I will also work on configuring the docker image for easier deployment process. That's it :')

Thank you for reading!

Week (8) - Weekly check-in

iifawzie@gmail.com (iifawzi_) — Mon, 02 Aug 2021 16:12:09 +0000

Hi everyone! Today, I will share some info about what I've been working on last week.

What did you do this week?

This week I've worked on introducing the suggestions functionalities, the idea is that we want to get our users involved in the icons improvements, therefore we've decided to introduce the suggestions APIs where the users can suggest tags or categories for any icon.

What is coming up next?

I will be working on finishing the searching layer, we will use Algolia for this, and then I will start working on the last milestone, it will focus on finalizing the work by finishing the documents, adding some unit tests to ensure that everything is working as expected, and creating a docker image for a solid deployment soltuion.

Did you get stuck anywhere?

No, it all went too smooth.

Week (7) - Introducing the caching layer

iifawzie@gmail.com (iifawzi_) — Tue, 27 Jul 2021 05:03:21 +0000

Hi, I hope that you're all doing great!
This Week I've worked on introducing the caching layer, previously we were depending on the MongoDB collection to fetch the requested icons, and then perform the requested changes ( icon customizations ), we wanted to make this process faster, thus we introduced the Redis layer, where we cache all the icons in the memory. The redis layer got updated whenever any update occurred to the db, this way our caching layer will always be up to date, and I've decided to cache all the icons because we need to decrease the miss probability ( checking the redis layer and not finding the icon ), and since we're just caching the raw non-customized icons, we don't need to wait until a miss happen to cache the icon.

What's next?

Currently, i'm working on the suggestions APIs, we need to provide the ability of suggesting tags, I've encountered some issues regarding how will we be able to manage the votes, no accounts are involved in this process. We didn't decide how we will manage it yet, stay tuned for the next blog!

Week (6) - Weekly check-in

iifawzie@gmail.com (iifawzi_) — Mon, 19 Jul 2021 11:39:07 +0000

Hi everyone!
Today, I will share some info about what I've been working on last week.

What did you do this week?

This week went too smooth, I've been working on introducing the caching layer, fixed some issues regarding the previous milestones.

The important thing that I've made so far is that we've updated the process of updating the icons, to only update the fields that changed, to reduce the db overhead, and introduced another two fields to give us the ability to show our users if an icon's look is changed.

What is coming up next?

Next, I will update our v2 APIs to use the caching layer, then I will be working on introducing the APIs that will deal with icons's tags suggestions, and the search APIs - most probably we will use Algolia for this - . That's it for this milestone.

Did you get stuck anywhere?

No, it all went too smooth.

That's it for this week

Week (5) - Finishing the version 2 APIs

iifawzie@gmail.com (iifawzi_) — Sun, 11 Jul 2021 23:04:53 +0000

Hi everyone!
In this week, I've worked on implementing the logic behind the version 2 APIs, we were having 6 APIs that were responsible for downloading png, svg or font icons, and it was only support customizing either the svg or the png, but not the font. Version 2 APIs is doing the same functionalities but with only 3 API, also I've added the possibility of generate a font for customized icons (flipping, rotation), and the ability of retrieving Bae64 string. by the end of this week, we've ended all the business functionalities, no additional features will be added, we will only work on improvements and making the process to the current apis much easier, by introducing caching and other features.

Updating icons process

The updating icons process were the hardest part in the previous two weeks, I will explain in short, how I implemented it. We depended on the Gitlab hooks to notify our backend if any new release is released, then the API will trigger the process of updating the icon and respond to the gitlab immediately. The updating process will start with updating the local <stron>eos-icons dependency to use it for fetching the latest icons by comparing the date stored in the icon object (retrieved from the json file as mentioned in the previous blog), and then compared to the date of the last update which's stored in the db, Then after preparing the objects with the corresponding svg codes, we add it to the database, The new icons are bulk inserted in the db, and the updated icons are first prepared using a factory function that creates the appropriate schema to be used against MongoDB, and each icon is stored separately, unfortunately, there's no better solution. We might discover one later.

What's next?

Next weeks, I will be working on implementing the searching API using Algolia, implementing the cache layer using Redis, and adding the APIs that will be responsible for suggesting the icons.</stron>

Week (4) - weekly check-in!

iifawzie@gmail.com (iifawzi_) — Mon, 05 Jul 2021 00:10:12 +0000

Hi everyone!
Today, I will share some info about what I've been working on last week.

What did you do this week?

All this week, I've been working on the process of updating the icons when a new release is released. Huh? I know you're confused now, lemme elaborate more on this, we have two repositories, the first one is used for the icons and it will automatically release the package in npm. The second repository, is the backend, which's the project i'm working on. We need to make the backend aware of any new releases, that's it.

For this process to work as expected, we need to depend on the Gitlab hooks (the icons repository is located there), when a commit is add to that repo and a tag is released, the backend should be notified, and then the backend will be responsible for updating the database with the new icons and perform all the required changes, How could it do that? we first read the eos-icons.json file, where all the info about the icons - except the SVG itself - are stored in JSON structure, and then updating the local package eos-icons to have the new icons, to be able then to fetch the SVG codes, and add it to the corresponding icon's object.

I might share some in-depth details in next week blog! don't miss it.
What is coming up next?
Next week, I will be working on adding Version 2 of the APIs, which will decrease the number of APIs, the code redundancy, and will have a fixed responses schema, I will not add any additional functionalities next week, it will only be an improved/refactored version of the v1 APIs, the new features will be added the week after next.
Did you get stuck anywhere?
Yeah, I've got stuck a lot this week, the process of updating the icons needed a good planing to avoid consume a lot of resources while doing it, looking/thinking and trying different approaches, took a lot of time. But in the end? hard work pays off, it's working like a charm now!.

That's it for this week
Stay tuned for the next week's blog.

Week (3) - What we've made so far?

iifawzie@gmail.com (iifawzi_) — Wed, 30 Jun 2021 14:18:18 +0000

Hi everyone,
Today I will share with you some in-depth details about what we've done in the last couple of weeks.

The road to the new version

We've migrated and improved all the old APIs, previously the application was using a singly entry point, where all server configurations, routes and controllers were in the same file, since we're going to add a lot of new features. We've decided that we need to organize it a little bit, decided that we will use a folder per component structure, where all the files related to one component (say, Users) will be located in the same folder, the controller, model, interfaces, router, and tests, This would help us in easily locating any file if any changes needed to be made, and also would help us in scaling if we wanted to move any component to elsewhere.

Then, we've worked on making the improvements, many of them were focusing on making the application behaves faster in a scale, and this was done by converting the synchronous calls that wouldn't make difference if it called asynchronously, and also using async and await for some calls instead of sync, You might say that this doesn't change anything, in-fact, calling functions asynchronously and then awaiting on them will help in not blocking the entire event loop while these functions are working, but instead it will only block the context at which this function is called in, this would lead to a huge difference that can be noticed with concurrent requests specially because we have many I/O calls.

What about Continuous integration

This was one of the most important things I've learned in the previous weeks, it was my first time trying to implement a small continuous integration process. We want to apply all the tests on each new PR or a commit to to our main branch, this was very needed and important to be made, to make sure that with each pull request, All the APIs are working as expected, the process might seems so easy to go! but it wasn't, I've faced some struggles to make it done and that's because we're using postman tests for Version 1 APIs, postman tests can be ran using newman command line, the problem was that we need to start the server, run the tests using newman and then, terminating the server. Github actions need the previous step to be terminated or finished before starting a new step or command, and we needed to start the server (step 1), keep the server running and then test the APIs (step 2) without terminating the step 1!, how would we even start the step 2 and step 1 is not terminated yet! Github will stay waiting for step 1 forever. That's being said I needed to figure another way, I've created a script to run the testing script from the server itself, this way we will only make the step 1 from Github, and then making the server dealing with newman command lines, and then terminating the server after the execution of the tests, with an appropriate status code to let the Github knows if the tests are passed or not. Worked like a charm!!

What I will be working on in the next 10 days?

The next 10 days, including this weekend and the next week, I will be working on establishing the process of updating the database icons, we've decided that we will use a MongoDB as a source of truth, that's being said, we need to handle the case if any icons are added to the main repository on Gitlab, This couldn't be done without using Gitlab actions, kinda interesting? we my discover it later in the future blog posts :'p Also, by end of this 10 days, All the new APIs will be created, an improved APIs that will do the same functionality but in a better manner, no new featured will be added, but probably we will decrease the number of APIs, and have a unified responses. Anyway, new features are in the plan, stay tuned!

That's for now!
Thank you for reading.

Week 2 - The end of the first Milestone!

iifawzie@gmail.com (iifawzi_) — Sun, 20 Jun 2021 23:23:35 +0000

Hi everyone!
A lot of things were made this week, and i'm super excited to share some details with you in this post!

What did you do this week?

This week we've worked on the migration of all of the old APIs to typescript, including the fixes and enhancements. Most of enhancements were focusing on making the code more contribution welcome and maintainable, by splitting the files to understandable structure, making the code more simple, cleaner and commented. Special thanks to all the contributors who contributed to the old APIs, I've just refactored a bit, most of the functionality code were written by them, thank you!.

What is coming up next?

For the next three weeks, I will be working on adding a MongoDB layer, where all icons will be stored there and we will then depend on that layer to retrieve the svg code of the icons them making the needed conversions or customizations. This addition will require some changes to the process of updating the icons, mainly we will depend on gitlab hooks to fetch the newly added icons, I will explain the process it in detail in the upcoming posts.

And two minutes ago before writing this post, I've created the last PR in this milestone, which's the configuration of Github CI to make a linting check, and testing the APIs for each push or pull request. I've learned a lot from playing with Github CI/CD docs, It took time, but I've enjoyed it.

Did you get stuck anywhere?

Not totally stuck, but I've spent a lot of time this week, thinking about the best decisions and the best approaches to approach some tasks, enjoyed and learned a lot.

Weekly Blog (1)

iifawzie@gmail.com (iifawzi_) — Sun, 13 Jun 2021 15:48:16 +0000

Hey! First week is almost done! we've worked on a lot of great things. In this blog I will give you a brief about the 7 Pull requests I've made last week. We started by initializing the project using typescript, and separating the server code from the application code. After that I've configured the main application configurations, routing settings, and error handlers. We've focused on making everything simple, efficient, and clean. And this couldn't be achieved without separating everything and creating pure functions. Later, I've migrated some of the utilities that were used before, with some enhancements,. The main change that I want to mention is, that we've changed all promise style functions to async and await functions for a better readability, and a consistent way, since it was used in some places in the code, the second most important change we've made is, converting some of the synchronous calls, to asynchronous calls and awaiting on them, this would help us not blocking the entire event-loop, by using async + await, only the context at which the await is called, will be blocked. We’ve also introduced a logger functionality to log the errors in the console and in a log file, this week included a lot of design decisions, which was the hardest part. Those are the main changes that we’ve introduced last week.

Initial Week - Community Bounding

iifawzie@gmail.com (iifawzi_) — Thu, 10 Jun 2021 09:36:41 +0000

Hi, my name is Fawzi Abdulfattah and I will be working on EOS-ICONS APIs, the backend system that do all the magic behind https://eos-icons.com/ In the last couple of weeks (Community Bounding) We were discussing, planning and deciding what aspects should be improved during the first milestone (first two weeks) and what technologies or services will be involved in other milestones, like what search engine should we use? what's the most appropriate translation API? and so fourth, also I've helped in the migration of a Repository from gitlab to github. In the next couple of weeks, we'll be working on refactoring the current codebase, migrating it to typescript, and adding some enhancements as well. That's it for now! Stay tuned, wish us luck.