Articles on sounak98's Blog

Final Report

sounak.98@gmail.com (sounak98) — Mon, 26 Aug 2019 05:55:35 +0000

Summer Rewind

Let’s rewind to the beginning of this year. We had started working on this application way before GSoC had even started. The goal was to have a working application which PSF will be able to use for this year’s GSoC for the management of their students. In this way, we will be able to make sure that students actually use it and we get a clear idea if the application is serving its purpose. I’m glad the plan worked out, because tons of bugs were reported and we could fix them. We also received valuable feedback from all users.

Schedulers and Builders

Allow me to introduce you to some of the most important modules of our system. Without any doubt the first on the list is our Scheduler, which can perform particular tasks from sending an email to archiving webpages. Well, the most powerful feature of this module is that it can perform those tasks at any particular date and time. Need to remind students that they have not written a blog on time? Not a big deal, Scheduler can do that for you. Now think of this, many students who are not like me publish their blogs on time, so we don’t really need to spam them with emails. Thus we built the module Builder which in turn builds Scheduler on different conditions.

Blogging Platform

We didn’t have to create a blogging platform as we integrated aldryn-newsblog but we had to tweak it a lot to fit into our system. Something that we had to work on was setting up custom permissions for each user so that they only have access to their blogs. We achieved this with the help of django admin which allows us to set add, view, change permissions based on querysets! Sanitizing the artilce contents was another challenge that we faced because aldryn-newsblog uses an editor which injects HTML so that users can customize their blog posts. Our system currently allows only particular tags like

, etc. Other tags are sanitized conditionally, like for iframes we only render iframes for YouTube videos so that users can add YouTube videos to their articles.

We also tweaked the article list templates to include our own reddit-styled comment system which makes use of recursion on django templates to display different threads. For more information on how to achieve this, check out the article Creating Reddit Styled Comment System with Django .

Tweaking Django

We used django-forms wherever we could, but there were cases where we needed to customize the forms to an extent which wasn’t supported out of the box. We have a form which lets suborg admins and admins add selected students to the system. Generally, this includes adding a lot of students (~50). Typing out the emails one by one is still okay, but selecting the GSoC year or the Suborg one by one for each student? Ask my mentor, and he will let you know what a pain it is. So, we tweaked the django form to add buttons which would let the admin select a particular Suborg, year for all the mentioned users.

We have RSS feeds for each blog separately (all the articles published by a student) and also for all the articles published on this platform. Django has Syndication Feed Framework which allows customizable RSS feeds, but the all articles feed was too long and took seconds to load. We needed to paginate the feed, and we were out of luck as django didn’t support this out of the box. This was a challenge, as we had to take in the request object and parse the url to get the page number and render the blogs accordingly. We also added the year argument which takes in the year and displays the blogs of that particular GSoC. The current feed url looks something like this https://blogs.python-gsoc.org/en/feed/?y=2019&p=2.

For most of the other admin features, we heavily relied upon django admin. The admin portal lets admins

Add new Schedulers and Builders
Check blog post histories of different articles to track changes
Add the current GSoC timeline which gets pushed to the schedule page in the github site repository
Send custom emails to users as admins
Review comments and delete them if necessary
Disable a user profile if the student fails mid-way

Integration with Github

We annotated some of the manual work that an admin has to put in to maintain the static site on Github. Our system creates pull requests adding new Suborgs in the Ideas page whenever a new Suborg Application is added, it also archives current pages when the GSoC program ends. These pulls can then be reviewed and merged to master by the admin.

Fixing Bugs

This was really a major part of the whole summer and it went hand in hand with the whole developmene process. There were bugs that were found by the users, and the others figured out by the mentors and me. There are a ton of “Bugfix” PRs which were basically bug fixes.

There was a time when we pushed some changes and it made the system send emails to all the users regardless of whether they have blogged or not. Yes, basically we spammed a lot of users unintentionally. This was another challenge that we had faced and overcame eventually by making a flag which would disable all notifications to any user. We also followed a strict push cycle to avoid any disturbance to students blogging at the end of the week.

Wrapping Up

We ran accessibility tests on our websites and fixed issues which decrease the accessibility of the website, like fixing contrast ratios of texts and background, adding alts to images, etc. We also worked on boosting the loading speed of out website on mobiles. In built tools provided by Chrome and Firefox gave us a list of issues after analyzing the website which we could work on.

We also ended up using a cache server to cache the data to fasten up the whole loading process. We also needed to manually override caching in some pages like the comments page, which would not show the new comment as the old one was cached. This is the issue which describes more about this bug and how we solved it.

Future Plans

Currently, the platform provides most of the functionality required for a smooth GSoC run at PSF, but there are features that would make it even smoother for the admins and make their lives a bit less painful. One of them is adding the mentors to the GSoC site automatically from the system’s database. This can be another nasty manual work (typing in the emails, names, etc. one by one) and needs to be done automatically. For more details, check this issue out.

We also need to write unit and integration tests for features that are not provided by django or any third-party packages.

I would love to work on these in future even when this GSoC ends, fix more bugs as and when they come up and be a part of this great community!

Credits

First of all, any of this would not have been possible without my mentors and other members of the PSF community. So a huge shout-out to them for helping me whenever I needed and for guiding me when I was clueless on how to proceed. While I was busy coding, my mentor would look for potential bugs in the system and point them out to me. This really kept me busy throughout the summer as I always had bugs to fix, and helped me make the system more stable.

Next, I would thank Google for organising such an amazing program for students who are passionate about coding and giving them an opportunity to gain some hands-on experience.

Last but not the least, I would thank my fellow applicants who also worked on building the application with me to bring it to a stage where it could be used in this year’s GSoC run.

Apologies

There are a lot of mistakes that I made and learnt from them. In the beginning I was not testing stuff through before making a PR, sometimes trying to do things faster and the other times just being lazy. It only made me spend more time on a particular feature as there were things that would not work.

Another thing that I should apologize for is being very irregular about posting blogs. This shouldn’t have come from me, as I was the student working on the very blogging platform itself.

Weekly Check-In #10

sounak.98@gmail.com (sounak98) — Mon, 26 Aug 2019 05:53:48 +0000

What did I do in these two weeks?

There were a lot of bugs posted, kind of like the last set of major fixes that we need to make.

be able to add gsoc_year
add username to password reset email
prepopulate have you partisipated before
'Settings' object has no attribute 'ADMIN_EMAIL' on suborg form since we removed that key
github username for org admins/mentors
add additional org admins as part of suborg form
cut out top part of suborg form
make a pr on suborg submit, dont auto commit
cookie notice
add mentors/suborg admins button
add psf privacy policy
checkbox on signing up for PSF terms and another for "opt-in to receive emails"
nuke all student users unless they click a link in an email, ie remove email name etc then readd with get request if they click at end of gsoc (gdpr)

In this week we mostly worked on all of these. Terri posted some issues with the comment notification emails that we had totally overlooked. I fixed those too. We also worked on fixing the existing feed and adding blog wise feeds for every blog.

What's next?

Publishing the final report and making some final changes to improve the accessibility of the site, like fixing the contrast ratios, etc.

Paginate Django Feeds

sounak.98@gmail.com (sounak98) — Mon, 26 Aug 2019 05:48:21 +0000

Django has a great framework for generating feeds, but sadly it doesn't support pagination out of the box. Currently we have tons of blogs and the feeds page was loading especially slow with all the pages in it. I was hoping django will have some class variable to enable pagination but I was out of luck. But it had a method which took in the request, so I knew I had to parse the page number from there.

from django.contrib.syndication.views import Feed

class BlogsFeed(Feed):
    ...
    def get_objects(self, request):
        ...
        page = request.GET.get("p", 1)
        ...
        return queryset_from_page

So in the get_objects method we need to get the page number from the GET args and then return the queryset according to the page number.

That seems to be enought right? Even I thought so. But the RSS standards say that we also need to add entries like url for last page number, first page number and the current page number. Well, we can even do that. The class BlogsFeed takes in a class variable called feed_type and we need to set it to the feed type class and django provides a class for that too called DefaultFeed. Well we will inherit this DefaultFeed and make out own type of Feed which will include the page numbers too.

from django.utils.feedgenerator import DefaultFeed

class PaginateFeed(DefaultFeed):
    content_type = "application/xml; charset=utf-8"

    def add_root_elements(self, handler):
        super(CorrectMimeTypeFeed, self).add_root_elements(handler)
        if self.feed["page"] is not None:
            if not self.feed["show_all_articles"]:
                if (
                    self.feed["page"] >= 1
                    and self.feed["page"] <= self.feed["last_page"]
                ):
                    handler.addQuickElement(
                        "link",
                        "",
                        {
                            "rel": "first",
                            "href": f"{self.feed['feed_url']}?y={self.feed['year']}&p=1",
                        },
                    )
                    handler.addQuickElement(
                        "link",
                        "",
                        {
                            "rel": "last",
                            "href": (
                                f"{self.feed['feed_url']}?y={self.feed['year']}"
                                f"&p={self.feed['last_page']}"
                            ),
                        },
                    )
                    if self.feed["page"] > 1:
                        handler.addQuickElement(
                            "link",
                            "",
                            {
                                "rel": "previous",
                                "href": (
                                    f"{self.feed['feed_url']}?y={self.feed['year']}"
                                    f"&p={self.feed['page'] - 1}"
                                ),
                            },
                        )
                    if self.feed["page"] < self.feed["last_page"]:
                        handler.addQuickElement(
                            "link",
                            "",
                            {
                                "rel": "next",
                                "href": (
                                    f"{self.feed['feed_url']}?y={self.feed['year']}"
                                    f"&p={self.feed['page'] + 1}"
                                ),
                            },
                        )

This code pretty much explains itself. But there is one catch here too self.feed dict does not have a 'page' or a 'year' key. We need to pass that from our BlogsFeed class. Let's see how.

class BlogsFeed(Feed):
    ...
    feed_type = CorrectMimeTypeFeed
    ...
    def feed_extra_kwargs(self, obj):
        return {
            "page": self.page,
            "last_page": self.last_page,
            "show_all_articles": self.show_all_articles,
            "year": self.year,
        }

That's it guys. Now you have your own paginated feed as per the RSS standards.

Weekly Check-In #9

sounak.98@gmail.com (sounak98) — Mon, 26 Aug 2019 05:02:52 +0000

What did I do this week?

We have come to the last phase of the summer. The application is almost ready but we needed to fix more bugs this time to fine tune the application. Some of those which we worked on this week were as follows.

Fixed the sitemap to include each blog post and also different pages for the paginated blog
Handling exceptions on sitemap
Use django for sending mails to admins when any exception is raised on the main server
Fixed a very very old issue by enhancing the user add feature. Now the admin can use single dropdowns to select the default values for multiple fields.

What will I do next week?

Will move towards finishing up whatever is left and work on more issues as and when they come up!

That will be all folks!

Embed Google Calendar like a Pro

sounak.98@gmail.com (sounak98) — Thu, 25 Jul 2019 18:39:13 +0000

Have you ever tried to embed a Google Calendar in your site only to find it's not responsive for mobiles and you need to scroll left and right? Well, I'm gonna share a simple solution which will work.

There's an agenda mode in google calendar which seems to fit properly in a mobile screen, so how about changing it to the agenda mode when the screen size is smaller? Simple HTML and CSS will be enough for this.

<style type="text/css">@media (max-width: 550px) {
     .big-container {
         display: none;
     }
 }
 @media (min-width: 550px) {
     .small-container {
         display: none;
     }
 }
 /* Responsive iFrame */
 .responsive-iframe-container {
     position: relative;
     padding-bottom: 56.25%;
     padding-top: 30px;
     height: 0;
     overflow: hidden;
 }
 .responsive-iframe-container iframe,   
 .vresponsive-iframe-container object,  
 .vresponsive-iframe-container embed {
     position: absolute;
     top: 0;
     left: 0;
     width: 100%;
     height: 100%;
 }</style>

<iframe height="600" src="null" style="border-width: 0;" width="800"></iframe>
<iframe height="600" src="null" style="border-width: 0;" width="800"></iframe>

This is be good enough for a responsive calendar. Want another pro tip? How about changing timezones of the calendar according to the client timezone?

Well we will need a bit of js for getting the client's timezone.

var offset = new Date().getTimezoneOffset();

This line of code gets the offset integer in minutes, so -330 for +05:30 timezone and so on. Then we somehow need to convert this to the region name as google calendar link takes an argument named ctz for that. Include the moment.min.js and the moment-timezone-with-data-10-year-range.min.js. These libraries will help us get the region name.

String.prototype.replaceAll = function(search, replacement) {
    var target = this;
    return target.split(search).join(replacement);
};
var timezone = moment.tz.guess(offset).replaceAll('/', '%2F')

Now you just need to add a new argument to the url.

newUrl = `${oldUrl}?ctz=${timezone}`
document.getElementById('calendar').src = newUrl;

That's it guys and you will be good to go.

Reference: https://answers.squarespace.com/questions/54774/how-to-embed-a-google-calendar-in-a-responsive-way.html

SEO 101: Creating custom sitemap using django

sounak.98@gmail.com (sounak98) — Thu, 25 Jul 2019 18:01:30 +0000

I must agree I was pretty impatient to use the codesnippet plugin on this editor.

print("tada!")

Well, a couple of days back I was trying to find a way to create sitemaps in django. One way is to use a crawler to crawl all your pages and generate the sitemap. But that's a bad solution as you need to manually update it everytime. We are looking for something more dynamic. Thanks to django, it already has a base class which helps in generation of sitemaps.

Let's start with the urls.py. We add the url for sitemap.xml and add the sitemaps object specifying which sitemap class to use. We can add multiple sitemaps together, and django will render it together for you.

from myblog import sitemaps

urlpatterns = [
    url(r'^sitemap.xml', sitemap, {
      'sitemaps': {
        'blogs': sitemaps.BlogSitemap
      }
    }
]

Then let's go on to sitemaps.py

import urllib.parse

from django.contrib.sitemaps import Sitemap
from django.conf import settings

from aldryn_newsblog.cms_appconfig import NewsBlogConfig
from aldryn_newsblog.models import Article

from cms.models import Page


class BlogSitemap(Sitemap):
    priority = 0.5

    def items(self):
        urls = ['/']
        blogs = NewsBlogConfig.objects.all()
        for blog in blogs:
            p = Page.objects.get(application_namespace=blog.namespace, publisher_is_draft=False)
            urls.append(p.get_absolute_url())
            articles = Article.objects.filter(app_config=blog).all()
            for i in range(len(articles) // 5):
                urls.append(f'{p.get_absolute_url()}?page={i + 2}')
            for article in articles:
                urls.append(f'{p.get_absolute_url()}{article.slug}/')
        return urls

    def location(self, obj):
        return obj

Generally items() should return an iterable of objects which will have a method called get_absolute_url() which will give the url, but django gives you the option to specify their locations too using the loaction(item) method. We are using this cleverly to just return a list of urls in the items method and then returning the item in the location method as the item is nothing but a string which is the url. In this way we can include pages like /blog-name/?page=2 which aren't returned by any object's get_absolute_url() method.

Weekly Check-In #8

sounak.98@gmail.com (sounak98) — Thu, 25 Jul 2019 17:45:37 +0000

A lot of fixes and new features came up this week. It was kind of a busy week.

We mostly worked on Search Engine Optimization (yes, SEO). We didn't really change any content but added appropriate meta description tags, created a uniform title format for both python-gsoc.org and blogs.python-gsoc.org. Also, we used services from search engines (yes, Google) to analyze our site for both PC and mobile performance. Some content was weird for mobiles, so we fixed that. We also created proper sitemaps for both the sites.

Except this, we also created a Send Email admin panel to send customized emails. It's like a small email client which can be used to sent emails. This will be used for sending emails to mentors and suborg admins or maybe some students when needed. It also has the feature of sending emails to groups like students, mentors, etc. The blog editor now also has options to upload image and attachments just by dragging and dropping. Yes, pretty useful and a cool feature right? :p

I'm gonna finish the leftover issues as fast as possible and get to testing!

We are almost there now!

sounak.98@gmail.com (sounak98) — Thu, 25 Jul 2019 16:46:14 +0000

Yes, we have almost finished implementing all the functionalities! It feels kinda sad that GSoC is going to end in a matter of weeks now. But nevertheless, this experience has been an enriching one for me. I am grateful to my mentor for his wonderful support throughout this. He has guided me through the whole development phase and helped me to be focused on the timeline and to keep a track of the bugs that come up.

What have I learnt from this till now? Well, the fact that software engineering is not an easy job! Especially when a couple of developers build a software it's easy to miss out a lot of cases which will surely come up as bugs. And even this has happened in our case, a new bug comes up while trying to fix an older one. So yeah, we need to keep an eye out for bugs, it's not always possible to build everything perfectly from the beginning.

Also, there has been times when I had a vision of building something (maybe implementing a feature) with utmost perfection and I was not being able to do that. Thanks to my mentor for pointing out that the vision I had was not something feasible and that we need to approach it in a different way. This was an important lesson for me as to what might seem to be perfect might not be perfect at all because it is not feasible.

And also a full testing is coming up now! I have never done this before and thus am looking forward to it very much. It will be a totally new experience and I hope it will be enjoyable. I guess more bugs will come up now :p.

Weekly Check-In #7

sounak.98@gmail.com (sounak98) — Thu, 25 Jul 2019 16:03:58 +0000

We have come to almost the end of the second phase. We are now spending time on the last few features that needs to be implemented and also the bugs that have come up in the recent past.

First of all we fixed bugs as usual. One of them was giving permissions to the users for adding notification blocks to their pages. We had implemented before GSoC even started but forgot to give permissions to any new user. Also, we were having issues tracking different types of blogs that the students were supposed to post. The system considered all the blog posts to be of the same type, whereas there are two of them. We implemented this into the system so that the students are notified accordingly.

Our system now automatically archives the Github Pages at the end of every GSoC, yaay for the maintainers! Oh and finally, I added the support for codeblock in the blogs (this was really necessary :p). The suborg forms also have been improved now as some of the fields get disabled conditionally, so overall a better UX for users filling out the form.

Will be doing more stuff like this in the next week too!

Weekly Check-In #6

sounak.98@gmail.com (sounak98) — Mon, 15 Jul 2019 03:32:37 +0000

The application is almost built now. Only minor changes, bugfixes and writing tests are left.

In the last week, I fixed a lot of new and pending bugs from the past. All the issues and the fixes can be found on https://github.com/python-gsoc/python-blogs/pull/273 and https://github.com/python-gsoc/python-blogs/pull/285. I was also sick for the last couple of days so there was a short pause. My mentor and I also figured out some SEO stuff that needs to be worked on. Our site was loading slower than usual. We fixed it by installing memcache which in turn gave rise to some other bugs (this has been happening since the very beginning :p).

In this week I plan to finish all the pending issues so that I can start with either testing or UI refactoring from the next week. For the testing we are thinking of running a whole GSoC program using scripts. Let's see how that turns out. Adios.

Creating Reddit Styled Comment System with Django

sounak.98@gmail.com (sounak98) — Fri, 28 Jun 2019 15:43:04 +0000

Okay, so you are probably blogging in this platform and have used the comment system once in a while. Those who have will notice that the comment system is pretty much like reddit without the upvote and downvote feature. This is actually pretty simple to create and can be created using only one django template which calls itself recursively! Yes you heard that right, recursing on django templates. It might sound pretty fancy, but is basically a very simple idea.

So let's begin with the model. We create a simple django model with the following fields

user (a foreign key to user referring to the author of the comment)
article (a foreign key to article referring to the article on which the comment was made)
content (the content of the comment)
parent (a self-referential field referring to the parent comment, and is a root comment if there is no parent)
created_at (the time and date of comment creation)

Once the model is made, we can create the migrations and migrate the database using the commands `python manage.py makemigrations` and `python manage.py migrate`. Now that the model is created and the database is migrated, we need to create views for addition and deletion of comments. We can also create a view for editing comments but that is not what we will discuss in this article. So the add view should be a POST view which takes an `user_id`, `article_id`, `content` and `parent`. Once we validate these data, we can create a new `Comment` using

Comment.objects.create(user=User.objects.get(id=user_id), article=Article.objects.get(id=article_id), content=content, parent=parent)

Make sure that `parent` is `None` if there is no parent comment.

Okay we are almost done. We just need to work on the template now.

So we will render the template with the context variables `root_comments` which will have all the root comments of the article. Now we will create another template called `comments.html` which will be passed some variables called `comments`, `user`, etc from the driver template. It will look something like this

{% include "aldryn_newsblog/includes/comments.html" with comments=root_comments article=article user=user csrf_token=csrf_token only %}

This `comments.html` will run a for loop to display all the comments and with each iteration it will also call the same template recursively like this

{% include "aldryn_newsblog/includes/comments.html" with comments=comment.replies.all parent=comment article=article user=user csrf_token=csrf_token only %}

We need to use the keyword only so that only these variables are passed and not the parent template variables which will override the correct ones. So, we will also need to pass some variables like `user`, `crsf_token`, etc. Also, we need to create a form which will POST to the add comment view with every template to reply to the parent comment in that template.

We can also implement a bunch of other features like hiding the reply threads, adding recaptcha verification, etc. The code can be found on this repo. Feel free to use it or take help from the same.

That's it for today. Hope it helps.

Weekly Check-In #5

sounak.98@gmail.com (sounak98) — Tue, 25 Jun 2019 10:59:41 +0000

What did I do this week?

Finished up with the suborg application form which when filled out by a user sends out a notification to the admin. Then the admin can review it and send messages to the user. The user can change the application as per the instructions and get the application accepted.

Also now anyone can create an account using their email. This is a new feature that has been implemented.

What's coming up next?

Personalised notifications and dashboard for users

Weekly Check-In #4

sounak.98@gmail.com (sounak98) — Tue, 25 Jun 2019 04:09:47 +0000

What did I do this week?

Added build automation using Travis CI
Started writing unit tests for the application
[MAJOR] Started working on the Suborg Application

What's coming up next?

Finishing up the Suborg Application flow
Write more tests

Weekly Check-In #3

sounak.98@gmail.com (sounak98) — Sun, 09 Jun 2019 04:14:37 +0000

What did I do this week

[MAJOR] Auto creates GSOC@PSF Google Calendar from timeline, events and due dates
Add the new management command googleapiauth for authenticating the Google API
Fixed reminder notification bugs
Send notifications for comments
Refactored models
Minor updates on cleaning up the article contents
Updates blog count when an article is created and not updated

What's coming up next

Create the SubOrg Form

Did I get stuck anywhere

No, nothing as such.

Thanks for staying till the end.

Weekly Check-In #2

sounak.98@gmail.com (sounak98) — Tue, 04 Jun 2019 14:47:44 +0000

Will keep it short and to the point this week.

What did I do this week

[MAJOR] Added feature for reminding students and mentors of missed blogs
Added functionality for reviewing blogs for admins
Fixed editing blogs (using a different field which didn't need separate permission, and thus also had to migrate data from the all the older blogs)
Add publish/unpublish feature for a student to display or to hide blogs from the main site

What's coming up next

Automatically create google calendar from timeline
Fix reminder notification bugs
Create comment notifications
Fix more bugs as they come up

Did I get stuck anywhere

Nothing major.

Thanks everyone!

Weekly Check-In

sounak.98@gmail.com (sounak98) — Sun, 02 Jun 2019 13:43:57 +0000

Hey everyone, welcome to my blog for this summer's GSoC@PSF!

An introduction first, I am Sounak Pradhan working with PSF-GSoC team of this year intending to build this very platform 😃. This is my first weekly check-in which should have been published some time back but better late than never.

What did I do this week

We had started building this platform back in February as a part of the the application process. Thanks to all the other applicants who made contributions to the project to bring it to a usable state and here we are blogging in this platform instead of a WordPress site which would have been a pain to set up and manage. So even before the program started our portal was ready. So what now? Well, I believe no piece of software is free of bugs and ours is no exception. As students started using it, a few bugs came up. We also had to implement small but necessary features like setting up auto-reminders for students who hadn't registered themselves on the site, displaying registration stats to the admin. We scan proposal pdfs for potential contact details so that we can inform the students and advise them to upload proposals without these personal details. This was broken as the scanner was taking way too much time to scan the pdfs. Had to fix this too.

What's coming up next

Next week, I'm going to set up the builders which will conditionally build schedulers. Schedulers are tasks which can be scheduled long before and will be executed at some later time. For example, I can schedule an email which needs to be sent after 3 days and after the said number of days the email will be sent automatically. We need builders to build these schedulers conditionally, like if a student user hasn't published a blog on time, only then send a reminder mail. We will need this to setup reminders for blogs to all students and also reminders to mentors, suborg admins when the student doesn't post the blog on time.

Did I get stuck anywhere

The way I thought I would handle the reminders for the blogs wasn't working out, botanic (my mentor) helped me out in that. Nothing else that was major, thanks to StackOverflow 😛.

That's it for this week. See you again in the next week folks!