Articles on bksahu's Blog

GSoC Wrap-up - Blog Post #6

bablusahoo16@gmail.com (bksahu) — Thu, 22 Aug 2019 14:42:31 +0000

Hello everyone!

GSoC 2019 has almost come to an end! It's the time to wrap up this mega event started back in May 2019. Under the mentorship of Mentor Hayen, my learning experience has undergone a roller-coaster ride and it has not only boosted my growth as a developer but also as an individual. Over the last 3 months the followings are my major contributions to this project:

Built-ins Optimizations

"any": PR #246 <https://github.com/Nuitka/Nuitka/pull/246> * Issue reported and closed Issue #349 <https://github.com/Nuitka/Nuitka/issues/349>
"all": PR #407 <https://github.com/Nuitka/Nuitka/pull/407>
"abs": PR #419 <https://github.com/Nuitka/Nuitka/pull/419>

The following are in progress: * "max" and "min": PR #442 <https://github.com/Nuitka/Nuitka/pull/442> * "zip": PR #462 <https://github.com/Nuitka/Nuitka/pull/462>

Test suite

Search mode "All": PR #378 <https://github.com/Nuitka/Nuitka/pull/378>
Search mode "Only": PR #333 <https://github.com/Nuitka/Nuitka/pull/333>
Reported and closed Issue #334: PR #336 <https://github.com/Nuitka/Nuitka/pull/336>

Documentation

Nuitka git work flow: PR #485 https://github.com/Nuitka/Nuitka/pull/485

And other minor doc fixes are added with their respective pull requests.

Over this period of time I have learnt a lot, starting from how to add new nodes to Nuitka and optimize them to optimization using reformulations. I plan to keep learning and contributing to Nuikta,

Thanks for stopping by!

bksahu

Weekly Check-in #6

bablusahoo16@gmail.com (bksahu) — Wed, 14 Aug 2019 14:07:41 +0000

1. What did you do this week?

This week I added support for Python 3 <tt>zip()</tt> built-in and added git workflow in the developer manual which can be found here: https://github.com/Nuitka/Nuitka/pull/485. And also worked on the remnants of Python 2 support for the same.

2. What is coming up next?

Now that support for keyword only arguments has been added in Nuitka, I will finish the remaining of optimizations support for min() and max() built-ins.

3. Did you get stuck anywhere?

Yes, I was stuck on reformulation of zip() for Python2.

Blog Post #5

bablusahoo16@gmail.com (bksahu) — Sat, 10 Aug 2019 10:16:07 +0000

Hi, welcome to my blog post #5!

Today we will see how are exception messages are displayed while working with Reformulations in Nuitka. The following is an example try block which is trying to assign iter() to the arguments if supported else throw an type error.

makeTryExceptSingleHandlerNode(
tried=makeStatementsSequence(
statements=[
StatementAssignmentVariable(
variable=zip_iter_variable,
source=ExpressionBuiltinIter1(
value=ExpressionTempVariableRef(
variable=zip_arg_variable, source_ref=source_ref
),
source_ref=source_ref,
),
source_ref=source_ref,
)
for i, zip_iter_variable, zip_arg_variable in zip(
range(len(call_args)), zip_iter_variables, zip_arg_variables
)
],
allow_none=False,
source_ref=source_ref,
),
exception_name="TypeError",
handler_body=StatementRaiseException(
exception_type=ExpressionBuiltinExceptionRef(
exception_name="TypeError", source_ref=source_ref
),
exception_value=makeConstantRefNode(
constant="zip argument #%d must support iteration" % (i + 1),
source_ref=source_ref,
),
exception_trace=None,
exception_cause=None,
source_ref=source_ref,
),
source_ref=source_ref,
)

You can see there that I'm calling create a StatementRaiseException with a template i.e "zip argument #%d must support iteration" in our case, which is basically to handle cases like for example: zip(1, [1,2,3]).

Thanks for reading

Weekly Checkin #5

bablusahoo16@gmail.com (bksahu) — Thu, 01 Aug 2019 10:09:22 +0000

1. What did you do this week?

I'm still going on with zip reformulation and at the same time also working on the keyfun parameter of the min and max builtin. To be completely honest this week has not be that productive because I am studying for an upcoming exam. But I'm hoping to get back on track in a next couple of days.

2. What is coming up next?

I will keep working on both them until they are ready to be merged.

3. Did you get stuck anywhere?

Yes I was stuck with zip reformulations for a while but with the help of my mentor everything is back on track.

Blog Post #4

bablusahoo16@gmail.com (bksahu) — Mon, 22 Jul 2019 14:50:27 +0000

Hi, welcome to my 4th blog post !

Last time we saw how reformulations of `min` and `max` works. This time we will drive in how reformulation of `zip` (only in Py27) works. If I were to implement `zip` using Python 2 it were to look something like this:

def _zip(*iterables):
     # zip('ABCD', 'xy') --> Ax By
     # import ipdb; ipdb.set_trace()
     sentinel = object()
     iterators = [iter(it) for it in iterables]
     zipped = []
     for _ in xrange(len(iterators)):
         result = []
         for it in iterators:
             elem = next(it, sentinel)
             if elem is not sentinel:
                 result.append(elem)
         if len(result) == len(iterators):    
             zipped.append(tuple(result))
     return zipped

But on reformulation, some things need to be changed. We also have to take define new temp variables inside the scope and release it when we are done. One more major optimizations that we have to do is to use one loop instead to two loops as seen above. So, finally our reformulation pseudo-code will look something like this:

def _zip(a, b, c, ... ):
   # First assign, to preserve order of execution,
   # the arguments might be complex expressions.
   tmp_arg1 = a
   tmp_arg2 = b
   tmp_arg3 = c
   ...

   tmp_iter_1 = iter(tmp_arg1)
   tmp_iter_2 = iter(tmp_arg2)
   tmp_iter_3 = iter(tmp_arg3)
   ...

   # could be more
   tmp_result = []
   try:
       while 1:
           tmp_result.append(
               (
                    next(tmp_iter_1),
                    next(tmp_iter_2),
                    next(tmp_iter_3),
                    ...
               )
            )
      except StopIteration:
          pass

   return tmp_result

In this way we can optimize the zip nodes in Nuitka.

Thank you for reading :)

Optimizing "map" and "zip" - Weekly check in #4

bablusahoo16@gmail.com (bksahu) — Mon, 15 Jul 2019 14:08:00 +0000

1. What did you do this week?

This week I started up with optimizing "map" built-in but later realised that I first need to optimize "zip" as there is no support on Nuitka for it as "map" is dependent on it. Initially I implemented the C backend for Python2 "zip" function but after discussing with my mentor we decided to keep only C backend for Python3 "zip" and depend on re-formulation for Python2 "zip" built-in.

2. What is coming up next?

Next up I will optimize zip for Python 3 and then get back to optimize "map" built-in.

3. Did you get stuck anywhere?

Well, I wouldn't say I was stuck anywhere but I had a lot of discussions on developing an optimization approach for the built-in mentioned above.

Thanks!

Batakrishna

Blog post #3

bablusahoo16@gmail.com (bksahu) — Sun, 07 Jul 2019 13:37:48 +0000

Hello everyone! Welcome to my 3rd blog post.

So, I am very happy to inform that I have passed my first evaluations with flying colors and now working on phase 2.

I learned a new way of optimization for Nuitka that doesn't need any CPython code replication. The idea is very simple i.e to build on top of existing nodes on Nuitka. Consider the following example, below is the optimization for min() built-in

makeStatementConditional(

    condition=ExpressionComparisonGt(

        left=ExpressionTempVariableRef(

            variable=min_result_variable, source_ref=source_ref

        ),

        right=ExpressionTempVariableRef(

            variable=min_arg_variable, source_ref=source_ref

        ),

        source_ref=source_ref,

    ),

    yes_branch=StatementAssignmentVariable(

        variable=min_result_variable,

        source=ExpressionTempVariableRef(

            variable=min_arg_variable, source_ref=source_ref

        ),

        source_ref=source_ref,

    ),

    no_branch=None,

    source_ref=source_ref,

)

The above is equivalent to (in Python):

min_result_variable if min_result_variable > min_arg_variable else None

In this way we can optimize reusing existing nodes. Now the only difficulty is in the case when there is more than one argument. Such type of problems are handled in Nuitka using something called reformulation. So, by using reformulation our problem will decomposed in to a simpler problem. For example:

min(1, 2, 3) ---(reformulation)---> min(min(1, 2), 3)

Hence, in this way we can optimize in an efficient way.

Thank you for stopping by!
Batakrishna

Optimizing "max" and "min" - Weekly check in #3

bablusahoo16@gmail.com (bksahu) — Tue, 02 Jul 2019 14:56:13 +0000

1. What did you do this week?

Finally I was successfully able to optimize the max and min built-in with my new understanding of Nuitka internals. Apart from that did some refactoring for my algorithm for BUILTIN_ALL optimization. It is almost ready to be merged to the main code base.

2. What is coming up next?

This is I will spend time on fixing bugs if any in the "abs", "min" and "max" optimization. Also, I will start with a new built-in optimization.

3. Did you get stuck anywhere?

Nope, I didn't get stuck anywhere.

Blog post #2

bablusahoo16@gmail.com (bksahu) — Tue, 25 Jun 2019 06:25:37 +0000

Hello. Welcome to my second blog post.

The first evaluations have started. When I look back into last month, I realize that I have learnt a lot and grown into a better developer. So, I am glad that I'm a part of this program and organisation.

So, as discussed in the last blog post I am proud to inform that I have successfully created the IterationHandles module that take care of the iteration needs. And the best past about this module is that it is not limited to any() built-in optimization but can be reused in many other built-in optimizations as well. This module made my job very easy while optimizing "all" built-in.

abs() built-in optimization: While going through the original CPython implementations of this function I noticed that it is has a nb_absolute member which stores the absolute value of a variable, which means I had to manually add slots for every shapes (Eg. int, float). Now, this optimization needs a final touch up after review.

I am hoping to see those optimizations in next release.

Thank you for reading!

Batakrishna

Optimizing "abs" - Weekly Check-in #2

bablusahoo16@gmail.com (bksahu) — Wed, 19 Jun 2019 11:04:48 +0000

Welcome to my third weekly check-in!

1. What did you do this week?

This week I spent time on optimizing the "abs" built-in and understanding the "operators" module.

2. What is coming up next?

Next up, I will be working on adding support for "max" and "min" built-in after discussing potential optimizations with my mentors.

3. Did you get stuck anywhere?

Yes, I was stuck while implementing the "BUILTIN_ABS" CPython implementation for Nuitka. I have fixed it by doings the version checks.

Thank you for reading!

Blog post #1

bablusahoo16@gmail.com (bksahu) — Sun, 09 Jun 2019 05:49:34 +0000

Hello everyone! Welcome to my first blog post.

There are a total of 69 built-ins in Python 3 and 76 built-in in Python 2 out of which 27 built-ins are yet to be optimized in Nuitka. My aim is to optimize those missing built-ins in Nuitka. This will make Nuitka a lot faster while working with CPython built-ins.

The first built-in that I'm going to start with is "any" built-in. I have already implemented a generic algorithm for the optimization for cases like any([0] * 10000). But there is problem with this algorithm i.e it doesn't have support for Dict and Sets datatypes because they are non-indexable. A solution to problem is embracing the fact that python internally uses iter() and next() while working with loops. Using these we can easily iterate over the elements in the dictionary and set datatypes. The idea is to have a limit for the number of elements we will iterate through and try to predict the True value, for example any([0]) or any([0]*100) is always going to be False. In the same way any([0]*1000000) is also False but it's too expensive to iterate of all the million elements and find their truth value that's why we need to a limit. This solutions will also be helpful with future optimization that need to iterate through every element.

Thank you for reading :)

Optimizing "all" - Weekly Check-in #1

bablusahoo16@gmail.com (bksahu) — Sun, 02 Jun 2019 07:25:15 +0000

Welcome to my second weekly check-in!

1. What did you do this week?

optimized the "all" built-in
refractored the "IterationHandles" class

2. What is coming up next?

Next up, I will be working on adding support for "abs" built-in after discussing potential optimizations with my mentors.

3. Did you get stuck anywhere?

No nowhere because all built-in is almost similar to any built-in.

Thank you for reading!

Batakrishna Sahu

Welcome blog post

bablusahoo16@gmail.com (bksahu) — Tue, 28 May 2019 05:46:11 +0000

Hello, world. Welcome to my first of many upcoming blog posts. I will be working on optimizing Python 2 and Python 3 built-ins for Nuitka this summer.

Thank you for stopping by!

Batakrishna Sahu