Articles on joaosferreira's Blog

Final Week Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Mon, 31 Aug 2020 12:19:35 +0000

What did you do this week?

This week I concluded the PR for the random module by adding multimethods for the Generator class. This PR was the last one of my GSoC project. I also wrote the project's final report for my third and final evaluation where I included a short description of the project, links to the the code, a summary of the current state of the project, discussed future work, and enumerated challenges and learnings.

What will you do after GSoC has ended?

They say all good things come to an end and this blog post marks the end of my GSoC journey. Soon I will start working on my master's thesis and look for job opportunities in the industry. I also intend to keep working on unumpy and other open-source software projects.

Week 12 Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Tue, 25 Aug 2020 15:15:41 +0000

What did you do this week?

This week I started a new PR that adds multimethods for NumPy's random module. This continues the work started by one of my mentors by revising some multimethods and adding new ones as well as important classes like RandomState and Generator along with their methods. The multimethods added so far are manifold and so I won't extend the length of this blog post by enumerating them. You can however read the full list of multimethods in the PR link provided above. I also patched one ongoing PR that added multimethods for statistical functions by refactoring some default implementations. The defaults were redundantly using, in most but not all cases, a helper function for reducing the array argument's dimensions. This was brought to my attention by one of my mentors which resulted in a simple refactoring of the defaults.

What is coming up next?

The final week of GSoC is ahead of me and with that said now is the time to finish my project and write the final report. As for the first, this means mostly concluding the random module PR. If I have time I will also work on the documentation of unumpy's multimethods by linking it to NumPy's documentation through Sphinx. Although this might be an easy task, since this is my first time working with Sphinx I don't have an estimate for how long it will take me to do it. Ultimately, this might have to be done after GSoC has ended. More importantly, this upcoming week I want to focus on writing a good final report to showcase the work done during the program. This is of the utmost importance, as it is a necessary step in order to successfully pass the third and final evaluation.

Did you get stuck anywhere?

There were no blockers this week.

Week 11 Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Wed, 19 Aug 2020 13:58:31 +0000

What did you do this week?

This week's work was divided into two parts: fixing some issues in the backends and patching up older PRs. While working on the PR for indexing routines we noticed a bug in the backends that caused some classes to not be dispatched properly. What was happening was that all the classes added in the PR were calling NumPy classes regardless of the backend being used instead of calling the equivalent classes in the respective backends. This issue also touched on another problem that I encountered while preparing the new PR for the random module. The backends were also not dispatching the multimethods in modules correctly. This was apparent by the linalg module's tests which at the time were fixed to the NumPy backend. Because of this I created a new PR that fixed both of these issues and extended the linalg tests to other backends which allowed the PR for indexing routines to be patched and merged. I also worked on a previous PR that adds multimethods for statistical functions by adding default implementations to most of the reduction methods (e.g., median, mean, var).

What is coming up next?

This week I was expected to start a new PR that adds multimethods for NumPy's random module but as I talked about in the first part of this blog post, some issues had to be fixed first. Although this delayed the PR to the next week all work seems to be on schedule. If I have time in this upcoming week I will also start writing documentation for the unumpy project and my GSoC final report. In the latter I intend to add a short description of the work that was done, allow who reads it to easily find the code, summarize the current state of the project, discuss future work, and enumerated challenges and learnings.

Did you get stuck anywhere?

I was able to unblock from last week's default implementation of median with some help. Overall, the other defaults were done without many difficulties.

Week 10 Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Tue, 11 Aug 2020 15:16:06 +0000

What did you do this week?

This week I started a new PR that adds multimethods for statistical functions. The multimethods added are the following:

Order statistics

percentile
nanpercentile
quantile
nanquantile

Averages and variances

median
average
mean
nanmedian
nanmean
nanstd
nanvar

Correlating

corrcoef
correlate
cov

Histograms

histogram
histogram2d
histogramdd
bincount
histogram_bin_edges
digitize

As of now these new additions have the essential parts of a multimethod, both an argument extractor and replacer but are missing default implementations. I've also modified some of the argument replacers to support the out keyword making them more general purpose thus removing the need for some of the other more specific argument replacers.

What is coming up next?

I'll continue the PR started this week by working on default implementations for the simpler multimethods like median and mean. I will also start a new PR that adds multimethods for NumPy's random module that picks up on an older PR left off by one of my mentors. I've been wanting to work on these multimethods for a while but wasn't able to because of other project commitments. After discussing with my mentors we decided to change the work expected for the following week from the JAX backend implementation to the random module since this has a higher priority.

Did you get stuck anywhere?

Yes, working on the default implementation for median. Although this might be one of the easiest defaults from the multimethods in the current PR it has proven to be a bit of a challenge. The idea is to transverse the array along the given axes and apply a reduction function. Because of the array manipulations necessary to accomplish this and since I can't use item assignment the default is being more complicated than initial thought. More recently one of my mentors provided a general template for implementing it which might help me unblock. If I can do this other reduction multimethods' defaults should easily follow.

Week 9 Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Tue, 04 Aug 2020 00:43:32 +0000

What did you do this week?

This week I continued the PR started in the previous week by adding more multimethods for indexing routines. The multimethods added are the following:

Generating index arrays

c_
r_
s_

Indexing-like operations

take
take_along_axis
choose
diagonal
select

Inserting data into arrays

place
put
put_along_axis
putmask
fill_diagonal

Iterating over arrays

nditer
ndenumerate
ndindex
nested_iters
lib.Arrayterator

Most of the work developed during this week was adding default implementations to the new mutimethods. Also, this is the first PR that makes use of the overriding of unumpy's classes to implement some classes for indexing routines. Unfortunately, I wasn't able to start the PR that adds multimethods for Numpy's random module that I set out to do in the last blog post. Nevertheless, since this is considered extra work there are no delays in the timeline. I will try to do this PR in the following weeks.

What is coming up next?

I will start a new PR that adds mutimethods for statistic functions. The work for this following week might be more demanding since the PR is expected to last only one week and it covers a considerable amount of multimethods. Also, some of the default implementations might add an extra challenge. Let's see how it goes! I also need to patch up the PR for the indexing routines which should be almost done.

Did you get stuck anywhere?

I didn't get stuck this week.

Week 8 Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Wed, 29 Jul 2020 00:13:47 +0000

What did you do this week?

This week I started a new PR that adds multimethods for indexing routines. The multimethods added so far are the following:

Genenating index arrays

indices
ix_
ravel_multi_index
unravel_index
diag_indices
diag_indices_from
mask_indices
tril_indices
tril_indices_from
triu_indices
triu_indices_from

I also finished the two PRs from the previous week, one that adds multimethods for mathematical functions and another for functional programming routines. As mentioned in my last blog post, this week I also managed to merge another PR that picks up on work previously done by one of my mentors. This PR extended the overriding of classes to other backends, more specifically, to Dask and Sparse.

What is coming up next?

I will continue the PR started this week by adding more multimethods that target indexing routines. This should be mostly writing default implementations for the new multimethods. Depending on how this goes I might also start a new PR that picks up on work previously started by one of my mentors on adding multimethods for NumPy's random module.

Did you get stuck anywhere?

There were no blocks this week, only one test that I could not understand why it was failing. Turns out that uarray, the backend mechanism behind unumpy, was calling the default implementations after these failed in a given backend with the arguments previously marked for dispatching not being coerced into array types. This was fixed on the uarray side by one of my mentors by skipping this last call to the defaults under certain conditions.

Week 7 Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Tue, 21 Jul 2020 02:40:35 +0000

What did you do this week?

I started a new PR that adds multimethods for functional programming routines. The multimethods added are the following:

apply_along_axis
apply_over_axes
frompyfunc
piecewise

Other work done this week was mainly adding default implementations to a previous PR that adds multimethods for mathematical functions. I've also managed to finally pick up the work started by one of my mentors that adds an overriding class (this was discussed in previous blog posts). This was done by adding the function overriden_class to both Dask and Sparse backends.

What is coming up next?

Next week I will be starting a new PR that adds multimethods for indexing routines and also patch up the ongoing PRs from this week. I will also try to open a new PR that branches off from this one and adds the work done this week on the overriding class. When this is merged it will enable me to add some mutlimethods that I've been holding off like nan_to_num and real_if_close.

Did you get stuck anywhere?

Trying to understand NumPy's implementations for some mathematical functions like unwrap and interp to implement their defaults was a bit difficult at first but I managed to self unblock. I haven't previously looked at NumPy's code for help on writing defaults but this week I found it to be very helpful although it can also be confusing sometimes (and can feel like cheating). I still didn't fully understand unwraps's implementation though.

Week 6 Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Wed, 15 Jul 2020 00:54:56 +0000

What did you do this week?

I started a new PR that adds multimethods for mathematical functions. The multimethods that were added are the following:

Trigonometric functions

degrees
radians
unwrap

Rounding

around (and its aliase round_)
fix

Sums, products, differences

cumprod
cumsum
nancumprod
nancumsum
ediff1d
cross
trapz

Other special functions

i0
sinc

Arithmetic operations

float_power

Handling complex numbers

angle
real
imag
conjugate

Miscellaneous

convolve
clip
nan_to_num
real_if_close
interp

I've also concluded the PR on array manipulation routines from previous weeks which still had some unresolved issues. It turns out that this week's work was mostly spent patching up the previous PR and less working on the new one unfortunately. This ended up delaying the new PR a bit but since the new one already added quite a few multimethods the schedule is unaffected.

What is coming up next?

Continuing the new PR by adding default implementations to the appropriate multimethods and starting a new PR that adds multimethods for functional programming routines. I already did some offline prototyping of default implementations for mathematical functions, the more easy ones at least. The more difficult ones (e.g., unwrap, convolve and interp) should be the focus of that part of next week's work.

Did you get stuck anywhere?

There were no major blocks this week. The only issue was the somewhat precedence that the older PR had which prevented the newer one to advance. For example, both PRs had failing tests in the XND backend so this backend was removed (the project is not maintained anymore) in the older PR. Since this has now been merged the newer PR can now receive those changes that fix its failing tests.

Week 5 Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Tue, 07 Jul 2020 15:33:04 +0000

What did you do this week?

I continued the PR started in the previous week by adding more multimethods for array manipulation. The following multimethods were added:

Tiling arrays

tile
repeat

Adding and removing elements

delete
insert
append
resize
trim_zeros

Rearranging elements

flip
fliplr
flipud
reshape
roll
rot90

Most of the work was adding default implementations for the above multimethods which relied heavily on array slicing. Contrary to what I mentioned in my last blog post, not much time was dedicated to writing overriden_class for other backends so I will try to compensate in the next weeks.

What is coming up next?

As described in my proposal's timeline I'll be starting a new PR that adds multimethods for mathematical functions. This will be the focus of this week's work as some of these multimethods are also needed by sangyx, another GSoC student working on uarray. He is working on the udiff library that uses uarray and unumpy for automatic differentiation. To avoid getting in his way I will try to finish this PR as soon as possible.

Did you get stuck anywhere?

There were times when I didn't know exactly how to implement a specific default but this was easily overcome with the help of my mentors who would point me in the right direction. Asking for help sooner rather than later has proven to be invaluable. Looking back I think there were no major blocks this week.

Week 4 Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Tue, 30 Jun 2020 16:08:50 +0000

What did you do this week?

I started a PR that adds multimethods for array manipulation routines. I'll name the multimethods according to the NumPy docs sectioning:

Basic operations

copyto

Changing number of dimensions

expand_dims
squeeze

Changing kind of array

asfarray
asfortranarray
asarray_chkfinite
require

Joining arrays

dstack

Splitting arrays

split
array_split
dsplit
hsplit
vsplit

As mentioned in my last blog post, this week I also started reviewing a PR that implements metaclasses from which classes in unumpy instantiate. They are used to override these classes with ones from the backend being used through a property method called overriden_class. Currently only the NumPy backend has this method but I've been trying to implement it in other backends as well.

What is coming up next?

The current PR should last one more week since I'll continue to add more multimethods for array manipulation routines. I will also be working on adding overriden_class to other backends as I've been doing this past week.

Did you get stuck anywhere?

I think the only place I got stuck was trying to implement overriden_class for other backends. To be more specific, I tried implementing it in the Dask backend first and foremost, however, this backend is different since it uses another backend internally. From my understanding this causes that some classes might have to be overridden by the inner backend and others by Dask itself. With that said, I might need help later on with this issue. In general, I feel that this metaclasses feature has been the most challenging part of my project so far. Although this wasn't initially included in my proposal and can be considered extra work I welcome the challenge and hope that my mentors keep entrusting me more of these features. Also, given that the semester is almost ending I start having more free time on my hands to tackle these problems which is what I want.

Week 3 Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Mon, 22 Jun 2020 15:20:37 +0000

What did you do this week?

I continued the work started in last week's PR by adding more multimethods on array creation:

Numerical ranges:

geomspace

Building matrices:

diagflat
tri
tril
triu
vander

These additions conclude the mentioned PR which at the time of writing this is almost ready to be merged. I also tried to implement a metaclass type for classes in unumpy.

What is coming up next?

I'll be starting a new PR that intendes to expand unumpy's coverage of array manipulation routines. The PR is expected to last two weeks given the extensive and diverse nature of the multimethods covered. Also, following the discussion about the inclusion of a metaclass type from which other classes in unumpy instantiate, it's expected that I review a PR by one of my mentors on this. This is a good opportunity for me to start getting used to reviewing PRs. As a note, I think the workload in the next two weeks might increase a bit but as college assignments start to decrease a good balance between the two should stay constant.

Did you get stuck anywhere?

The biggest challenge this week was trying to implement the metaclass type. Although I was given some good pointers on how to proceed with this I still faced some problems on doing this correctly. Creating a metaclass with the added difficulty of writing multimethods for its methods was a bit overwhelming. I spent almost an entire day on this and got little results. Looking back I should have asked for help which would surely have helped me unblock. I'll be sure to keep that in mind next time. With that said, I'm still learning the ropes but I already feel that my Python skills are improving significantly.

Week 2 Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Mon, 15 Jun 2020 16:55:12 +0000

What did you do this week?

This week I started a PR that further adds multimethods for array creation routines. I'll name the additions according to the NumPy docs sectioning:

Ones and zeros:

identity
ones_like
zeros_like
full_like

From existing data:

asanyarray
ascontiguousarray
copy
frombuffer
fromfile
fromfunction
fromiter
fromstring
loadtxt

During this week I also started learning about metaclasses in Python. This was encouraged by my mentors as a way to level up my Python skills so that I can tackle more complicated features in unumpy. This was mostly done by reading documentation, posts and articles about the subject and watching one of my mentors code some examples.

What is coming up next?

The current PR will be stretched to this upcoming week where I'll be covering more multimethods for these routines. Also, as briefly discussed in my previous blog post, I will eventually be picking up a PR that adds multimethods for the random module. However, as stated in this issue, classes in unumpy like np.random.RandomState could benefit from a custom metaclass. Given the importance of this feature for current and future implementations additional work during this week will be focused on the creation of this metaclass.

Did you get stuck anywhere?

There were some difficulties writing tests for some of these routines as there were a lot of errors that I didn't understand at first. Going over each one separately and writing why I thought they were failing actually helped in fixing some of them. At the time of writing this they are almost all resolved with only one exception.

Week 1 Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Mon, 08 Jun 2020 00:50:55 +0000

What did you do this week?

I started a new PR that adds mathematical constants and mutlimethods for logic routines. These included:

Math constants:

euler_gamma
nan and aliases
inf and aliases
NINF
PZERO
NZERO
newaxis

Logic functions:

isposinf
isneginf
iscomplex
iscomplexobj
isreal
isrealobj
isscalar
array_equal
array_equiv

I've also continued to work on the PR started during community bonding. This one also originated a new issue.

What is coming up next?

I'll be adding more multimethods, this time addressing routines for array creation. I've already done two multimethods on array creation when I was applying for GSoC which might help this coming week. I'm also thinking about picking up another PR left off by one of my mentors. The one about NumPy's random module seems interesting and challenging. However, this still has to be discussed with my mentors since this was not part of my proposal and it also depends on the available time I'll have. Although I have an exam and a project assignment to deliver I hope to keep a good rhythm going during this week. Let's see how it goes!

Did you get stuck anywhere?

I haven't had any major blocks this week. The default implementation for norm in the Linear Algebra module was the toughest as I was trying to implement a simple solution that would take into account all the corner cases. I realized that sometimes an elegant solution is not easily attainable and that correctness is more important. The result was not that bad I hope.

This week was also the start of my weekly meetings with my mentors. We discussed my PRs and other things about the project which was quite helpful. I really appreciate their availability and commitment.

Community Bonding Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Fri, 29 May 2020 15:21:35 +0000

What did you do during this period?

I had an onboarding meeting with my mentors where we got to know each other a bit better. They advised me to play around with uarray and unumpy without any goal in mind which I found to be a very good advice. I played a bit with special methods by implementing a simple Vector2D class and used the code in this notebook with some print statements to understand better the protocols and how they are called. I wanted to start earlier on my project so I took over a PR from one of my mentors which adds multimethods for the linalg module.

What is coming up next?

I'm going to continue the PR that I have been working on since it still isn't finished and I will also follow the proposed timeline and start adding multimethods for other routines like checking class equality in array elements. Some mathematical constants and their aliases are also missing so I will be adding these too and probably refactoring the existing ones into classes. This week marks the end of my college classes but I still have some assignments and exams coming up in the following weeks so there's a lot of work ahead of me to proper balance both university studies and GSoC but I wouldn't have it other way.

Did you get stuck anywhere?

I consider the PR that I started working during this period to be a challenging one since some mathematical intuition is needed to translate Linear Algebra routines into proper functions. Things like decomposition of a matrix into eigenvalues and eigenvectors to calculate its n-th power is something I'm not too familiar with specially in a programming context. With that said there hasn't been a roadblock for me up until now and usually I can wrap my head around these concepts in half a day. It should be noted that mentor help plays a huge part in this as they frequently give me very good advices. Despite that I usually think a lot before doing a commit to make sure that what I'm pushing is correct I notice that I still can't avoid some mistakes, even ones that should be obvious to me. I guess these mistakes are normal and they are corrected soon after so no harm's done but I'm training myself to not do them as often.