Articles on plant99's Blog

weeklyCheckIn[12]

shivashispadhi@gmail.com (plant99) — Tue, 20 Aug 2019 05:43:55 +0000

What did I do this week?

Last week, me and my mentor started testing mscolab. At first, I exposed my port through ngrok. But the network I/O were extremely slow. I couldn't demonstrate all of mscolab's features. Then we setup an AWS instance with mscolab server deployed. We could test and improve on many of its features. Some things I worked on this week:

Adding support to use Postgres
Refactoring tests, so that the library code is independent of test configurations.
Adding newly created projects to project-list, in real time
Add newly permitted collaborators to users-list in project window, in real time.
Fix autosave inconsistency when projects are switched

What's coming up next?

There are very few development tasks remaining. Since this is the last week of GSoC, I'd be working on documenting the code I've written, externally. That would take most of my time this week.

Did I get stuck anywhere?

Yes, I got stuck in a bad place for some-time this week. While setting up Postgres, I had to manually autoincrement sequence to assign 'id' to tables, if I explicitly insert rows. I have documented the issue and solution in this blogpost if you're interested in learning more about it.

weeklyCheckIn[11]

shivashispadhi@gmail.com (plant99) — Mon, 12 Aug 2019 01:09:52 +0000

What did I do this week?

Earlier this week, I worked on a number of major refactoring listed below.

displaying user and project details in project window
send service message for each 'file-save'
add user/add project features added
refactoring test/real data setup
disable action buttons for non-permitted users.

Some parts of it have been merged with my branch in origin repository. I'm working on setting up the project for Postgres for internal usage, and disabling autosave for non-admin users. After that, I'm making a PR

What's coming up next?

Next up, are the internal testing of mscolab at my mentor's workplace. Then we can come up with bugs, and features to work on for the next and final week of development cycle.

Did I get stuck anywhere?

No, nothing bothered me for long this week. I did however come up with a design dilemma, where I had to import variables for tests as different names, and manually detect if tests are running, and then set the variable names. However, the app can't be configured to use TEST_* params. We have to set a test environment to change all those params to test params.

weeklyCheckIn[10]

shivashispadhi@gmail.com (plant99) — Mon, 05 Aug 2019 07:57:24 +0000

What did I do this week?

As discussed in the earlier blog, I worked on tests for GUI to 'checkout' to a previous version of flightpath. After creating a pull request, I parallelly started working on GUI to add users and projects. I also got the pull request reviewed by my mentor.

What's coming up next?

This week, I have to finish up on modifications suggested by my mentor. After first request gets merged, I have to work on displaying user's, project's details in chat window, and active view windows. After that, I'd have to fix an issue due to which table view can't be created more than once for mscolab. Before making the second pull request, I also have to write tests for GUI part related to creation of users and projects.

Did I get stuck anywhere?

I was stuck for a very short while trying to decide if QInputDialog works with multiple inputs, then I designed my own QLabel for my usage.

weeklyCheckIn[9]

shivashispadhi@gmail.com (plant99) — Mon, 29 Jul 2019 16:40:21 +0000

What did I do this week?

After completing tests and features for mscolab's project window and getting the PR merged, I started working on a more tricky part of the project. Introducing version control to mscolab server's storage. As explained on the previous blog, this is done to introduce some features to project. Mainly the following

undo/redo a particular change(commit) - TO BE IMPLEMENTED
checkout to a particular commit - IMPLEMENTED

I used 'gitpython' library to handle git related functionalities. For now, the commit hash is stored in 'Change' model. It might seem like it's redundant, which is true, but we'll work on this later to find a way to store every detail in the commit, to scrap the 'Change' model.

What's coming up next?

The aforementioned features are already implemented, I have to write tests before I make a PR. Once the PR is created, I will parallelly start working on GUI to create projects, users, and displaying user and project details in project window. After this, my mentor has planned a testing of added features in development server.

Did I get stuck anywhere?

There were two major places where I got stuck.

While introducing version control to file storage, git commit would not work if this file is opened in 'w'(write mode) in any part of program. I had to close the opened reference, so the most recent changes are committed.
I couldn't find a reason to why this happened, but while opening a 'question' with PyQt's MessageBox in an event handler of QListWidgetItem's activate event, it opens multiple times if the 'QListWidget' is reloaded. Maybe because if an item is activated, and another item is added to the list in the event handler, the new item in that index is activated again, in a loop. I avoided it by setting an 'active_change_id' to keep track of the activated item's id and then manually asking an user to press a button to checkout to that change.

weeklyCheckIn[8]

shivashispadhi@gmail.com (plant99) — Sun, 21 Jul 2019 05:38:34 +0000

What did I do this week?

As it was planned the previous week, I worked on tests for GUI of mscolab's main window. Here is the link to pull-request with the tests and autosave synchronization feature.

Next important thing I worked on is the project window, which has collaborators-listing, chat feature, and 'change' listings. I've finished almost all of the features, with some finishing touches remaining for UX improvements(colorize the UI). I'll make a pull-request for this module after writing some features on Tuesday(2019-07-24). If you want to test features, please look into 'GSOC19-plant99' branch in my fork.

What's coming up next?

Next up, is writing tests for features of project window, and improving the GUI. Then this PR's review might take upto 1.5 working days, considering the amount of new changes and code introduced.

If there's still time, the next feature is to introduce version control in mscolab server's storage. This essentially means, moving from a 'file' based storage to a 'project' based storage. Why are we doing this? To introduce undo/redo functionality, basically the whole 'mss project' would be a git enabled directory with a linear-commit history. If one wants to undo a change, we checkout to the commit before, and write this text to the file, commit in front of HEAD. Thus 'undo'ing the changes introduced after that commit.

Did I get stuck anywhere?

I had a little difficult time figuring out the Widgets I had to use to build the GUI. I ended up using QListWidgets, even for chat-log, and change-log. I doubt if it's the right tool to use. But since it has not obstructed development of other modules, it is not a very big issue, and I will fix this later.

Non-technical note: This week was where every bit of work from the past two months were bundled together and put to use. I realized the importance of writing tests, especially for APIs. I could reuse the clients I wrote for tests, and could get done with things quicker. I'd recommend writing extensive tests, I know I still am missing out on more benefits of them. The note is to express a tiny bit of overjoy I feel when I see everything working nice, and makes me more confident about finishing up for the first release of mscolab.

weeklyCheckIn[7]

shivashispadhi@gmail.com (plant99) — Sun, 14 Jul 2019 12:41:13 +0000

What did I do this week?

As per last week's plan. I completed implementation of 'projects-list' part of the UI. The additional features were,

spawning top/side/table views on click of buttons
write socket handlers for client, to implement file-save, and file-load
an 'auto-save' feature which can be activated by admin/creator of a project. This helps to reflect changes on everyone's local version as they happen, instantly. In a non-autosave mode, everyone works on a local desktop on the project. (Post GSoC, we are planning to improve it for situations where some collaborators can be in autosave mode, while some not.)

What's coming up next?

The next part is to write unit tests for the functions developed in the past 1.5 weeks. After this, I have to get them reviewed in a pull request along with the 'autosave synchronization' feature.
With that concluded, I have planned to work on the 'project window' which would include

users' list
chat
change log

The project window would take about 6-7 days to get developed, including unit tests.

Did I get stuck anywhere?

Not for a very long time in a problem. But I came across small 'gotcha's like how Qt requires a child to be bound to at least one element in self.window as its parent. I had a hard time using super for parent constructors as well, because I didn't know how to in Python3.

Link to the PR reviewed this week: https://bitbucket.org/wxmetvis/mss/pull-requests/657/upto-ui-1-mscolab-dashboard/diff

weeklyCheckIn[6]

shivashispadhi@gmail.com (plant99) — Mon, 08 Jul 2019 06:47:28 +0000

What did I do this week?

This week started with filing a PR for integration tests and other refactoring. Then I assigned 1.5 days to design the software architecture for `msui` it's a big software and is a result of 7-8 results of development. I had to make sure my design doesn't break the policy of mss software, and is modular and pluggable. I have worked out a significant amount of design and basic implementation, you can find an active PR on this here.

What's coming up next?

For the next week, I'd still be working on working with mscolab landing window. The following things have to be done in the upcoming week.

consistent opening of topview, sideview, tableview from projects listing
modify 'file save' to handle call differently if mscolab is activated (it has to send the file content to remote)
write socket connection handler for UI part

Did I get stuck anywhere?

I sure faced some challenges while designing the architecture for mscolab's UI. I had to make decisions regarding functions, connectors' definition, available classes and how they can be extended to fit needs of mscolab. After implementing a basic prototype, I'm more confident about the design now.

weeklyCheckIn[5]

shivashispadhi@gmail.com (plant99) — Sun, 30 Jun 2019 14:03:33 +0000

What did I do this week?

For the past week I was involved in two major tasks. They are:

Introducing the 'Change' module, this is an addon, which helps keep track of 'diffs' of old and new version of ftml file, for analytics purpose. In future, post gsoc, this module is to be replaced by a VCS which would support advanced features like undo and redo.
Writing integration tests for APIs on HTTP endpoints, and some other refactoring which removed all occurences of executing `python server.py` with a subprocess.exec.

The 1st task was finished and reviewed fully, I've started a PR for second task earlier today.

What's coming up next?

Next up, I'll discuss the changes required in this PR with Reimar. Once merged, 50% of the work is done.
For the next part, I have to work on GUI for this server. The gui would be a part of existing msui. Before starting to write code, I'm planning to write a proper document concluding the architectural design of the new code, since it has to properly be integrated with msui.

Did I get stuck anywhere?

No, the week's work was mostly writing tests, so didn't get stuck. But I reached a conclusion with a feature request and put 'wontfix' status. This is an excerpt from our slack group. image

weeklyCheckIn[4]

shivashispadhi@gmail.com (plant99) — Mon, 24 Jun 2019 06:51:34 +0000

What did I do this week?

Last week, I've been working on building the File-based event handlers, on top of the existing socket layer. Specifically, I introduced a FileManager class, which handled I/O with PyFilesystem2 library. Some things which I learnt while my mentor reviewed the PR involving this work is, we have to make sure this application runs on all infrastructure. So you might want to avoid some programming techniques involving

bash commands' execution from code directly is wrong, this might fail for windows/iOS
while chosing a library to work on, make sure it's compatible with a broad range of environments

I learnt to avoid using 'os' while I can use 'fs', because the latter is more generic and 'os' based functions won't work when filesystems like DAV are used.

What's coming up next?

While I proposed to work on refactoring and integration tests this week. I've talked with my mentor and swapped this task with that of the next week i.e development of 'Change' related handlers. Changes are saved for each 'save' just like commit but with limited functionalities. Two major parts of this work are to

develop the models, and api for storing and fetching changes
store the change id in tags, for easy access. For example, an user changed a waypoint, this change id is stored inside waypoint.

Later, if we add version control to storage, the extra benefit we get is we can undo, redo changes.

Did I get stuck anywhere?

No, this week was fairly smooth. Though I stopped and learnt a bit more about decorators and their usage, when Reimar asked to refactor a bit where I authorized a request. There were a lot of scope for reuse for 2-3 lines repeated for all API end-points.

Link to this week's PR

weeklyCheckIn[3]

shivashispadhi@gmail.com (plant99) — Mon, 17 Jun 2019 06:11:52 +0000

What did I do this week?

Last week, I've been working on integrating sockets, and build the 'chat' functionality on top of that. This involved, writing event handlers for chat events. The most interesting part was writing tests for this module, I've never written tests for real-time applications. Plus this application, has some additional complexity like permissions etc., which combinedly put a challenge to work on. I successfully got my code reviewed and merged, upto chat implementations. I also worked on integrating Sqlite as an alternative to MySQL database.

What's coming up next?

This week, I'm starting to work on File based handlers. To give a better idea, this project facilitates file synchronization across systems. In this part, I'm trying to tap change events and change the master database. Then I'd emit an event to all users that the file has been changed. To better handle this change, next to next week I'll work on the Change module(more on this on my proposal).

Did I get stuck anywhere?

During choosing library for implementing socket clients, to write tests, (Note that we'd use the same client to implement socket client for user interface) I got stuck for a day while trying to figure out the right one. The famous one 'python-socketio' had different distributions for client and server usage. It was hard to maintain the versions with 'flask-socketio' and 'python-socketio[client]' considering we were use conda-forge to install and distribute apps. For a while, I started writing tests with 'socketio-client' library. Me and my mentor, after a bit of discussion decided to use python-socketio.

weeklycheckIn[2]

shivashispadhi@gmail.com (plant99) — Sun, 09 Jun 2019 06:07:22 +0000

What did I do this week?

Past week, I've worked on restructuring the project when I introduced

- flask-sqlalchemy: a library to manage an ORM of the database and app

- token based authentication: I moved the project from an email-password authentication to a token-based authentication. I got a pull-repository of this changes reviewed and merged to my branch in origin repository.

Wrote tests for the user related routes. After this, I started working on sockets module, in a new class in the file `sockets_manager.py`.

What is coming up next?

I have been writing code to test sockets. It's my first time testing a socket-based application, so I am quite excited about it. After I get my code up to sockets reviewed, I will be coding more on modules facilitating message I/O on top of the socket layer.

Did I get stuck anywhere?

I got stuck in multiple places, there were a few circular dependencies issues I had to face which led me to restructure my app, during usage of flask-sqlalchemy, and sockets. Besides this, I realized that I can't use flask-socketio, and python-socketio for the same app due to a dependency issue as of now. I'll look more into that with my mentor later. For now, I am using a different library to implement a websocket client.

weeklyCheckIn[1]

shivashispadhi@gmail.com (plant99) — Mon, 03 Jun 2019 04:20:58 +0000

What did I do this week?

As planned, I set up the mscolab project. I decided to use 2 libraries to handle web-sockets, python-socketio for client side connection handling and a more specific flask-socketio from the same author, to develop an integrated socket server with Flask. I created migrations for the schemas defined, and setup basic authentication handlers. To check the socket connections, I wrote a html and another python client. I have successfully developed my proposed functionalities till 06/05/2019.

What is coming up next?

For the first part of my next week, i.e 4th and 5th I'd be working on cleaning of my code, restructuring it, and integrating tests for the same. I have asked my mentor to conduct my first review on 5th of June. After that, I'll be working more on socket APIs and handlers, some major functionalities to be developed and tested are

rooms: for each project, there'd be a room. For each change emitted to the room, it has to be sent to users in that room.
Besides that functionality, the addition and removal of users in and out of the room should be done.
events the events should be classified and developed to facilitate chat/file i/o functionalities

Did I get stuck anywhere?

Since I am new to Flask, I haven't been able to structure the database interaction of the app. For now, I am using a Mysql library which you can use to write direct SQL queries. As a part of refactoring in the next two days, I am working on integrating an ORM. The authentication part is also very basic, I will look again on improve it before 5th.

weeklyCheckIn[0]

shivashispadhi@gmail.com (plant99) — Mon, 27 May 2019 02:25:40 +0000

Hi, I am Shivashis. I will be contributing to MSS during GSoC'19. This is my first blog of the series.

What did I do this week?

I was looking more into Flask-socketio library which I would need during my project. Also, contributed some code towards 1.8.0 release of MSS software.

What is coming up next?

Next week, I would work on User and authentication related APIs of mscolab. More about it in an upcoming blog.

Did I get stuck anywhere?

Last week, I was mainly working on tested code functioning as tutorials. So I didn't come across any non-trivial blockers.