Articles on shivamshan's Blog

Blog Post #6 : Structuring Pygame Lectures

shivam.stpaulsdarjeeling@gmail.com (shivamshan) — Mon, 22 Aug 2022 18:21:12 +0000

I spent the 6th week structuring the Pygame lectures using the POC I provided before. My aim was to develop the lectures in a way that was fun along with being informative.

Blog Post #5 : E2E Playwright Tests

shivam.stpaulsdarjeeling@gmail.com (shivamshan) — Sat, 06 Aug 2022 10:56:54 +0000

In this post I will explain in detail the process of writing E2E(end-to-end) tests and how they they ensure the proper working of this course.

Weekly Check-In #5

shivam.stpaulsdarjeeling@gmail.com (shivamshan) — Sat, 06 Aug 2022 10:54:48 +0000

This week was spent on one of the most important aspects of code development, testing. I spent the 5th week writing code that tested whether the previous code I wrote worked without any errors.

Blog Post #4 : Más conceptos de sql

shivam.stpaulsdarjeeling@gmail.com (shivamshan) — Sat, 06 Aug 2022 10:49:27 +0000

This blog covers in detail the advanced concepts taught in the third lecture and all the necessary changes done to the SQlite wrapper to allow this to work.

Weekly Check-In #4

shivam.stpaulsdarjeeling@gmail.com (shivamshan) — Sat, 06 Aug 2022 10:47:45 +0000

Since it's creation, the sql section of the code had only two lectures. In a discussion with my mentors, we decided that we should add one more lecture to this section, teaching more advanced sql concepts. I spent the 4th week sorting and choosing between the concepts that could be taught while maintaining a good resemblance with the SQlite code structure.

Blog Post #3 : Conceptos básicos de SQL

shivam.stpaulsdarjeeling@gmail.com (shivamshan) — Sat, 06 Aug 2022 10:38:58 +0000

In this blog I will explain the details of implementing the second sql lecture.

Weekly Check-In #3 : SQL lecture no. 2

shivam.stpaulsdarjeeling@gmail.com (shivamshan) — Sat, 06 Aug 2022 10:28:13 +0000

In week no. 3 I dove deep into the SQL lectures, trying to make them as interactive as possible and making the learning experience more similar to that of SQlite.

Week 2 : The TWP42 Conundrum

shivam.stpaulsdarjeeling@gmail.com (shivamshan) — Wed, 29 Jun 2022 13:05:19 +0000

This week I will tell you how we finally concluded a problem that had been pending since the last GSoC. It took the combined effort of my mentor, me and Angela(contributor who worked on this problem under GSoC 2021) to put this problem to bed. The problem was very simple😉. We needed to make the TWP42: Bases de Datos lecture interactive. These set of lectures aim at teaching basic SQL concepts to the students. The core problem with this was the same as with the pygame lecture. The skulpt interpreter did not support the sqlite3 library which was to be used to teach these lectures. In the rest of the blog I will try to explain the initial approaches made by Angela last year and the final approach that we came up with to tackle the problem.

Let me take you back

Back in August, 2021, Angela had 3 approaches to solve this problem. All these approaches were good in their own way but came with their own problems. I have listed the 3 approaches here with their problems below in Angela's own words:

Using the activecode SQL language from Runestone: SQL activecode approach #185
Some configuration to use this SQL tool from Runestone must be done, because when you try to access the file surf.db, a CORS error shows up in console and nothing is shown. There might be something related with the book not being served by Runestone because in this error it tries to access to the db from an URL in runestone site: /runestone/books/published/PyZombis/_static/sql-wasm.wasm asuming that the book is published.
Using sql.js library from javascript: Refactor lecture TWP42: Bases de datos (sql.js approach) #184
This other option loads the database using the .. raw:html instruction from runestone.
This is the best attempt since you need to find a way to access to the global javascript variable db from the brython activecode block. Enter this PR for more detail, but if you call db in browser console you can see the database.
With indexedDB: Refactor Lecture TWP42: Bases de datos (indexedDB approach) #171
This aproach works, but the syntax changes a lot from the original lecture where sqlite3 was used.

Out of all these approaches, the second one(one using sql.js) seemed the most solvable to me as there was one problem with it: inability to access the global databse variable from within the activecode block. Although the variable was global, accessing it from within the activecode block was giving problems. As it turns out the solution to this was just a line of code 🤡.

My initial approach and the sqlite3 wrapper

The path to the one line solution to our problem was long. First, I wrote a sqlite3 wrapper that used sql.js at its core to process and execute the queries. Since the database variable was not accessible, I had to pass the queries using attributes of the script tag. To achieve this I created a mutation observer that reloaded the contents of the script with id "create-db-script", anytime the attributes of this particular tag was changed. This accounted for multiple queries to be executed. However handling COMMIT and ROLLBACK commands was another problem, as they required knowledge of the previous command and their results.

The following code explains my method:

<script id="create-db-script" ,="" query="" result="" flag="0" error="" first_exec_flag="">
  config = {locateFile: filename => `https://cdnjs.cloudflare.com/ajax/libs/sql.js/1.5.0/sql-wasm.wasm`}
   // a fuction is created that returns a promise
   // this function can be called everytime the query is executed
   function getData(config){
     return initSqlJs(config).then(function(SQL){
           if(document.getElementById("create-db-script").getAttribute("first_exec_flag") == "true"){
              console.log("first_exec_flag is true")
              //Create the database
              window.db = new SQL.Database();
              //Create table surfers
              db.run("CREATE TABLE surfers (id, nombre, pais, media, estilo, edad)");

              //Insert records
              db.run("INSERT INTO surfers (id, nombre, pais, media, estilo, edad) VALUES (101, \"Johnny 'wave-boy' Jones\",\"USA\",8.32,\"Fish\",21)");
              db.run("INSERT INTO surfers (id, nombre, pais, media, estilo, edad) VALUES (102, \"Juan Martino\",\"Spain\",9.01,\"Gun\",36)");
              db.run("INSERT INTO surfers (id, nombre, pais, media, estilo, edad) VALUES (103, \"Joseph 'smitty' Smyth\",\"USA\",8.85,\"Cruizer\",18)");
              db.run("INSERT INTO surfers (id, nombre, pais, media, estilo, edad) VALUES (104, \"Stacey O'Neill\",\"Ireland\",8.91,\"Malibu\",22)");
              db.run("INSERT INTO surfers (id, nombre, pais, media, estilo, edad) VALUES (105, \"Aideen 'board babe' Wu\",\"Japan\",8.65,\"Fish\",24)");
              db.run("INSERT INTO surfers (id, nombre, pais, media, estilo, edad) VALUES (106, \"Zack 'bonnie-lad' MacFadden\",\"Scotland\",7.82,\"Thruster\",26)");
              db.run("INSERT INTO surfers (id, nombre, pais, media, estilo, edad) VALUES (107, \"Aaron Valentino\",\"Italy\",8.98,\"Gun\",19)");
           }
           
           // the below code sets the flag and query attributes of the script tag depending on the result of the query
           // this is done to avoid the need for a callback function
           // the query attribute is set to the query string
           // the result attribute is set to the result of the query
           // the flag attribute is set to 0 initially
           // the flag attribute is set to "data found" if the query is successful and the result is not empty
           // the flag attribute is set to "no data" if the query is successful and the result is empty
           // the flag attribute is set to "query error" if the query is unsuccessful
           try{
           res = db.exec(document.getElementById("create-db-script").getAttribute("query"));
           console.log(res);
           } catch(e) {
              console.log(e);
              document.getElementById("create-db-script").setAttribute("error",e);
              document.getElementById("create-db-script").setAttribute("flag","query error flag");
              document.getElementById("create-db-script").setAttribute("result",JSON.stringify("query error"));
              res=undefined;
              console.log(document.getElementById("create-db-script").getAttribute("flag"));
              console.log(document.getElementById("create-db-script").getAttribute("result"));
           }
           
           if(res !== undefined && res.length > 0 && res[0].values.length > 0) {
              document.getElementById("create-db-script").setAttribute("result",JSON.stringify(res));        
              document.getElementById("create-db-script").setAttribute("flag","data found");
              console.log(document.getElementById("create-db-script").getAttribute("flag"));
              console.log(document.getElementById("create-db-script").getAttribute("result"));
           }
           else if(res.length==0){
              document.getElementById("create-db-script").setAttribute("flag","no data");
              document.getElementById("create-db-script").setAttribute("result",JSON.stringify("no data"));
              console.log(document.getElementById("create-db-script").getAttribute("flag"));
              console.log(document.getElementById("create-db-script").getAttribute("result"));
           }

        });
  }

  let coun = 0;
  // creating a mutation observer function to observe changes in attributes of script with id "create-db-script"
  var observer = new MutationObserver(function(mutations) {
  mutations.forEach(function(mutation) {
     if (mutation.type === "attributes") {
           if(coun == 0){
              console.log("first mutation");
              coun++;
              document.getElementById("create-db-script").setAttribute("first_exec_flag","true");
           }
           else{
              console.log("next mutation");
              document.getElementById("create-db-script").setAttribute("first_exec_flag","false");
           }                  
             getData(config);
     }
  });
  });
  // the observer below only observes the attribute "query" of the script tag with id "create-db-script"
  // if the query attribute is changed, the function getData is called
  observer.observe(document.querySelector('#create-db-script'), {
     attributes: true ,
     attributeFilter: ['query']
  });

</script>

After this I designed a basic sqlite3 wrapper around sql.js that could mimic the basic functionalities of the original package:

import document, time, json

def formato_datos(data):
   """
   esta función formatea el resultado de sql.js para que coincida con el formato de salida de sqlite3
   params -> data: diccionario con los datos a formatear
   """
   final_list = []
   columns = data['columns']          
   values = data['values']

   for v in values:
       Dict = {}
       for c,d in zip(columns,v):
           Dict[c] = d
       final_list.append(Dict)
   
   return final_list

# creando un contenedor para imitar el módulo sqlite3
# actualmente solo algunas funciones como connect, execute, fetchall son imitados
class Cursor:
   def __init__(self):
      self.prev_query = """"""
      self.curr_query = """"""
      self.prev_result = None
   def execute(self, query):
      query = query.strip()
      if(query[-1] != ";"):
         query += ";"
      self.curr_query += query
      self.curr_query += "\n"
   
   # the below function returns the result of the query
   def fetchall(self): 
      # get the script tag with id "create-db-script"
      script_element = document.getElementById("create-db-script")
      # set the "query" attribute of the script tag to the query string
      if(self.curr_query == ""):
         self.curr_query = self.prev_query
      if(script_element.getAttribute("first_exec_flag") is None):
         self.curr_query = "BEGIN TRANSACTION;\n" + self.curr_query
      if(self.prev_query == self.curr_query):  
         return formato_datos(self.prev_result)
      else:
         script_element.setAttribute("query", self.curr_query)
      self.prev_query = self.curr_query
      self.curr_query = ""
      # the below code waits for result
      # the result of the query can be one of 4 things
      # 1. the result of the query is data present in the table
      # 2. the result of the query is an empty list as no data is present in the table satisfying the query
      # 3. the result of the query is an error because the query is invalid
      # 4. the query is taking too long to execute, in this case if the qury takes more than 5 seconds, the flag is set to "query error flag" and the result is set to "query error"
      start = time.time()
      while(script_element.getAttribute("flag") != "data found" and script_element.getAttribute("flag") != "query error flag" and script_element.getAttribute("flag") != "no data"):
            time.sleep(0.1)
            if(time.time()-start > 5):
                script_element.setAttribute("flag", "query error flag")
                script_element.setAttribute("result", "query error")
                break

      result = json.loads(document.getElementById("create-db-script").getAttribute("result"))
      
      if(result=="no data"):
         document.getElementById("create-db-script").setAttribute("result", "")
         document.getElementById("create-db-script").setAttribute("flag", "0")
         raise Exception("No data found in the table for the query.\n\nException raised by sqlite3")
      elif(result=="query error"):
         document.getElementById("create-db-script").setAttribute("result", "")
         document.getElementById("create-db-script").setAttribute("flag", "0")
         error_msg = str(document.getElementById("create-db-script").getAttribute("error")) + "\n\nException raised by sqlite3"
         raise SyntaxError(error_msg)
      else:
         result = result[0]
         self.prev_result = result
         document.getElementById("create-db-script").setAttribute("result", "")
         document.getElementById("create-db-script").setAttribute("flag", "0")
         return formato_datos(result)

   
   # the below code resets the flag and result attributes of the script tag for the next query
   def close(self):
      document.getElementById("create-db-script").setAttribute("result", "")
      document.getElementById("create-db-script").setAttribute("flag", "0")               

class Connection:
   def __init__(self):
      self.curr_cursor_obj = Cursor()
   def cursor(self):
       return self.curr_cursor_obj
   def commit(self):
      self.curr_cursor_obj.curr_query =  self.curr_cursor_obj.curr_query +"COMMIT TRANSACTION;\nBEGIN TRANSACTION;\n"
   def rollback(self):
      self.curr_cursor_obj.curr_query =  self.curr_cursor_obj.curr_query +"ROLLBACK TRANSACTION;\n"
   def close(self):
      pass

def connect(table_name=None): return Connection()

It can be seen that the code got unnecessarily complex and a better way to handle different queries was needed.

What was the solution?

The final solution to this problem came to my mentor who discovered that the global Database variable could be accessed within the activecode block by using

db = window.parent.Database

This simple solution simplified the wrapper a lot and also ended the need to pass queries to attributes and need of a mutation observer.

The database could now be initialised or even loaded from a file very simply:

   <script type="module">
      const sqlPromise = initSqlJs({
      locateFile: file => `https://cdnjs.cloudflare.com/ajax/libs/sql.js/1.5.0/sql-wasm.wasm`
      });
      const dataPromise = fetch("../../_static/surfers.db").then(res => res.arrayBuffer());
      const [SQL, buf] = await Promise.all([sqlPromise, dataPromise])
      window.Database = new SQL.Database(new Uint8Array(buf));  
   </script>

Thus came the end of one of the important issues in my proposal this year. Issue #193 could now be closed😊. The final PR can be seen here.

Weekly Check-In #2 : Sqlite POC

shivam.stpaulsdarjeeling@gmail.com (shivamshan) — Mon, 27 Jun 2022 23:22:29 +0000

This week like the previous one will be dedicated to a POC that has been pending since the last GSoC. Like Pygame, the skulpt interpreter doesn't support sqlite3 library which is the decided library to be used to teach the SQL(Bases de Datos) lectures. The lectures aim to teach the basics like reading a database file, writing sql queries and fetching results.

What did I do this week?

I spent this week resolving some conflicts and upgrading a proof-of-concept PR that I had made earlier. This PR aims at serving as an example on which the SQL lectures can be based on. To sum up my work this week, I switched the code interpreter from Skulpt to Brython, so that I can use JS elements more directly and freely. I was also shown a solution by my mentor on how to use the database elements directly without using flags or complex logic of any sort. I ended this week by modifying the ssqlite3 wrapper that I had already written from scratch. This wrapper is supposed to almost exactly mimic the original sqlite3 library with functions such as COMMIT, ROLLBACK and BEGIN TRANSACTION.

What is coming up next?

The wrapper is still very new has limited functionality. To teach the lectures more efficiently, more functions of the original sqlite3 library has to be added. The wrapper has sql.js at its core and thus the wrapper has to be designed keeping this fact in mind too. Up next I need to work more on the wrapper and give more sqlite3 like functionalities to it. Only than can we move forward with designing the final lectures.

Did I get stuck anywhere?'

Although most of the upgrade was not major, I had a difficult time finding out a way to implement the load database from a file functionality.

base_datos = connect('surfers.db')

This line loads the database surfers.db to the variable base_datos. Implementing this using sql.js was troublesome for me as there is no straightforward method to load a database in sql.js. The closest one can get is to first convert the database into an array buffer and then load the array buffer. Figuring out this bit took some tinkering and surfing stack overflow 😂, but I found a solution that served our needs.

Week 1 : The dreaded Pygame Wrapper

shivam.stpaulsdarjeeling@gmail.com (shivamshan) — Mon, 27 Jun 2022 13:19:23 +0000

The inability of the skulpt interpreter to provide support for pygame and sqlite3 is what caused my selection into this year's GSoC program, well mostly(😉). One of my major under-takings this summer is to make the Pygame lectures interactive so that students may test their code on the go as they learn, which is the case for earlier exercises. This blog describes in detail how I reached to the solution(for now 😁) and what is left to be done.

PYGJS to the rescue

After several unsuccessful attempts of making the old brython fork work, I decided it was time I tried something else. It was then I came across this abandoned fork of pygjs. This library was originally intended to serve the same purpose I was currently looking for, making Pygame work in the browser. Initially it seemed liked the previous brython fork that I had to deal witth, but as I dove deeper into the code I found that this library uses gamejs along with brython for its implementation. Now, all that was left was to find a way to make use of this library considering it worked(😅).

Nothing is as easy as it seems

Although pygjs proved to be a very good option there will still some problems that need to be sorted before it could be used in the lectures. The first problem I faced was due to the project being abandoned. Though most of the code was usable, there were changes or modifications required to make the library behave as it was expected. For example, there were bugs in the key and mouse modules of pygjs that were causing unexpected behaviour. Secondly, the library needed to be used in a specific way . What I mean here is running some very basic instructions in brython like endless while loops, caused the JS thread to be blocked and the crashed the window. This problem was solved using timer.set_interval function offered by brython. One of the other problems was speed. Brython allows import of packages if they have a __init__.py present. But importing packages in this way in the browser makes the loading very slow. To solve this I had to package the pygame module into a pygame.brython.js file. This made loading the pyagame package mush simpler and faster. This PR serves as the final POC for the pygame wrapper and can be used to develop the exercises later.

Does it work?

I think it's safe for to say that Yes, it works for now and can be used to teach Pygame to the students. I have tried using all of the functionalities that may be required to teach Pygame to beginners and this library handled them all, starting from basic drawing to making animations and to playing music in the browser. You can check this out for yourself here: http://pyar.github.io/PyZombis/222/index.html .

Weekly Check-In #1 : Pygame POC

shivam.stpaulsdarjeeling@gmail.com (shivamshan) — Mon, 27 Jun 2022 12:19:38 +0000

This post marks the beginning of my official GSoC coding period. Starting from this post I will try to update everyone reading these posts about my weekly progress and the difficulties that I faced. So let's get to it.

What did I do this week?

According to my proposal I was supposed to spend the first two weeks closing the previous PRs and implementing a basic sqlite wrapper that could be further extended for the Bases se datos lecture. However, since no major PRs were required to be closed, me and my mentors decided that I should better complete the Pygame POC that was still not in place. Although we had decided upon how we wanted the Pygame lectures to be implemented, there was no proof-of-concept that actually showed that our idea would work. So I spent my first week preparing a basic Pygame wrapper.

What is coming up next?

Next, we have have decided to complete the SQL wrapper and get that PR merged. Doing so we will have put both our proof-of-concepts in place and then we can start designing the lectures (if we don't run into any bugs😝).

Did I get stuck anywhere?

I think there isn't going to be a week where the answer to this question is going to be "No" 😂. I definitely got stuck at many places. First of all I couldn't decide upon a particular idea that I wanted to go forward with. I had two options that me and my mentors had decided previously. I could use the old and unmaintained(😖) brython fork of pygame and build my wrapper using this or I could use gamejs API and build my wrapper around that. Although the second option sounded better, this meant I had to design the Pygame wrapper from scratch and that would take more time than I had. The problem with the first option was that it was not well maintained and had too many bugs. I tried my best to make it work by sorting out some bugs but soon realised it was not enough to implement the lectures. I had made a PR earlier to show that some parts of this old fork could be used but still couldn't be finally used.

Week 0 : a nuevos comienzos 🌅

shivam.stpaulsdarjeeling@gmail.com (shivamshan) — Fri, 24 Jun 2022 10:42:36 +0000

Hey everyone, I am Shivam Shandilya, final year student at Birla Institute of Technology, Mesra, Ranchi, India. This blogs aims to serve as a medium to record my experience as a contributor for PyZombis project under Python Argetina with PSF.

In this blog I will try my best to share the problems I face as I proceed and the thought process that I hope will get me to the final solution. I also plan to share with the readers the inputs received from my mentors so we know what areas required what kind of work.

This week marks the beginning of my GSoC experience and this time has been provided to us to bond with fellow contributors in this program. I had the pleasure to talk to some of the contributors of other projects and know their experience of getting into GSoC. I also had the opportunity to share my experience of open source and Python Argentina. We all agreed that it was the promptness and willingness of the maintainers to help us anytime that got us through. Thus I would like to begin my journey by thanking my mentors Mariano Reingart and Utkarsh Kumar for their help and constant support.

Looking for a great summer with you all.