Articles on Utkarsh's Blog

Final Blog post GSoC'21

utkarshdhsbgp@gmail.com (Utkarsh) — Fri, 20 Aug 2021 13:12:15 +0000

Hi everyone! It’s time for the last GSOC blog .I have learned a lot and faced many professional challenges. In the end it was really worth it. I hope GSOC experience will help me in the future and that my project will be useful to open source. You can check my final report at : https://github.com/PyAr/pyafipws/wiki/GSoC-2021:-Final-Summary

Thanks to both of my mentors for constantly helping me whenever I got stuck. Thank you for the constant help and advice; mentors that I would recommend to any student! It brings me great joy to end on a fulfilling note for contributing to Python and PyAfipWs project as part of Google Summer of Code 2021. I am happy and content with the work produced. I have previously worked on open source projects before, completing my tasks tasks for this project brought a level of satisfaction that cannot be matched. So that’s it. Thanks to all once again!

Hope to see you all in next GSoC or any other open source work.

Weekly Check-In #10 : Wrapping up

utkarshdhsbgp@gmail.com (Utkarsh) — Mon, 16 Aug 2021 06:03:50 +0000

Hey everyone! With this check-in we have come to the end of GSoC. Though some things are still left to be done, most of the work is completed and PRs have been merged. In this week I will be solving as many of these as possible issues this week. Most of the issues are from the PRs generated before are easier to solve. But there are some issues that need a more careful look into. I look forward to solving these problems and complete the project.

What issues are still not solved?

Out of all the left issues, this is the most important. It covers the testing of the ws_sr_padron.py script. The problem is internal and requires the attention of my mentors. However when this issue is resolved I can finish with writing tests for this script and increase the coverage to an acceptable percentage.

Another issue that needs to be looked at is regarding the pyfepdf script and pdf generation. The issue here was that different facturas.txt files were created for Python 2 and 3. Though I identified early that the error was with Unicode, I couldn't understand how to solve it. Solving this issue is important because without it we cannot check the completeness of the generated facturas.txt file(which is used to generate the pdf file).

Next issue, although not a major one, is present in many PRs and has to be solved for all of them. This issue deals with the location of CONFIG_FILE that contains the configuration of many scripts. I had directly use the original file present in the conf folder, which my mentors advised was a risky thing to do. Therefore, I had to modify the workflow file to make a copy of the rece.ini file in the working directory, so that the original is not modified in any way.

Blog Post #9

utkarshdhsbgp@gmail.com (Utkarsh) — Thu, 12 Aug 2021 03:57:40 +0000

In the penultimate week of my GSoC experience, I wrote unit tests and increased coverage for the scripts, namely, pyemail.py, pyqr.py, and pyi25. These scripts added email, QR, and barcode functionality respectively. My work for each of these scripts can be found in PR#94, PR#95, PR#96.

What did I do ?

As mentioned above, my work this week was to write unit tests for the scripts which responsible for email, QR , and barcode services. This week was mostly fun due to the fact that these scripts worked independently and the results were visual. By this I mean that you could see the QR and barcode generated by invoking the scripts and errors could be found more easily and fixed.

PyEmail

This script provided email functionality that could be used to send email to other users using a particular user's Email and password. It made use of smtplib library to make this possible. My work while developing unit tests was to mock this particular library using pytest-mock. This was necessary for testing as we couldn't otherwise check if the service was working properly or not. Therefore just checking if a particular function was invoked could help us to test the service. I wrote the following code to achieve this:-

def test_main_prueba(mocker):

    mocker.patch("smtplib.SMTP")
    sys.argv = [ ]
    sys.argv.append("/prueba")
    sys.argv.append("user@gmail.com")
    sys.argv.append("pass123")
    main()
    smtplib.SMTP.assert_called_with('smtp.gmail.com', 587)

One can easily see that I am using pytest-mock to mock the functionality of smtplib library and just checking if the SMTP fuction is invoked.

PyQR

This script provides QR image generation functionality in this project. Based on the data provided, it can be encoded into the QR image, which when scanned will provide the user with a link or url in this case. To test the script and check if it was working correctly, I compared the QR image generated during testing with a sample QR saved previously. To compare the images, I used Image and ImageChops library:-

def test_main_archivo():
    archivo = "prueba_qr.png"
    sys.argv = []
    sys.argv.append("--archivo")
    sys.argv.append(archivo)
    main()
    assert os.path.exists(archivo)
    #compare the image with a reference one
    if sys.version_info[0] < 3:
        ref = Image.open("tests/images/qr2.png")
    else:
        ref = Image.open("tests/images/qr.png")        
    test = Image.open(archivo)
    diff = ImageChops.difference(ref, test)
    assert diff.getbbox() is None

If one analyzed the code, they could see that I have tested the generated image with different images depending on the python version used. This is due to the fact that since strings are handled differently on both python versions, they lead to a different url being generated for both the python versions. But both URLs lead to the same page.

Pyi25

This script provided barcode functionality to the project and to test the generated barcode I used the same logic as above. ImageChops library is used to compare the pixel wise difference between two images.

def test_GenerarImagen():
    "Prueba de generación de imagen"
    barras = "2026756539302400161203034739042201105299"
    archivo = "prueba.png"
    pyi25.GenerarImagen(barras, archivo)
    assert os.path.exists(archivo)
    #compare the image with a reference one
    ref = Image.open("tests/images/prueba-cae-i25.png")
    test = Image.open(archivo)
    diff = ImageChops.difference(ref, test)
    assert diff.getbbox() is None

Also, two of the scripts, pyqr.py and pyi25.py provided an option to view the QR or barcode generated. Now on windows it's pretty straightforward. We just check if the image file is opened or not since on windows, the code uses os.startfile to open the images. But on linux eog service is required to be installed in order to view the image. So to check if this viewing functionality would work correctly on linux based systems, I mocked the os.system library and checked if it was invoked or not. This is sufficient because we can assume that if the user has eog image viewer installed, everything will work fine.

def test_main_mostrar(mocker):
    mocker.patch("os.system")
    sys.argv = []
    sys.argv.append("--mostrar")
    archivo = "prueba-cae-i25.png"
    main()
    if(sys.platform == 'linux2' or sys.platform == 'linux'):
        os.system.assert_called_with("eog " "%s" "" % archivo)

Weekly Check-In #9

utkarshdhsbgp@gmail.com (Utkarsh) — Sun, 08 Aug 2021 14:57:51 +0000

This blog marks the check-in for the 9th week of my GSoC experience. As this week is the penultimate week before the second evaluation, my work is to finish any previous left work and also to wrap up for the final submission. My mentors asked me to work on 3 scripts, namely, pyemail.py, pyi25.py and ,pyqr.py. My work was to develop unit tests for the scripts and also to increase their code coverage.

What work is to be done?

As mentioned above, this week, like the previous week had me working on unit tests and code coverage. The scripts that I was told to work on were different from the previous ones in the sense that these scripts were not dependent on other scripts for functioning , which made working with them a little less tiresome.
Also the unit tests had to be developed differently for them. Let me explain with an example:-
The script pyemail.py can be used for email services and requires the valid email and password of the user in order to work correctly. Since this is not possible while testing, my mentors suggested me to use pytest-mock, which be used to mock the functionality of modules while testing. In this example that module was smtplib.

Did I get stuck anywhere?

Yes, initially I had no idea to test such scripts that require the authentication of user's personal credentials in order to function. I was helped greatly by my mentors who informed me about pytest-mock and how it can be used to check if a certain function was being called or not. As you can understand this was enough in this case.

Blog Post #8

utkarshdhsbgp@gmail.com (Utkarsh) — Tue, 03 Aug 2021 22:47:19 +0000

In this blog I talk about my work done in the 8th week of my GSOC experience. This week my mentors had me working on 4 scripts, namely rece1.py, recem.py, recex1.py and receb1.py.

What did I do?

Testing and code coverage are two of the most important requirements of any project. Therefore much of my work lately can be seen focused around these two topics. Last week I had already written unit tests and increased the code coverage of 12 most important scripts. Each represented a web service and it was necessary that the code did not have unnecessary problems.

This week I worked on the above mentioned scripts and my work done for each of them can be found in the following links:-

What problems did I face ?

Most of my work this week comprised of finding minute errors that were causing failure of scripts. One the common errors in all 4 scripts was - 'entrada.txt not found error. All the 4 scripts required this file for working properly and it was generated with the help of /prueba command line option.

Another common error was in the function - depurar_xml(). This function was responsible for creating request.xml and response.xml files. Due to the difference in ways in which strings are handled in Python 2 and Python 3, this function required some modification in order to generate correct ouputs.
Similarly there were many instances where unicode errors were causing problems.

One other major problem that I faced was the presence of the sys.exit(0) statement that was present in all 4 scripts along with the command line options. Normally these statements seem harmless, but if pytest encounters them while testing, it leads to failure. Same goes with the input() function.
Example:-

        if "/dummy" in sys.argv:
            print("Consultando estado de servidores...")
            ws.Dummy()
            print("AppServerStatus", ws.AppServerStatus)
            print("DbServerStatus", ws.DbServerStatus)
            print("AuthServerStatus", ws.AuthServerStatus)
            sys.exit(0)

Writing a test for the /dummy command line option, would give an error when run.

Same goes with statements like these:-

        if "/consultarcaea" in sys.argv:
            periodo = input("Periodo: ")
            orden = input("Orden: ")

Here the error is due to the input() method, as pytest doesn't allow them. To make the above command line option work one has to read the command line arguments.
Something like this:-

            if len(sys.argv) > sys.argv.index("/consultarcaea") + 1:
                periodo = sys.argv[sys.argv.index("/consultarcaea") + 1]
                orden = sys.argv[sys.argv.index("/consultarcaea") + 2]
            else:
                periodo = input("Periodo (año-mes, ej 201108): ")
                orden = input("Orden (quincena, 1 u 2): ")

Overall I had fun while solving these problems. Watching little green dots appear after the pytest command offer an another level of satisfaction and content 😉.

Weekly Check-In #8

utkarshdhsbgp@gmail.com (Utkarsh) — Sun, 01 Aug 2021 19:43:34 +0000

This check-in marks the beginning of the 8th week of my GSoC journey. This week is also going to be same in terms of the work done, as I will again be working on code coverage of 4 more scripts.

What will I be doing ?

This week I will be working on adding unit tests for 4 scripts and also increasing their coverage. The scripts are :-

rece1.py
recem.py
recex1.py
receb1.py

Did I get stuck anywhere till now?

Since I had already worked on the coverage of 12 scripts before, I did not face any major roadblock till now. But the entire week in in front of me and I am excited for the challenges.

Blog Post #7

utkarshdhsbgp@gmail.com (Utkarsh) — Sun, 01 Aug 2021 11:15:09 +0000

In this blog I cover the work done by me in the 7th week of GSOC. This week had me working on code coverage. The initial target set by my mentors for code coverage was 85%. The following lines explain what I was able to achieve, what problems I faced and what still needs to be done.

What did I do?

As already mentioned above, I worked on increasing the code coverage for the package this week. I worked on a total of 12 scripts this week. In some of the scripts I was able to increase the coverage more that 80%, but in a few I was not able to due to some internal server errors. Overall I achieved 81% coverage.
My work on individual scripts can be seen in the following links:

Also the coverage for individual files can be seen below:-

pytest --cov-config=.coveragerc --cov=pyafipws tests/                   
=========================== test session starts ===========================
platform win32 -- Python 3.9.6, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\shiva\OneDrive - Birla Institute of Technology\Desktop\GSoC_online\pyafipws, configfile: pytest.ini
plugins: cov-2.12.1, freezegun-0.4.2, html-3.1.1, metadata-1.11.0, vcr-1.0.2
collected 360 items

tests\test_pyfepdf.py ..............                                                                                                                                             [  3%]
tests\test_ws_sr_padron.py s.sx                                                                                                                                                  [  5%]
tests\test_wsaa.py ..............                                                                                                                                                [  8%]
tests\test_wsbfev1.py ...............                                                                                                                                            [ 13%]
tests\test_wscdc.py s...........                                                                                                                                                 [ 16%]
tests\test_wsct.py ..........xx..xxxxxxxxxxssxxxxxx..                                                                                                                            [ 25%]
tests\test_wsfev1.py ..........................                                                                                                                                  [ 33%]
tests\test_wsfev1_dummy.py .                                                                                                                                                     [ 33%]
tests\test_wsfexv1.py .................                                                                                                                                          [ 38%]
tests\test_wslsp.py ......................................x....................                                                                                                  [ 54%]
tests\test_wsltv.py ............................x.................                                                                                                               [ 67%]
tests\test_wslum.py ........................x...........                                                                                                                         [ 77%]
tests\test_wsmtx.py ..........................................                                                                                                                   [ 88%]
tests\test_wsremcarne.py ........................................                                                                                                                [100%]


----------- coverage: platform win32, python 3.9.6-final-0 -----------
Name                                                                          Stmts   Miss  Cover
-------------------------------------------------------------------------------------------------
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\__init__.py         4      0   100%
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\pyfepdf.py        802    162    80%
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\wsaa.py           350     81    77%
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\wsbfev1.py        382     53    86%
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\wscdc.py          213     32    85%
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\wsct.py           429    177    59%
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\wsfev1.py         684     83    88%
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\wsfexv1.py        454     73    84%
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\wslsp.py          622    103    83%
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\wsltv.py          561     84    85%
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\wslum.py          461    111    76%
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\wsmtx.py          607    119    80%
py3\Lib\site-packages\PyAfipWs-3.9.b_2522_-py3.9.egg\pyafipws\wsremcarne.py     432     66    85%
-------------------------------------------------------------------------------------------------
TOTAL                                                                          6001   1144    81%

======================= 333 passed, 5 skipped, 22 xfailed, 66 warnings in 57.58s =======================

Weekly Check-In #7

utkarshdhsbgp@gmail.com (Utkarsh) — Thu, 29 Jul 2021 07:40:39 +0000

After 4 days of evaluation, I am back to working on the remainder of my proposal. This week's work is one of the most important aspect of my proposal, that is ,code coverage. My mentors have decided to keep the acceptable code coverage to 80% , but also told me that 85% is achievable.

What will I be doing?

. Code coverage is directly related to unit testing and determines what percentage of the code has been covered by the tests. Whenever a large application or service is designed, it is important to test that all the features are working properly before deploying. Therefore my work is to write more tests if required and fix bugs that will lead to the acceptable code coverage range. Currently, the overall code coverage is 46% and you can understand that I have a long week ahead of me. Also there are going to be many changes made and thus my mentors have asked me to make separate PRs for all the scripts. That means in total there would be about 12 PRs in this week alone.

Did I get stuck anywhere?

Yes, right of the bat I ran into some errors. Firstly the certificates required for authorization had expired. So they had to be replaced in order for me to continue. Secondly, some of the services were not linked with the new certificates. SO they too had to to linked by the mentors. Lastly, some of the command line options were not giving the expected output due to internal errors. I created issues for each of the above problems and informed my mentors for guidance. The issues can be seen here:- #70, #83, #84, #85.

Blog Post #6

utkarshdhsbgp@gmail.com (Utkarsh) — Thu, 29 Jul 2021 07:03:49 +0000

This week marks the end of the first half of my GSoC experience and also the start of my first evaluation.

FIrst Evaluation....What has been done till now?

So, during the first 5 weeks my work comprised of the following :-

Merging the work from previous GSoC
Fixing errors that arose due to the merge
Fixing bugs due to Python versions
Adding pytest-vcr to the testing suite
Adding cassettes for later use
Adding more unit tests for the scripts
Adding Python entry Points

Blog Post #5

utkarshdhsbgp@gmail.com (Utkarsh) — Wed, 14 Jul 2021 15:54:27 +0000

This blog covers the work done by me in the 5th week of GSoC. This week also marks the end of the first phase and with this work I will enter the first evaluation phase which is scheduled from 12th to 16th July. The work mentioned in this blog can be found in PR#69

What did I do?

As mentioned before my mentors had asked me to work on python entry points which basically allow us to call scripts directly from the command line using one word name. These entry points are installed when we install the package with the command python setup.py install. They allow an easy way to call scripts with their command line options with just one word instead of mentioning the whole whole module.

To do this I had to add a main() function to many scripts that contained all the command line options. This function would ultimately serve as the entry point of the scripts when called from the command line. While doing this I also had to keep in mind the variables that were modified inside this function because I would have to define a global declaration for them. This mainly constituted my work in the 5th week. The scripts for which I added entry points were:-

cot
iibb
padron
pyemail
pyfepdf
pyi25
pyqr
rece1
receb1
recem
recet
recex1
rg3685
sired
trazafito
trazamed
trazaprodmed
trazarenpre
trazavet
wdigdepfiel
ws_sire
ws_sr_padron
wsaa
wsbfev1
wscdc
wscoc
wsct
wsctg
wsfecred
wsfev1
wsfexv1
wslpg
wslsp
wsltv
wslum
wsmtx
wsremazucar
wsremcarne
wsremharina

I chose to add the entry points inside the setup.cfg file instead of setup.py file, because the configuration file was already present inside the package.

What problems did I face?

I think this was the only week that I can say that I did not face any significant problems. Since adding entry points was a straight-forward task, I only had to invest some time for this task. Although initially I did get stuck while finding the function that would serve as the entry point for the scripts. The problem was that not all scripts had a main() function in them, nor did the have a proper function that would serve as the entry point. Then with some tinkering and the help of my mentors I decided to add a main function in all the scripts. After this it was only a matter of adding them in the configuration file and making global declarations for the variables that were modified.

Results

$ wsaa

para soporte de WebClient debe instalar httplib2
Encodign in cp1252
Usando CRT=reingart.crt KEY=reingart.key URL=https://wsaahomo.afip.gov.ar/ws/services/LoginCms SERVICE=wsfe TTL=36000
WSAA Version 3.11c  False

<loginticketresponse version="1.0">
    
        <source>CN=wsaahomo, O=AFIP, C=AR, SERIALNUMBER=CUIT 33693450239
        <destination>SERIALNUMBER=CUIT 20267565393, CN=reingart2019pub</destination>
        <uniqueid>657428365</uniqueid>
        <generationtime>2021-07-09T13:51:57.149-03:00</generationtime>
        <expirationtime>2021-07-10T01:51:57.149-03:00</expirationtime>
    
    <credentials>
        <token>P+Cg==</token>
        <sign>s4M0=</sign>
    </credentials>
</loginticketresponse>

$ wsfev1 --prueba

para soporte de WebClient debe instalar httplib2
Leyendo TA de C:\Users\pyafipws\py3test\lib\site-packages\pyafipws-3.9.b_2514_-py3.9.egg\pyafipws\cache\TA-2b4277f5962bd2cb0265c0fac4b32304.xml...
b"FECAESolicitar(Auth=*{'Token': <class 'str'="">, 'Sign': <class 'str'="">, 'Cuit': <alias 'long'="" for="" '<class="" 'int'="">'>}*, FeCAEReq=*{'FeCabReq': 
*{'CantReg': <class 'int'="">, 'PtoVta': <class 'int'="">, 'CbteTipo': <class 'int'="">}*, 'FeDetReq': [*{'FECAEDetRequest': *{'Concepto': <class 'int'="">, 'DocTipo': <class 'sg':="" <class="" 'str'="">}*}*], 'Errors': [*{'Err': *{'Code': <class 'int'="">, 'Msg': <class 'str'="">}*}*]}*}**:\n\nSolicitud de C\xf3digo de Autorizaci\xf3n Electr\xf3nico (CAE)\nHeaders: None"</class></class></class></class></class></class></class></alias></class></class>
Nro. Cbte. desde-hasta 493 493
Resultado A
Reproceso
CAE 71283954824423
Vencimiento 20210719
Observaciones 10217: El credito fiscal discriminado en el presente comprobante solo podra ser computado a efectos del Procedimiento permanente de transicion al Regimen General.
71283954824423 == 71283954824423
3 != 2
10217 != 10017
El credito fiscal discriminado en el presente comprobante solo podra ser computado a efectos del Procedimiento permanente de transicion al Regimen General.

$ pyfepdf --prueba --mostrar

para soporte de WebClient debe instalar httplib2
Encodign in cp1252
Prueba!

Weekly Check-In #5

utkarshdhsbgp@gmail.com (Utkarsh) — Sat, 10 Jul 2021 19:36:26 +0000

This article covers the work assigned to me for the fifth week of my GSOC experience. This is last week before the first evaluation and I am very excited to conclude the first half and begin the next half with same grit and motivation.

What will I be doing?

This weeks my mentors want me to work on developing entry points for the scripts in the module. An "entry point" is typically a function (or other callable function-like object) that a developer or user of your Python package might want to use.

The most popular kind of entry point is the console_scripts entry point, which points to a function that you want made available as a command-line tool to whoever installs your package.

Though there is more than one way to define entry points, I will define them in setup.cfg file.

What problems did I face?

The first problem that I encountered as soon as I began this task was that not all scripts had a main() function defined in them. Now this is necessary because entry points access a function in the script that they make available to the used via the command-line directly.

So, first that is what I would like to focus on. While doing this I also have to keep in mind that I don't change too much code as it makes the review process very time-consuming for the mentors.

Blog Post #4

utkarshdhsbgp@gmail.com (Utkarsh) — Sat, 10 Jul 2021 16:17:51 +0000

This blog covers the work done by me in the fourth week of my GSOC experience. I spent this week fixing 7 skipped or xfailing tests and added coverage report to the the github workflow. Now the number of tests passing has reached up to 209. Also we can now view the code coverage due the tests in the coverage report.

What did I do?

As mentioned above I fixed some tests that were marked as xfailing or skipped and added support for code coverage. These works can be found in PRs #67 and #68.

Fixed Tests

The 7 tests that I fixed majorly had two issues. The first issue was related to how str object was dealt with in both Python versions(2 and 3). The second issue was that some tests used datetime object to verify values or certificates. This was fine until date was within the range of expiration of the ticket or value. But once those values and tickets expired, the tests would fail.

To solve the first issue I used the following imports that fixed it on both the Python versions:-

>>from past.builtins import basestring
>>from builtins import str

To solve the second issue I used pytest-freezegun module, that literally allows you to freeze time for the particular marked tests or test script. Using this module allowed the same request to be made every time to the servers therefore the above mentioned issues would never arise.

Added coverage report

The second task that I completed this week was to add coverage report to the workflow. This allowed one to view the detailed code coverage script wise and also the overall percentage. This was important because in later weeks my work is to increase the code coverage up to 85%. Adding coverage also helps in the way that it allows one to view the code that has been covered or not so that the tests can be developed in a better way.

Did I get stuck anywhere?

Yes, I definitely got stuck in the early part of the week when I was figuring out the reasons for the failing tests. It was a bit tricky to find the issue was because it was flaky. It was not a constant error that occurred every time I ran the tests.

The fact that tests had to make the same request every time for the assertions to be true was not obvious at first. It took a lot of debugging and constant help from my mentors to find the and solve the issue completely.

One other issue that caused me to waste time and effort was to re-record cassettes for the xfailing tests. As mentioned above they would work perfectly once and then after some time later would cause the tests to fail.

Results

Fixed Tests

Now with the above fixes, the number of passing tests rose from 202 to 209.

for python 2
$ pytest

============================= test session starts =============================
platform win32 -- Python 2.7.12, pytest-4.6.11, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\shiva\OneDrive\Desktop\Pyafipaws_Utkarsh\pyafipws, inifile: pytest.ini
plugins: html-1.22.1, metadata-1.11.0, vcr-1.0.2
collected 236 items

tests\test_ws_sr_padron.py s.sx                                          [  1%]
tests\test_wsaa.py .........                                             [  5%]
tests\test_wsbfev1.py ..........                                         [  9%]
tests\test_wscdc.py s.......                                             [ 13%]
tests\test_wsct.py ..........xx..xxxxxxxxxxssxxxxxx                      [ 26%]
tests\test_wsfev1.py ......                                              [ 29%]
tests\test_wsfev1_dummy.py .                                             [ 29%]
tests\test_wsfexv1.py ..........                                         [ 33%]
tests\test_wslsp.py ......................................x.             [ 50%]
tests\test_wsltv.py ............................x.                       [ 63%]
tests\test_wslum.py ........................x.                           [ 74%]
tests\test_wsmtx.py .................................                    [ 88%]
tests\test_wsremcarne.py ...........................                     [100%]


======= 209 passed, 5 skipped, 22 xfailed, 1 warnings in 21.75 seconds ========

for python 3
$ pytest

======================== test session starts =====================
platform win32 -- Python 3.9.2, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\shiva\OneDrive\Desktop\Pyafipaws_Utkarsh\pyafipws, configfile: pytest.ini
plugins: html-3.1.1, metadata-1.11.0, vcr-1.0.2
collected 236 items

tests\test_ws_sr_padron.py s.sx                                          [  1%]
tests\test_wsaa.py .........                                             [  5%]
tests\test_wsbfev1.py ..........                                         [  9%]
tests\test_wscdc.py s.......                                             [ 13%]
tests\test_wsct.py ..........xx..xxxxxxxxxxssxxxxxx                      [ 26%]
tests\test_wsfev1.py ......                                              [ 29%]
tests\test_wsfev1_dummy.py .                                             [ 29%]
tests\test_wsfexv1.py ..........                                         [ 33%]
tests\test_wslsp.py ......................................x.             [ 50%]
tests\test_wsltv.py ............................x.                       [ 63%]
tests\test_wslum.py ........................x.                           [ 74%]
tests\test_wsmtx.py .................................                    [ 88%]
tests\test_wsremcarne.py ...........................                     [100%]


================ 209 passed, 5 skipped, 22 xfailed, 3 warnings in 13.89s ===========

Added coverage

One can view the coverage report in the Artifacts section of the runs section of Actions page.

Weekly Check-In #4

utkarshdhsbgp@gmail.com (Utkarsh) — Wed, 07 Jul 2021 14:52:16 +0000

This is my weekly check-in as I head into the fourth week of my Gsoc experience. In this article I talk about the work that I will carry out and the approach that I will accept to do so.

What will I be doing?

My mentors have asked me to fix some more unit tests before I can move forward and work on python entry points. These are tests that were either marked as skipped or xfailed because of the unexpected results they were giving. The scripts that contained these skipped or xfailing tests were:-

test_wsaa.py
test_wsfev1.py
test_wsmtx.py

There were 7 such tests that needed to be fixed, taking the number of passing tests to 209.

Since some xfailing tests were fixed, their cassettes had to be re-recorded. Also my mentors had asked me to move all tests from test_wsaa_crypto.py to test_wsaa.py. This was done so that there was only a single test script for all the scripts that were tested.

Apart from the tests, my mentors had also asked me to add coverage report for the tests, so that we could have the code coverage in percentage for the scripts that were tested.

Did I get stuck anywhere?

My approach for the tests was simple, unmark the skipped and xfailing tests and fix the tests one test at a time. Soon I realized that main problems with the tests that I was supposed to fix was that they were dependent on validating or asserting certain values to variables that were time-dependent. In other words , the values that were tested would change with time .These values were generated by the servers depending on the time and date. This was the reason that tests were failing now but passed during the time they were created. Initially I was stuck while understanding the cause of errors. But my mentors were really helpful and explained the error in detail.

Blog Post #3

utkarshdhsbgp@gmail.com (Utkarsh) — Sun, 04 Jul 2021 17:52:59 +0000

This blog covers the work done by me in the 3ed week of my GSOC experience. My work during this time was focused on fixing version issues that were causing 11 tests to fail on python 2 environment, but not on python 3. All the work can be found in this PR.

What did I do?

The majority of my work was to find a solution to the issues that were causing different behaviors on two different environments, i.e. Python 2 and Python 3. The test scripts that were giving problems were:-

test_wsaa.py
test_wsbfev1.py
test_wsfev1.py
test_wsbfexv1.py
test_wslsp.py
test_wsltv.py
test_wslum.py
test_wsmtx.py

The main problem was the way in which string is treated on python 2 vs python 3. The str object in Python 3 is quite similar but not identical to the Python 2 unicode object. The major difference is the stricter type-checking of Py3’s str that enforces a distinction between unicode strings and byte-strings, such as when comparing, concatenating, joining, or replacing parts of strings.
Along with this I also added a pytest configuration file to register custom markers that I used to stop using fixtures in particular scripts and test methods.

Did I face any problems?

Yes, I faced a lot of problems finding the particular errors that were causing the test failures. Also I trouble finding the exact solution to the problem otherwise a different solution would lead to tests failing on python 3 environment as well. Apart from all this, my other point of focus was to make my solution robust. For example, at first I decided to handle the tests for different versions differently in the test method itself like this

if sys.version_info[0] == 3:
        assert isinstance(wsbfev1.CAE,str)
    elif sys.version_info[0] == 2:
        assert isinstance((wsbfev1.CAE),future.types.newstr)

But then it was correctly pointed out by my mentors was that this made the code look ugly was not a robust solution. Therefore I found a different solution. future module contains a newstr type that is a backport of the str object from Python 3. This inherits from the Python 2 unicode class but has customizations to improve compatibility with Python 3’s str object. We can use it as follows:

>>> from __future__ import unicode_literals
>>> from builtins import str

Using str in this way eliminated the error and was also a robust solution.

Results

Using the above mentioned solution fixed the 11 tests that were not working on python 2 environment to run and pass smoothly. With this 202 tests now passed on both the environments.

Python 2

$ pytest

============================= test session starts =============================
platform win32 -- Python 2.7.12, pytest-4.6.11, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\shiva\OneDrive\Desktop\Pyafipaws_Utkarsh\pyafipws, inifile: pytest.ini
plugins: html-1.22.1, metadata-1.11.0, vcr-1.0.2
collected 236 items

tests\test_ws_sr_padron.py s.sx                                          [  1%]
tests\test_wsaa.py ...X.                                                 [  3%]
tests\test_wsaa_crypto.py ....                                           [  5%]
tests\test_wsbfev1.py ..........                                         [  9%]
tests\test_wscdc.py s.......                                             [ 13%]
tests\test_wsct.py ..........xx..xxxxxxxxxxssxxxxxx                      [ 26%]
tests\test_wsfev1.py ...xxx                                              [ 29%]
tests\test_wsfev1_dummy.py .                                             [ 29%]
tests\test_wsfexv1.py ..........                                         [ 33%]
tests\test_wslsp.py ......................................x.             [ 50%]
tests\test_wsltv.py ............................x.                       [ 63%]
tests\test_wslum.py ........................x.                           [ 74%]
tests\test_wsmtx.py ..............x......x..........x                    [ 88%]
tests\test_wsremcarne.py ...........................                     [100%]

== 202 passed, 5 skipped, 28 xfailed, 1 xpassed, 1 warnings in 27.46 seconds ==

Python 3

$ pytest

============================= test session starts =============================
platform win32 -- Python 3.9.2, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: C:\Users\shiva\OneDrive\Desktop\Pyafipaws_Utkarsh\pyafipws, configfile: pytest.ini
plugins: cov-2.12.1, html-3.1.1, metadata-1.11.0, vcr-1.0.2
collected 236 items

tests\test_ws_sr_padron.py s.sx                                          [  1%]
tests\test_wsaa.py ...X.                                                 [  3%]
tests\test_wsaa_crypto.py ....                                           [  5%]
tests\test_wsbfev1.py ..........                                         [  9%]
tests\test_wscdc.py s.......                                             [ 13%]
tests\test_wsct.py ..........xx..xxxxxxxxxxssxxxxxx                      [ 26%]
tests\test_wsfev1.py ...xxx                                              [ 29%]
tests\test_wsfev1_dummy.py .                                             [ 29%]
tests\test_wsfexv1.py ..........                                         [ 33%]
tests\test_wslsp.py ......................................x.             [ 50%]
tests\test_wsltv.py ............................x.                       [ 63%]
tests\test_wslum.py ........................x.                           [ 74%]
tests\test_wsmtx.py ..............x......x..........x                    [ 88%]
tests\test_wsremcarne.py ...........................                     [100%]

== 202 passed, 5 skipped, 28 xfailed, 1 xpassed, 1 warnings in 27.46 seconds ==

Weekly Check-In #3

utkarshdhsbgp@gmail.com (Utkarsh) — Mon, 28 Jun 2021 16:18:06 +0000

This week my work is try and get the tests working that were previously not working with python version 2. There are 11 such tests that pass only on python 3 and not on python 2.

What did I do this week?

This week I searched and fixed the bug that was causing 11 tests to fail on python 2 and pass without any issue on python 3. After much grinding I realized the issue was in the way python 2 handled strings. As the tests were written on a python 3 environment, when evaluated on a python 2 environment, they threw errors if a sequence of characters enclosed within curly brackets was compared to a string object(str). Python 2 has an unicode object which is quite similar to str in Python 3 but not identical. Due to this, issues arise in the way a string is compared in both python versions.

Did I get stuck anywhere

Not really, but finding the correct way in which strings are handled on different versions took some considerable research and patience. The problem was to write code that would make the tests run smoothly on both versions. For this I needed to know exactly what a function used in the tests returned. Whether it was a string or byte-string or some other data type. Also apart from these problems, while fixing some tests, I had to re record some cassettes to find the error.

Blog Post #2

utkarshdhsbgp@gmail.com (Utkarsh) — Mon, 28 Jun 2021 10:44:04 +0000

The second week of my GSOC experience mainly comprised of me fixing the tests that were merged into the main branch. The problems mainly arose due to the difference in main branch and the environment in which the tests were developed.

What did my work comprise of?

My work was mainly to get most of the tests working and to develop cassettes to record the traffic for faster retesting. I had to learn how to use pytest-vcr and to implement in such a way that was efficient and removed redundancies. There were 236 tests in total that covered 11 different scripts. The main reason for recording the traffic was that almost all the services offered require WSAA authentication and they require the wsaa.Autenticar() function to be called. The problem with this is that once this function is called, calling it immediately after causes an exception:- Falla SOAP: ns1:coe.alreadyAuthenticated: El CEE ya posee un TA valido para el acceso al WSN solicitado.
Another problem that I faced was the overwriting of the cassettes once after the recoding was done. This was seen in the exception :- CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('cassette.yaml') in your current record mode ('once').

How did I get about solving these problems?

I figured out that the reason for the CannotOverwriteExistingCassetteException was that there were many redundant requests and while replaying were causing the cassettes to be re-recorded. One way to handle this as to define a fixture that would handle the incoming requests.
I used the following fixture to fix this issue:-


@pytest.fixture(autouse=True)
def auth(request):
    if 'dontusefix' in request.keywords:
        return
    z=request.module.__obj__
    z.Cuit = CUIT
    wsaa=WSAA()
    ta = wsaa.Autenticar(request.module.__service__, CERT, PKEY)
    z.SetTicketAcceso(ta)
    z.Conectar(CACHE, request.module.__WSDL__)
    return z

Also I had to record the traffic script by script and make cassettes. It was a tedious process but the result was worth it. Initially was running the pytest, it would take about 5 mins for all the tests to run and show the results of the test. Now with the cassettes in play, it was only a matter of seconds before all the tests ran. The cassettes also helped in solving the Already authenticated exception. This was because now when the scripts made request to the servers, the recorded response was played for them. This resulted in faster and efficient testing.

Final Result

Finally with the help of cassettes, I managed to get 202 tests passing on python 3 and 191 tests passing on python 2. The difference in the number of tests passing was due to some unicode errors on python 2. I plan to resolve these issues in the upcoming week so I can get maximum tests passing on both the versions. Here is a summary of tests on python 3:-

$ pytest

============================= test session starts ==============================
platform linux -- Python 3.9.5, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: /home/runner/work/pyafipws/pyafipws
plugins: vcr-1.0.2, html-3.1.1, metadata-1.11.0
collected 236 items

tests/test_ws_sr_padron.py s.sx                                          [  1%]
tests/test_wsaa.py ...x.                                                 [  3%]
tests/test_wsaa_crypto.py ....                                           [  5%]
tests/test_wsbfev1.py ..........                                         [  9%]
tests/test_wscdc.py s.......                                             [ 13%]
tests/test_wsct.py ..........xx..xxxxxxxxxxssxxxxxx                      [ 26%]
tests/test_wsfev1.py ...xxx                                              [ 29%]
tests/test_wsfev1_dummy.py .                                             [ 29%]
tests/test_wsfexv1.py ..........                                         [ 33%]
tests/test_wslsp.py ......................................x.             [ 50%]
tests/test_wsltv.py ............................x.                       [ 63%]
tests/test_wslum.py ........................x.                           [ 74%]
tests/test_wsmtx.py ..............x......x..........x                    [ 88%]
tests/test_wsremcarne.py ...........................                     [100%]

-- Docs: https://docs.pytest.org/en/stable/warnings.html
- generated html file: file:///home/runner/work/pyafipws/pyafipws/report.html --
============ 202 passed, 5 skipped, 29 xfailed, 6 warnings in 6.68s ============

and with python 2:-
$ pytest

============================= test session starts ==============================
platform linux2 -- Python 2.7.18, pytest-4.6.11, py-1.10.0, pluggy-0.13.1
rootdir: /home/runner/work/pyafipws/pyafipws
plugins: vcr-1.0.2, html-1.22.1, metadata-1.11.0
collected 236 items

tests/test_ws_sr_padron.py s.sx                                          [  1%]
tests/test_wsaa.py ..sxs                                                 [  3%]
tests/test_wsaa_crypto.py ....                                           [  5%]
tests/test_wsbfev1.py .....s....                                         [  9%]
tests/test_wscdc.py s.......                                             [ 13%]
tests/test_wsct.py ..........xx..xxxxxxxxxxssxxxxxx                      [ 26%]
tests/test_wsfev1.py .ssxxx                                              [ 29%]
tests/test_wsfev1_dummy.py .                                             [ 29%]
tests/test_wsfexv1.py .....s...s                                         [ 33%]
tests/test_wslsp.py ......................................xs             [ 50%]
tests/test_wsltv.py ............................xs                       [ 63%]
tests/test_wslum.py ........................xs                           [ 74%]
tests/test_wsmtx.py ...........s..x......x..........x                    [ 88%]
tests/test_wsremcarne.py ...........................                     [100%]

-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated html file: file:///home/runner/work/pyafipws/pyafipws/report.html --
======== 191 passed, 16 skipped, 29 xfailed, 2 warnings in 6.24 seconds ========

Weekly Check-In #2

utkarshdhsbgp@gmail.com (Utkarsh) — Mon, 21 Jun 2021 09:39:42 +0000

Hey everyone, this is me checking in after my second week in GSOC. This week I had to work on unit testing with pytest and pytest-vcr-- PR#63. My work included fixing the already developed tests for the new test suite and also generate cassettes for replaying the tests. This was done to countermeasure the situation if the AFIP servers ever went down.

What did you do this week?

This week’s work was comparatively tougher than the last week’s. I had to fix the tests that were designed in Gsoc 2019. These tests were for the major python scripts in this project. I mainly worked on the issues that arose from PR#59.
These issues were mainly due to the merging of the tests in the new main branch. I also had to work on generating cassettes for the tests as I recorded them using pytest-vcr. This was very useful and necessary because the tests continuously made requests to the server for authentication. Once authenticated the servers would return a message - Falla SOAP: ns1:coe.alreadyAuthenticated: El CEE ya posee un TA valido para el acceso al WSN solicitado. This was the case because the tickets generated were already authorized and we needed to wait sometime before trying to authenticate again. Since the tests needed to run without waiting in the Github wokflow, cassettes for all tests were generated.

Did you get stuck anywhere?

Yes, and badly. As this was my first time working with pytest-vcr, using the right decorators and fixtures was a nightmare. The main problem was that the tests were making a lot of redundant requests and while replaying were casing the error - CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('cassette.yaml') in your current record mode ('once').. Fixing this issue was most overwhelming to me because there were more that 120 tests and I had to fix errors in each.

Blog Post #1

utkarshdhsbgp@gmail.com (Utkarsh) — Sun, 13 Jun 2021 20:57:22 +0000

In this blog I briefly talk about the work done and issues fixed in week 1 and what I will be doing for the following 2nd week.

What did I do?

The first week of my GSOC experience proved to be a great and fruitful as I successfully completed the week’s tasks under the guidance of my mentors. During the first week my task was to fix issues that arose due to the merging of PR #25 into the main branch. This PR replaced M2Cryto as the standard library for cryptographic recipes. These issues included support inline passing of certificate and key in wsaa.py script and passphrase support for encoding the private key. Along with these issues my PR-- PR#58 also added more unit tests for wsaa.py.
Apart for this particular PR, more 2 of my PR’s were merged which fixed minor bugs(PR #57) and added pytest-vcr support(PR #56) for dummy tests.

What will I be doing next?

Now for my 2nd week I have to work on testing wsaa.py script. Mainly my work will be to develop unit tests that adapt to the current test suite and improve the tests using fixtures and VCR. Although working on test-coverage is scheduled for later weeks, I will be developing the tests keeping the overall test-coverage in mind as the final test-coverage goal is set at 85%.
Needless to say I had lot of fun during the 1st week, mainly during developing tests as seeing them pass is the best feeling ever 😉. Also I am very grateful to my mentors who have made the environment very comfortable to work in. They are there to listen to my silliest queries and are very prompt in their replies. Looking forward to their guidance as I enter the 2nd week of my GSOC experience.

Weekly Check-In #1

utkarshdhsbgp@gmail.com (Utkarsh) — Tue, 08 Jun 2021 04:54:55 +0000

Hello! My name is Utkarsh. I am working this summer with Python Argentina for the project Pyafipws. During all GSOC period I will be posting each week here on my blog updates of my work, starting today! The community bonding period helped me to know my mentors and setup a working plan for the next weeks.I put the milestones mentioned on my proposal as GitHub milestones in my forked repo.I run few tests using pytest VCR.There were few errors in wsaa.py and wsfev1.py which I fixed it.I kept on practicing writing unit tests with VCR for Wsfev1.CompUltimoAutorizado,Wsfev1.CompConsultar methods.I will merge pull requests of previous gsoc in new branch with new features this week.There are some errors which I need to fix in wsfev1.py and also I will keep writing tests and run them.I was not able to generate TA.xml from wsaa.py in new branch and also there were few errors coming while running the tests.But my mentors are very helpful, they helped me a lot and that is the reason I was able to fix this issues.Hoping for great summer ahead!!

Community Bonding period

utkarshdhsbgp@gmail.com (Utkarsh) — Sat, 29 May 2021 11:22:00 +0000

Hey there! I am Utkarsh from india working on pyafipws project of python argentina(PSF).Community bonding period is going on now and I am busy in reading the documentation & preparing of a work plan.I am constantly communicating with the mentors and discussing about the ideas.Its really going to be a great summer ahead!!