Articles on Sando's Blog

See you later PSF-GSoC

nicolassandovalunm@gmail.com (Sando) — Tue, 20 Aug 2019 00:16:18 +0000

Hi everyone!

"""

The truth was that it was an incredible experience I recommend it
Anyone who likes challenges in new technologies,
development and especially python.

Work under the supervision of titans such as PSF and Google
It gives an interesting boost.
Thank you for supporting the development of the potential of
people.

As for work, the second stage was complicated enough for me
I was running out of time and my lack of knowledge complicated
even more, set aside the web service testing by the client test of pyafipws that for now is more important for the project.

I will continue working on my knowledge,
A lot of ink remains in the inkwell.

I want to thank my mentors for the support,
to this beautiful community that is PyAr,
to all the participants of Gsoc-2019 I hope you had a great experience,
to sounak98 you did a great job on the blog,
to the botanic for being aware of us,
and of course,
to the administrators :!: team-psf / \ team-gsoc :!: for having followed up and marked the development, which was very important for our performance.

To improve my knowledge I could not have had more luck working with the pyafipws project, seeing the code you can see the great handling of the python language, it runs through it and being complex at the same time, with which you can learn a lot about the management of it and its tools, my congratulations to Mariano Reingart the author / maintainer of this opensource project.

I don't want to leave the opportunity to mention an excellent book,
which is so far the best one I read and which has become author In my referent, very complete, extraordinary handling of the writing and explanation connecting all topics, using experiences and problems of my own in work done and the star is 'dive in to Python3' its author Mark Pilgrim, to whom I give my thanks.
and to the translator of his Spanish version Jose Miguel Gonzalez Aguilera.
PS: that you have read it does not mean that you have learned :)

Well, I leave you. I have to finish the tests!

"""

Regards and Good Life!

Weekly Summary #12

nicolassandovalunm@gmail.com (Sando) — Mon, 12 Aug 2019 03:17:23 +0000

Hello everyone!

"""

What did you do last week

I was working on the web service testing with ladon the idea is also to do it with django + spyne and flask of which I was generating some basic tests.
I was also making some progress in the tests.

This week

The idea is to finish the basic web service leaving aside some
more advanced implementations, finish the wsaa mock which is the web service that generates the access ticket to be able to use the other web services and wsbfev1 to lay the foundations
of the other developments of the mock-ws.
Also, finish the highest amount of test with high coverage.
At the end of the week, see with the mentors details and suggestions of some changes in the project.

Difficulties

One of the attributes for which I decided work with ladon, is that it contains a graphical interface to test the web services created, similar or similar to soapUI which I would have liked to try but I could not because I hung up the system, I only say so I could see and read from your documentation.
It is costing me the implementation of the clients with pysimplesoap due to my lack of experience,
which led me to investigate other resources and I found a SOAP client programmed in python called zeep.
Which I was testing, has a simple use and good documentation, I could interact with the basic test web services created.
I remembered that one of the mentors had suggested changing this dependency, but I wasn't sure which one to use, I think Zeep would be a good candidate but we would have to prove it more deeply and see how it works with security implementations.

"""

Regards and Good Week!

Ladon, web services-last weeks

nicolassandovalunm@gmail.com (Sando) — Mon, 05 Aug 2019 18:01:15 +0000

Hello everyone!

"""

The last two weeks I was seeing some topics in the tests
an improvement in my initial file on travis. Then I was seeing tools to go testing.
In which I was watching soapUI, the handling of WDSL.exe with visual studio.
I was also seeing ladon and the truth seems to me an interesting tool, simple to use and very descriptive.

But, after struggling to be able to correct it, since in Linux probe of everything after the installation, and it still didn't work, the next day I decided to try it back and it was perfect.
Sure, I needed to reboot to apply the changes. (In windows I still couldn't)

Only 2 weeks left to finish and I am already feeling a little nostalgic.

There are some very interesting topics I was seeing, some related to Apache.

I could see that in a moment in the .local / share folder on the linux mint platform an .xbel file began to appear.
Containing?
The description of each new file generated on the machine in xml format and the address points to freedesktop.org.
Let's see if I understand, something is generating this file now?
Why does the GTK need this information?Its strange.
For now this is not relevant but I wanted to share it, I will be seeing how to delete it after the end of gsoc with psf.

Note: I was able to run the Apache 2.4.9 server on Linux but with error 403.

I could also find how to fix the unicode error when trying to decode using.
.decode ('unicode_escape'), instead of .decode ('utf-8')
It is not yet implemented.

I will advance as much as I can with the tests, and the web service testing with ladon.

"""

Regards and Good Week!

Weekly Summary #10

nicolassandovalunm@gmail.com (Sando) — Mon, 29 Jul 2019 22:55:15 +0000

Hi everyone!

""""

What did you do last week

In this week I was working on the development of the web service for testing, also advanced in the wsmtxca test of electronic billing with details and an improvement in the functionality with travis.

This week

In this week I will continue with the development of the web service, move forward with the project tests and correct unicode runtime problems.

Comments

In the development of the web service I was seeing several projects to test the clients for web service, such as soapUI written in java, this hang up my system it seems that I and Oracle are not compatible, summary, I could not use it until now. Seeing tools in python like Flask with which I ran simple tests and ladon to handle a web service development tool in which I could not achieve much yet. But nothing more complicated than the development itself.

"""

Regards and Good Week!

Travis, New North

nicolassandovalunm@gmail.com (Sando) — Mon, 22 Jul 2019 17:42:05 +0000

Hello everyone!

"""

Monday, first of July-.I'm finishing with the tests in a module was 0000, I think great, this is run correctly. After the tests I decided to send the commit with the last part missing, they had done 02:00 am, I have to finish this tomorrow I have a meeting with the mentors, I thought. Ready finished task, what comes next.
After the meeting there are tasks to be done and my mind is already headed towards a new direction.
I solve some of the issues and I put myself with the central objective of the project, to incorporate travis to the pyafipws project. Again in my mind I think, -well this is easy I generate the file .travis.yml some tests and ready continuous integration run in the repository. . . It had been a week and I was still doing tests, collecting data and seeing what was the best way for Travis to work with the project. There were important points that had not taken into account which were generating conflicts for the deployment, was a constant error, error, failed... then automatically activate travis notifications mail, not only see that it not work, but now also remind me this by mail. I deactivate the option, it is a useful tool to know how the constructions are working.

A first problem occurred when trying to use the certificate with the key to generate the access ticket for the Afip servers, to generate at least the TA(access ticket) for the tests to work since they are made to interact with the test servers of Afip. Without TA, there is no test.
To test, I added the certificate and the key to the directory, for this I had used one of the web service tests of afip the client wsbfev1.

Result, empty access ticket!

Obviously this is returned when the ticket can not be generated, the causes can be varied.
well, but the idea was that both the certificate and the RSA key were not within the repo. Then I see the possibility of using the environment variables that travis offers from the configuration of its web API.
This generated a problem in pyafipws, since the content is used for the ticket request as a file, not as a variable, in order to use it in this way, for now we must extract the environment variable and generate the necessary files, a modification is made in the test for use with the variables.

Result, empty access ticket !.

Well, what is happening?
How are you constructing travis and how are the variables returned in the construction of the virtual machine?
Well for this travis has an explicit notes on how to place it if they have special characters.
But if there are spaces like the ones he takes and the spaces he takes but which are line breaks ('\ n') good for this my mentor @reingart had some very good suggestions to handle it.

Result, empty access ticket!

Then I start to doubt the travis environment variables and perform simple tests, they work correctly. I perform tests to adjust how the variables are returned and the result
expected for the tests to work.
Good results are achieved the variable returns the expected file in the construction.

Result, empty access ticket!

But hey, it was a breakthrough, travis was giving me back what I needed.
A question that arose was if the AFIP servers had origin detection and if they were not from Argentina
there they were blocked, let's know where the servers of travis, amazon, googlecloud ...?
I change the test and start using the module that generates the step-by-step access ticket in the methods
and I find something interesting. At the beginning of the process a tra that is an XML that is dated is required
of generation and expiration in the range of one hour (example: generated at 12:00 and expires at 13:00)
Immediately I think clearly what is the schedule that this generates in the VM travis, correct the server of travis has a different schedule. This is configured in the travis file and the correct time in the generated tra.

Result, empty access ticket!

We must continue investigating.
In the build of travis when I get to the loginCMS test which is the method that returns the TA,
everything in light red, until that moment I had configured the .yml with the tests for versions 3.5, 3.6, 3.7, 3.8 of python in three empty access ticket meeting !, one version appeared in red but it was the TA! how I had configured the test to show me what I returned and not the assert (example: assert TA == <anything>). Then I perform the test to test the status of the servers of afip, this time configured with the expected response and EUREKA .</anything>

Three constructions with empty access ticket and one in green with PASSED test.

Now why was this happening ... and there I agreed, that when I was making the adaptation for cryptography in the test response of the AnalyzeCertified method I saw that this is dated on 07-13-2019, what a coincidence? Maybe this was not the problem itself. When I told my mentor that they were no longer able to carry out the tests he said there was a problem with that then he was going to generate an exclusive certificate for the project because the same certificate can not be used at the same time to require an TA, he has to see a time of 10 minutes. The access ticket gives you a 12-hour use but once generated it can not be used again from another place until after 10 minutes.
Of course a 10-minute lapse in time, the other constructions fail, the servers do not authorize it, the first one asks for access, the rest do not obtain.
Well to test this I remove python test versions and leave only one.
I start to try the other tests with the modifications and excellent everything in green except for some issues. It was perfect the things working well .
It was Thursday, google mail with reminder of second evaluation, that it was being evaluated and that the mentors had to go see the PR and that it was not useful to send everything last moment.
I start to think and this was correct PR only 2 in the first week, but from the tasks of the meeting the rest of the tasks performed only in commits to the issues because if it generated the PR I did not know how they would affect the mergeability of these since they were changes that went through the whole project I was missing the PR of cryptography
I remember that at the time I had not done it because one of the mentors asked me for the module test to check the functionality of the new unit. I generate the PR of the dependency, a DRAFT PR with the content of the evolution of travis in the project, the PR of the file .travis.yml.
Remembering this, I think, -we test if it works in travis, replacing the old one with the new WSAA that is the generator of access tickets if it does not work, all the tests are dropped.
Perfect performance! I start generating the settings of the test_wsaa and it occurs to me to try the module without the dependency so I remove cryptography from requirements.txt

Result, empty access ticket!

Well I will wait, because another problem is that I use the same certificate on my local machine so if I do a test on my machine if 10 minutes have not passed, the seim problem is generated in the construction of travis.
20 minutes later I try again, the same result.
One of the advantages that pyafipws offers is that without having the M2Crypto or cryptography dependencies installed, it gives the possibility to generate the TA directly from OpenSSL.
I test on my local machine without cryptography and it didn't work. I realized that the error was in declaring a variable that acted as a condition, since this was taken as true when the dependency was installed but there was no contrary condition for this to execute correctly.
Why had this happened?
I remembered, I tried both parts separately (with and without dependency) and when I did the test together I had only done it with the dependency installed, in my mind it was left, - everything works correctly. With which I realized that a new test is never too much.
After so much suffering I began to want to travis, it is a great tool now this branch helps me to test everything I am going to send in both commits and PR.

And now I'm going to deploy a web services simulator, I'll also continue with the tests in the process.

"""

Regards and Good Week!

Weekly Summary #8

nicolassandovalunm@gmail.com (Sando) — Mon, 15 Jul 2019 21:14:36 +0000

Hello everyone!

"""

What did you do?

In this week i was working on the deployment of travis for the implementation of continuous integration in the project, some problems were fixed in the initial file of travis.

What are you going to do?

I will finish with the deployment of travis and will continue to perform tests and modifications suggested to the project code

What problems did you have?

So far I could see Travis CI in a great tool for the development of the code, but so far the tests I can not run on their platform the same as in linux and windows, the problem is that the tests require a initial authorization that generates a valid access ticket with 12 hours of duration once obtained is saved in cache folder, with this you can run clients of web services, and each test with this authorization performs processes within the servers such as the generation of invoices, consultations, etc.
But for this AT to be generated, it needs to go through a process to give it the proper format for the initial web service to accept it, for which initially an RSA key and an x509 certificate are needed. which travis I can not get me to take to generate the AT.
It's probably a security protection issue in Travis.

"""

Regards and Good Week!

Pyca / cryptography and some other things

nicolassandovalunm@gmail.com (Sando) — Mon, 08 Jul 2019 21:26:52 +0000

Hi everyone!

"""

In the last two weeks it was decided to change m2crypto by pyca/crytography, well all very nice but I knew about cryptography ... little and nothing. I had read the code and more or less wise how it work. I was going to be a challenge quite high since the module in which I had to work is the backbone of the project (wsaa.py) which gives the functionality to the clients of the AFIP web services.
Well, I thought, - where we started now ... and I remembered a set of letters, RTFM, then I went directly to Eric Holscher's documentation page "Read the Docs" and found the doc of pyca/cryptography.
I read in the first example:

from cryptography.fernet import Fernet

And, the mere fact that this module has this name I like since it represents a delicious drink of common consumption in Argentina accompanied by a known soda.
There were four medodos that had to be changed and for this the "hazmat" module was used.
I had a sign at the entrance that suggested: enter at your own risk and that there could be dinosaurs with laser weapons, I said: Wow! This is for me.
And so I started to develop the methods (I invite you to follow the issue # PyAr # 15),
for the first 3 It reached me with the documentation (and a lot of testing) which left for the last was the most important since it contains the tra (an xml) and it must be signed together with PKCS7 and the private key + the certificate.
But I did not find anything at first glance that would lead me to results so I used another one of the most solid tools, the Stack Overflow and I found a response that could help me .Then I could see that the pyca / cryptography arrives "Bindings" and the high level code ends, I would have to use a low level code to achieve it, , although it was well detailed there were parts that did not let me work the code, since the service of Afip requests the data in a special way in smime format and only the body. I continue looking for an answer then.
After a lot of sailing and it was already a Monday I had spent the week dedicated, I had to finish.
I went straight to the repo of cryptography in github and found a very interesting (and long) issue that dealt with the subject and at the end of this encounter a commit that I start to read and it appears as magic the low level function that I was missing to change from pkcs7 begin to the smime.p7m header, with which I could use part of the final code that had been developed for m2crypto, which was nothing more than to cut this header and return only the body of the message. Howerever, I liked that it happened in this way.

Tuesday meeting with the mentors

In this meeting important topics are touched on how the month in the project had been that things were missing and that another could be added.
And the list was as follows:

Update requirements.txt
Pass the .py files all to utf-8.
Eliminate coding and shebang lines in the beginning.
Delete the modules that are going to belong to another project.
Remove py2exe from setup.py
Eliminate the unnecessary command line code from the modules.

I start to do the above mentioned until I get to the encoding to utf-8 I remember the words of my mentor when he explained to me and showed me with the "file" command the type of format that most tex / x-python had, UTF-8, with end of line in crlf.
With this command he showed me a world that I did not have in mind.
He told me it's simple, open and save in utf-8 and that's it.
Ok, I thought, it's easy. Until now I could not get my text editor to encode a new file into pure utf8.
Everything is saved in us-ascii, which is fine is a subset of Unicode, I say good should be a problem of the editor. I have configured the default options : "UTF-8".
I save the files with "save with encoding utf8" shows me the file as utf8,
but when I run :

file -i * .py

outcome: command, text / plain , us-ascii,

In this test I run from the platform windows 10 in the bash of git that I have installed.
In a virtual environment that I have linux mint I managed to change the project files to utf8 with the command

iconv -f (encoding) -t utf-8 file.py > file_out.py

But there was one nsis.py that was in us_ascii and there seems to be gods that change it probe with an option that I found that it was to change directly to utf-16 and after changing it to utf8 and it worked, utf8 at the end I say I put the command file and I get utf-8 with bom, (.-Ufs that don't see good), I read many negative comments of the BOM markers with java compilers (and that would have to do with python? I do not know, but just in case ...). I have to keep looking, I think it could be a platform problem.

I install linux mint directly on the machine. Until now I had the linux mint and ubuntu platforms in Virtual Box (At another time I will comment on the problems that I had with Oracle's VirtualBox)
but there were no solutions still turning everything into us-ascii
although in properties appear in utf8.
And the other thing I saw is that the line ends are for different platforms:

Unix lf
Windows drlf
Mac dr

Change editor, find some plug in, make some script ... are possible solutions that hover in my head.
Summing up the project worked before and works now . Nevertheless, I could not say if for these questions at some point something could fail. I will continue investigating since it seems an interesting and delicate topic.

In that week I also started with the implementation of Continuous Integration with a first advance with Travis CI and the initial file in my .travis.yml repository with test lines to check functionality, without good initial results. In the built of it, I received alerts of "deprecation of Python 2". I suppose it must be for the master branch and that in one of the dependencies I could not find the request module.

In this week that begins, the challenge is to be able to start using travis efficiently and go completing the necessary.

"""

Thank you and Good Week!

Weekly Summary #6

nicolassandovalunm@gmail.com (Sando) — Mon, 01 Jul 2019 16:40:28 +0000

Hi there, How are you?

"""

What did you do last week

In the previous week is they were doing tests on the modules.
As it was managing the possibility, by the instability in the installation in different platforms, it was decided with the mentors to change the dependency m2crypto by pyca/cryptography for the management and signature of the certificates. Substitution was completed in 3 methods: analyze certificate, create private key, create CSR (certificate signing request).

This week

It will end with the substitution m2crypto dependency by cryptography.
This week, i will begin with the implementation of continuous integration through travis CI in the project.

Comments

This week there are no problems to report.

"""

Regards and Good week!

Weekly summary #5

nicolassandovalunm@gmail.com (Sando) — Mon, 24 Jun 2019 15:57:58 +0000

1013/5000

Hello everyone, how are you?

This is my weekly summary.

"" "
What did i do last week

This week multiplatform checks were made, the tests were continued trying to advance with the coverage.

This week

For this week the tests will continue and the operation of the m2crypto unit will be analyzed with the project

Comments

I had a warning running the tests with unittest.

ResourceWarning: unclosed ssl.SSLSocket

Apparently the connection would not be closing during the execution of the tests and I found this,
because Python requests a design option, there is a warning about a connection that is not closed when running the tests:
https://github.com/kennethreitz/requests/issues/3912
A solution would be, suppress the warning as discussed in:
https://github.com/kennethreitz/requests/issues/1882
"" "
Regards and Good week!

Weekly summary #4

nicolassandovalunm@gmail.com (Sando) — Mon, 17 Jun 2019 15:30:11 +0000

Hello, how you there! this is my weekly summary.

'' '
What did you do last week

This week will be added to the dependencies configuration and the installation tests were run.
Minor arrangements were made in the imports
and the tests with unittest were improving.

This week

This week they will continue with the tests and starting to see how we will continue with the m2crypto dependency

Complication

I had problems with the installation of a dependence after running the installation tests, by the way it was installed through the dependency links, but then it was solved and the dependence on the installation is 100% functional.
'' '
Greetings and Good week!

Weekly summary #3

nicolassandovalunm@gmail.com (Sando) — Mon, 10 Jun 2019 14:39:37 +0000

Hi there, how are you? I leave the summary of the week.

'' '
What did you do last week

This week is doing the updates in the pyk3 branch, testing.
in each advance.
Test tools and code for the progress of the project.

This week

This week we will continue with the web services test, add some missing updates and add the dependencies to setup.py

Problems found

I was having problems in the system exit in Windows,
running some tests but nothing out of the ordinary in sys.stdout to see information of the parameters received in the xml,
and some problems with the web service on external test servers
With the help of the mentors the problems were solved.
I also received suggestions on the incorporation of tools to improve the quality of the commitments and the project.
'' '

Regards and Good week!

Advance

nicolassandovalunm@gmail.com (Sando) — Mon, 03 Jun 2019 14:37:24 +0000

Weekly summary #2

nicolassandovalunm@gmail.com (Sando) — Mon, 03 Jun 2019 14:33:41 +0000

Hello, how are you all! Here I leave the summary of last week.

'''
What did I do this week

In the previous week we searched for changes in the main branch that were missing in py3k around 20 commits to update.
They were also doing checkups of the web services electronic invoices with the PDF generation module with real communication against the Afip servers.
A new issue came from one of the members of the PyAr community @Gilgamezh, to directly add the dependencies to the setup.py. With which, as a first step what I did was to check these dependencies, although this was part of the project in week 3, I thought it was important to see it, tests were made of intalacion and functionality in the environments Windows 10 and Linux Mint.

For next week

This week we will incorporate the necessary changes in the py3k branch depending on the results of the tests. On the other hand, we will be moving forward with the unit tests trying to reach as much coverage as possible in the web sevices to be tested functionality This week the necessary issues will be added to complete the first stage.

Complications

The most complicated part was when they tested the facilities of the dependencies which are in a requirements.txt which installed only 1 of the 6 (minimum) (platform of first test Windows 10), it was discovered that the problem was generated for the dependency m2crypto not only was not installed but it broke the intalaciones of others when the command was executed:
⦁ pip install -r requirements.txt
This dependence being one of the topics to be discussed later in the project, we proceeded to omit m2crypto in the requirements.txt and the dependencies were installed correctly. Like Pyafipws can be used directly installing openSSL, after the meeting with the mentors, I remain clear the need to evaluate if we continue using this dependency or some other as cryptography which will be reviewed at the time.
'''

Thanks !, it will be until next week.
Good week!

Start first week - GSoC with PSF

nicolassandovalunm@gmail.com (Sando) — Tue, 28 May 2019 19:42:59 +0000

Hey there!, I'm Nicolás Sandoval from Argentina.

This week I start with the first part of my project in PyAfipWs!