Week-02

I've finished my exams in this week, and return to work on benchmark.

Benchmark: 

• Using:
  • Benchmark will work in a generic way to compare between LPython and Python compilers in any stage you want (but now i implemented parsing stage, if any one want to add other stage to compare, it will be easy to be added).
  • Files structure is similar to testing strucutre:

    .
    ├── benchmark
    │   ├── benchmark.toml
    │   ├── long_statment.py
    │   ├── report_file.dat
    │   ├── test_math.py
    │   └── very_long_statment.py
    └── run_benchmark.py
    

    we will add files we want to compare to benchmark dir and we will add file in toml file and specify what stage we want to compare: 

    #Possible comparisons
    # parser: Parsing time
    
    [[benchmark]]
    filename = 'test_math.py'
    parser = true
    other_stage = true
    
    [[benchmark]]
    filename = 'long_statment.py'
    parser = true
    
    [[benchmark]]
    filename = 'very_long_statment.py'
    parser = true
    other_stage = true
    

    here we have files, and every file has comparison stages

    we can imagine files like this:

    

     

  • so parser comparison set has very_long_statment.py, long_statment.py and test_math.py, thats mean we will compare between Lpython and python on (Parsing) stage by these files.

  • other_stage comparison set has long_statment.py, that mean we will compare betweeen LPython and python on (other_stage) stage by this file.

  • Output will be table or pars, and you will have args flags to choose what you want:

    $ python3 run_benchmark.py 
    		 parser comparison 
    LPython is first raw, Python is second one.
    Time in ms.
    
    benchmark/test_math.py         : ▏ 237.24T
                                     ▏ 741.59T
    
    benchmark/long_statment.py     : ▏ 3.77 
                                     ▇▇ 22.98
    
    benchmark/very_long_statment.py: ▇▇▇ 39.04
                                     ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 570.24
    
    
    File Name                          LPython      Python
    -------------------------------  ---------  ----------
    benchmark/test_math.py            0.237236    0.741587
    benchmark/long_statment.py        3.77516    22.9778
    benchmark/very_long_statment.py  39.0398    570.236
    

    $ python3 run_benchmark.py --help
    usage: run_benchmark.py [-h] [-n] [-p] [-c COMPARE [COMPARE ...]]
    
    Lpython benchmark
    
    options:
      -h, --help            show this help message and exit
      -n, --numerical       show results as numerical table
      -p, --plots           show results as graph of pars
      -c COMPARE [COMPARE ...], --compare COMPARE [COMPARE ...]
                            What stages you want to compare, for now we have['parser']
    

• Implementation:
  • For every stage we want to implement we should make a class for the stage and make two statics functions for get time(or any value we wnat to compare) for Lpython and python

    class Parser:
        @classmethod
        def get_lpython_result(cls, file_path):
    
            lpython_run = subprocess.Popen("lpython --new-parser --time-report " + file_path, shell=True,
                                           stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    
            try:
                stdout = lpython_run.communicate()[0].decode('utf-8')
                parsing_value = re.search(r"\bParsing: .*ms\b", stdout)[0]
                parsing_value = parsing_value.replace("Parsing: ", '')
                parsing_value = parsing_value.replace('ms', '')
                parsing_value = float(parsing_value)
    
            except:
                parsing_value = None
    
            return parsing_value
    
        @classmethod
        def get_cpython_result(cls, file_path):
            input = open(file_path).read()
            t1 = clock()
            a = ast.parse(input, type_comments=True)
            t2 = clock()
            return float(t2 - t1) * 1000
    

    here parser class implementation.
  • I used time-report for getting times of stages in LPython.
• Related pull requests:
  • Basic benchmark #684