Articles on Ishaanj18's Blog

Weekly Blog Post #18

ishaan454@gmail.com (Ishaanj18) — Thu, 28 Sep 2023 05:23:14 +0000

Created pull request PR#234 and completed a basic implementation of the aritificial information fillter.

What is coming up next?
Will optimize and try to make the function more generalised.

Weekly Blog Post #17

ishaan454@gmail.com (Ishaanj18) — Thu, 28 Sep 2023 05:04:45 +0000

Integration of Artificial Information Filter: One of the key tasks I tackled was integrating the artificial information filter as a parameter within the get_keepbits function. This addition represents a significant step forward in making the function more versatile and adaptable to different datasets and use cases. The filter allows us to selectively identify and manage artificial information within the data, contributing to improved data quality.
Challenges with Different Dimensions: During the implementation of the artificial information filter, I encountered some issues related to handling datasets with varying dimensions. Ensuring that the filter works seamlessly across datasets of different sizes and structures is a complex task, and I dedicated time to address these challenges effectively.

What is coming up next?
Will complete the addition of filter to get_keepbits function.

Weekly Blog Post #16

ishaan454@gmail.com (Ishaanj18) — Thu, 28 Sep 2023 04:55:39 +0000

What did I do this week?

Introduced a new parameter to my function, namely the "threshold." By default, this threshold is set at 70%. This parameter is crucial as it helps determine where the keepbits should be positioned within the dataset. Specifically, it ensures that keepbits are strategically placed when the information content of a variable reaches 70% or more. This addition adds flexibility and adaptability to the function, allowing it to better manage keepbits in a dynamic manner.

What is coming up next?

In the coming weeks, I plan to further refine and test these updates to ensure the function's robustness across various datasets and scenarios.

Weekly Blog Post #15

ishaan454@gmail.com (Ishaanj18) — Wed, 06 Sep 2023 02:32:43 +0000

What did I do this week?

Worked on variables like like E2,Q2,QVAPOR,RH2,SH2,MLCAPE,PSFC,T2,TRAD,TSK,TLP,U10,V10. The function which returns true keepbits is working for the decoded DataArray for most variables but in some cases the true keepbits are not coming right for original DataArray .

Weekly Blog Post #14

ishaan454@gmail.com (Ishaanj18) — Tue, 29 Aug 2023 11:52:52 +0000

What did I do this week?

Over the past week, my focus was on delving into the CONUS404-Monthly dataset, where I didn't see aritificial information in variables in the dataset. My function to check aritificial information worked fine for majority variables returning plus minus 1 keepbit.

What is coming up next?

Will work towards increasing the efficiency of my function.

Weekly Blog Post #13

ishaan454@gmail.com (Ishaanj18) — Tue, 22 Aug 2023 04:21:28 +0000

What did I do this week?

This week was dedicated to fine-tuning the function responsible for calculating true keepbits, a critical aspect of our data analysis. I focused on experimenting with different values of tolerance to achieve accurate results. Despite my efforts, I encountered some challenges. Although the function produced correct keepbits for the decoded dataset, it still struggled to yield appropriate keepbits for the original dataset in some cases.

What is coming up next?

In the upcoming week, I plan to take a deeper dive into the function's inner workings and experiment with different strategies to improve its accuracy for the original dataset.

Weekly Blog Post #12

ishaan454@gmail.com (Ishaanj18) — Tue, 15 Aug 2023 10:34:59 +0000

What did I do this week?

Throughout this week, I focused on enhancing our understanding of the datasets and exploring potential artificial information. I approached this task from two angles: Firstly, I experimented with a new method of chunking variables. By modifying the way variables are chunked, I aimed to uncover any hidden patterns or anomalies that might indicate the presence of artificial information. Secondly, I leveraged the power of matplotlib to visualize the data on a map. I plotted the variables' data points onto a map, providing a spatial perspective that could help reveal geospatial trends or irregularities.

What is coming up next?

In the upcoming week, I plan to delve deeper into the findings from these experiments.

Weekly Blog Post #11

ishaan454@gmail.com (Ishaanj18) — Mon, 07 Aug 2023 18:06:19 +0000

What did I do this week?

During this week, I made a significant improvement to my function's methodology for removing artificial information from datasets. Instead of monitoring the frequency of CDF values and selecting the mantissa bit with the highest frequency as the keepbit, I adopted a gradient-based approach. I introduced a calculation to determine the gradient at each mantissa bit, and then identified the bit where the gradient becomes zero as the keepbit. This new approach proved to be more accurate in identifying and retaining relevant data while effectively removing artificial information

What is coming up next?

Will introduce maps to showcase the distribution of variables across United States and add a feature to showcase the true keepbit in bitinformation plot.

Weekly Blog Post #10

ishaan454@gmail.com (Ishaanj18) — Mon, 31 Jul 2023 18:58:00 +0000

What did I do this week?

During this week, my main focus was on testing the function responsible for removing artificial information from various datasets. The function utilizes two essential parameters, namely the Cumulative Distribution Function (CDF) and bit information, to accurately identify and eliminate artificial information, returning the true keepbits.
I then applied the function to each dataset and carefully analyzed the behavior of the CDF and bit information for each case. By closely examining the CDF, I identified instances where it started becoming constant. In such situations, I utilized the function to cut out the trailing bits beyond the point where the CDF stabilized. Furthermore, I observed datasets where the bit information started becoming zero. For these cases, I utilized the true keepbits to truncate the trailing bits, effectively eliminating the artificial information from those datasets.

The insights gained from this testing phase will be crucial as we move forward to integrate the function into our data processing pipeline for our project.

What is coming up next?
Will try to tackle artificial information in variables where artificial information dosen't necessarily pop in just trailing mantissa bits.

Weekly Blog Post #9

ishaan454@gmail.com (Ishaanj18) — Tue, 25 Jul 2023 04:18:08 +0000

What did I do this week?

During this week, I focused on addressing the issue of artificial information in datasets, specifically targeting cases where it appears in the trailing mantissa bits. To tackle this problem, I employed two parameters, namely CDF (cumulative distribution function) and bitinformation, to develop effective mitigation strategies.
For variables where the CDF starts becoming constant, I utilized this behavior as an indicator of potential artificial information. By identifying the point where the CDF becomes constant, I considered it as a threshold and removed the trailing mantissa bits beyond that point.
In cases where the bitinformation starts to become zero, I employed this observation to determine the keepbits. The point where the bitinformation becomes zero represents a potential threshold for distinguishing artificial information. By selecting the bit where bitinformation starts to decline and setting it as the keepbits, I discarded the trailing bits, assuming they likely contained artificial information.

What is coming up next?

Will try to tackle artificial information in variables where artificial information dosen't necessarily pop in just trailing mantissa bits.

Weekly Blog Post #8

ishaan454@gmail.com (Ishaanj18) — Thu, 20 Jul 2023 02:14:00 +0000

What did I do this week?

Throughout this week, I devoted my efforts to categorizing variables within the CONUS404 dataset based on the presence of artificial information. After categorizing the variables, I focused on developing effective methods to address the artificial information for each category.

What is coming up next?

Will try working on methods which will work on most of the categories.

Weekly Blog Post #7

ishaan454@gmail.com (Ishaanj18) — Mon, 10 Jul 2023 18:05:13 +0000

What did I do this week?

This week, I examined various variables of CONUS dataset to determine if any artificial information was present, and if so, whether there is some reemerging information for trailing mantissa bits or the information is never dropping to zero.

What is coming up next?

Will test the function which I have created to monitor artificial information using keepbits as check variable.

Weekly Blog Post #6

ishaan454@gmail.com (Ishaanj18) — Tue, 04 Jul 2023 16:46:44 +0000

What did I do this week?
This week, I dedicated my efforts to improving the efficiency of the function I developed, which automates the workflow of encoding, decoding, and comparing specific variables of the CONUS404 dataset. The primary goal was to optimize the function for better performance and effectiveness. To enhance efficiency, I carefully analyzed the code and identified areas where improvements could be made. I focused on streamlining the encoding and decoding processes.

What is coming up next?
Will try categorizing factors leading to artificial information popping up and try to come up with a solution to tackle it.

Weekly Blog Post #5

ishaan454@gmail.com (Ishaanj18) — Tue, 27 Jun 2023 18:52:17 +0000

Hello! My name is Ishaan Jain, a Information Technology undergrad at Manipal University Jaipur. I will be working on developing an information theoretic approach to filter out artificial information and real information in geospatial datasets for Xbitinfo during Google Summer of Code 2023.
What did I do this week?
This week, I focused on developing a function that automates a workflow for encoding, decoding, and comparing specific variables of the CONUS404 dataset. The primary objective of this function was to analyze and understand the presence of artificial information within the dataset.
To achieve this, I created a streamlined process that involved encoding the selected variables of the CONUS404 dataset, followed by decoding them. By comparing the resulting plots of the bitinformation from the original dataset, encoded dataset, and decoded dataset, I gained insights into the nature and origins of artificial information.
Overall, this week's accomplishments involved creating a function that automates the workflow of encoding, decoding, and comparing specific variables of the CONUS404 dataset. This function has proven to be a valuable tool in identifying and examining artificial information, leading to a deeper understanding of its origins and effects.
What is coming up next?
Will try categorizing factors leading to artificial information popping up and try to come up with a solution to tackle it.

Weekly Blog Post #4

ishaan454@gmail.com (Ishaanj18) — Mon, 19 Jun 2023 18:17:59 +0000

Hello! My name is Ishaan Jain, a Information Technology undergrad at Manipal University Jaipur. I will be working on developing an information theoretic approach to filter out artificial information and real information in geospatial datasets for Xbitnfo during Google Summer of Code 2023.

What did I do this week?
During this week, I focused on investigating the presence of artificial information within datasets, specifically using the CONUS404 dataset. To accomplish this, I followed a multi-step process. Firstly, I encoded the CONUS404 dataset using linear quantisation encoding technique. By applying encoding, I aimed to explore whether artificial information existed within the dataset and, if so, to gain insights into its nature and origin.
The CONUS404 dataset comprised various variables, including RAINCVMAX, PREP_ACC_NC, SNOWENERGY, COSALPHA ,LANDMASK etc. These variables offered a diverse range of hydro-climate and meteorological data, making them ideal for analysis and investigation.
Next, I proceeded to decode the encoded datasets to retrieve the original data. This step allowed me to compare the bitinformation (or information content) of the resultant datasets with that of the original datasets. By examining any differences or discrepancies, I sought to observe the potential impact of encoding on the information content of the dataset and identify any artificial information introduced in the process.

What is coming up next?
In the upcoming week, my plan is to try to come up with some ideas on how to tackle this issue of artificial information popping up in bitinformation analysis.

Did I get stuck somewhere?
I was stuck while plotting subplots of bitinformation of various variables using matplotlib and could not find a good way to get many bitinformation analysis figures in one plot.

Weekly Blog Post #3

ishaan454@gmail.com (Ishaanj18) — Mon, 12 Jun 2023 17:22:02 +0000

What did I do this week?

During this week, I successfully completed writing tests for PR#188 and focused on enhancing the functionality of the code by trying to add both keepbits and inflevels to the add_bitinfo labels function. Furthermore, I dedicated time in understanding few possibilties to identify artificial information in datasets. Started working on CONUS dataset to try to identify the how the artificial information is getting induced.
Overall, this week was marked by the completion of tests for PR#188, as well as the exploration of methods to detect artificial information in datasets. By actively working on the CONUS dataset, I am making progress towards uncovering the influences and effects of artificial information.

What is coming up next?
In the upcoming week, my plan is to observe and understand the development of artificial information using CONUS dataset and try to come up with some ideas on how to tackle this issue of artificial information popping up in bitinformation analysis.

Weekly Blog Post #2

ishaan454@gmail.com (Ishaanj18) — Mon, 05 Jun 2023 17:36:34 +0000

What did I do this week?
Throughout this week, I engaged in a range of activities. Firstly, I dedicated time to understanding linear quantization, including concepts like scale factor, offset and how it works on different datasets. In addition, I focused on writing tests for the function I contributed to in PR#188. Another area of exploration this week was researching different projections available in Cartopy.

What is coming up next?

In the upcoming week, my plan is to apply linear quantization to larger datasets, such as CONUS. By studying the effects of quantization on these bigger datasets, I aim to observe where artificial information is induced and evaluate the impact of quantization on data fidelity.

Weekly Blog Post #1

ishaan454@gmail.com (Ishaanj18) — Thu, 01 Jun 2023 06:46:36 +0000

Hello! My name is Ishaan Jain, a Information Technology undergrad at Manipal University Jaipur. I will be working on developing an information theoretic approach to filter out artificial information and real information in geospatial datasets for Xbitnfo during Google Summer of Code 2023.

What did I do this week?
This week was filled with diverse learning experiences. I started by delving into information theory, where I explored concepts such as mutual information, entropy, data compression. Understanding how information is measured, encoded, and transmitted provided me with a solid foundation in this field. Additionally, I dedicated time to studying the floating-point representation of numbers, which involved understanding how the placement of sign, exponent and mantissa bits takes place for different data types. This knowledge is crucial for ensuring numerical precision and accuracy in data analysis. Moreover, I deepened my familiarity with key libraries like xarray, matplotlib, and numpy, honing my skills in data analysis, visualization, and scientific computing. Lastly I also worked on PR#188 which involved adding inflevel labels based on keepbits. Overall, it was a productive week filled with expanding my knowledge in information theory, floating-point representation, and essential libraries.

What is coming up next?

In the upcoming week, my focus will be on writing tests for the function I contributed to in PR#188. Additionally, I plan to dedicate time to explore libraries like Cartopy, delving into their documentation to gain a deeper understanding of their functionalities and how they can enhance my data analysis and visualization tasks. Furthermore, I will be working with various datasets to investigate the effects of linear quantization on them. This exploration aims to observe how the process of quantization impacts the real information contained in the datasets and whether any artificial artifacts are introduced.

Did I get stuck somewhere?
I did not get stuck anywhere this week and was able to handle things.