This week, I had a quick sync with the mentors as it seemed like I couldn't derive systematic results from running my previous experiments.

I liked the idea to introduce some noise to some dataset that has a low rate (e.g. less than 1-3% of misclassified labels to compare baseline with cleanlab. I used Fashion MNIST and introduced some random corruption to the training set by flipping the labels.

In one of my experiments, I set the maximum noise to 50% and gradually introduced 5% the noise at each step, comparing the performance of baseline and cleanlab in parallel. This time, instead of relabelling, I decided to prune the samples with low confidence scores. Here’s a quick example: if I have 60,000 samples in the dataset, at 10% of the noise, I’d randomly flip 6,000 labels. The baseline would then be trained with all samples (60,000), but the cleanlab would be trained only on the labels that weren’t classified as erroneous by cleanlab . For example, If cleanlab found that 5,000 are labeled incorrectly, then I would only use 55,000 images for training.

It seems that pruning the samples with lower confidence works well, as cleanlab seems to remove the labels that were introduced with each noise level. We can also see that the accuracy stays around 80% with cleanlab, while with the random noise (e.g. without removing any samples) it drops linearly. On average, I can also see that cleanlab  on average prunes more labels than I initially introduced in the data. The mean of additional samples that cleanlab discards is ≈4500 samples across all noise levels. Since I don’t know the true noise in the original dataset, it’s hard to say whether cleanlab is doing a good job on removing these, but I would argue that cleanlab seems to be overestimating. However, it seems to systematically pick up the newly introduced noisy labels and identify them as erroneous.

I was surprised that after introducing random noise, CL would still prune the erroneous labels. The way CL algorithm works is by accurately and directly characterizing the uncertainty of label noise in the data. The foundation CL depends on is that label noise is class-conditional, depending only on the latent true class, not the data . For instance, a leopard is likely to be mistakenly labeled as a jaguar . Cleanlab takes that assumption and computes joint distribution among different classes (e.g. 3% of the data is labeled leopard (noisy label), but the true label is jaguar). The main idea is that underlying data has implications for the labeler’s decisions, and I basically took this assumption out of equation after randomly swapping labels in the dataset (e.g. I didn’t care if a certain class would be more likely to be mislabelled as another class). I would think that CL algorithm relies on this assumption heavily to guess which label to prune, but the performance was still accurate and stable. In the real-world noisy data, the mislabelling between different classes would have a stronger statistical dependence, so I can say that this example was even a bit more difficult for cleanlab.

I’ve tried to experiment with different threshold values for pruning and relabelling the images (e.g. remove 20% of the images with lowest label quality, but leave and relabel the rest). I’ve started with a threshold of 0% (e.g. relabel all labels to the ones predicted by cleanlab ) and then gradually increased the threshold value with a 10% step till I reached 100% prune level (e.g. remove all labels that were found to be erroneous by cleanlab). As before, I run these from 0 to 50% of the noise level.On the graph, I plotted the accuracy of the models trained with training sets that were fixed with different threshold values. For example, 100% Prune / 0% Relabel indicates the accuracy of the model when all erroneous samples and their labels were deleted, while 0% Prune / 100% Relabel shows the accuracy of the model when all of the samples were left but relabelled.Looking at the graph, I can say that cleanlab definitely does a great job at identifying labels, but not necessarily at fixing them automatically. As soon as I increase the % of labels that I’d like to relabel, the accuracy starts to go down in linear way. The training set with 100% of pruning got the highest accuracy, while the training set with all labels relabelled got the worst accuracy on the fixed model.As a next step, I can try to see what happens if we only remove a certain % of erroneous samples, but leave the labels of the other erroneous samples as they are. (edited) 

I also run this pipeline on the Roman Dataset (DCAI). This dataset is quite noisy, so there’s not a ton of improvement on 0-10% of the noise level. However, as I introduce more noisy labels, looks like cleanlab is still able to pick them up. Running a few more trials to see what’s the performance with different Prune vs Relabel threshold. 

This week, I worked on running the experiments on a variety of datasets. 

Here are a few exeriments that I did to benchmark cleanlab performance of three different datasets (MNIST Roman (DCAI), Flower 102, Fashion MNIST). For all experiments, I used a fixed model and applied resizing and normalization to the original images.

What do 1️⃣, 2️⃣, 3️⃣, 4️⃣ mean? I re-run the entire fitting with a a few random seeds to get an estimate of the variance between the accuracies of baseline and cleanlab.

1️⃣  = seed(0)

2️⃣  = seed(1)

3️⃣  = seed(123)

4️⃣ = seed(42)

First Training Run

Training with 20 Epochs

In the results below, I used epochs = 20 instead of epochs = 10. The rest of the parameters (e.g. Network, Transform) are unchanged.

Training with 30 Epochs

In the results below, I used epochs = 30 instead of epochs = 20. The rest of the parameters (e.g. Network, Transform) are unchanged.

Training with Resnet50

In the results below, all of the parameters stay the same as in run above, but this time I changed the network to resnet50() instead of resnet18(). Epochs are also back to epochs = 20.

Training with Validation Set

In the results below, all of the parameters stay the same as in run above, however, in this run I tried to run trainings with validation set. Therefore, 20% of the dataset is used for the internal training validation. In other datasets where the validation set exist, it’s used as a validation set. I set the model back to resnet18() as it seems it gives better baseline accuracies over three datasets.

Training with Validation Set (Stratified Sampling)

Using arbitrary random seed can result in large differences between the training and validation set distributions. These differences can have unintended downstream consequences in the modeling process. As an example, the proportion of digit X can much higher in the training set than in the validation set. To overcome this, I’m using stratified sampling (sampling from each class with equal probability) to create the validation set for the datasets where it’s not available by default (e.g. MNIST Roman, MNIST Fashion, KMNIST).

Training with Early Stopping

Notebooks to reproduce results:

• Roman MNIST

• Flower 102

• Fashion MNIST

What did you do this week?

This week, I've set up a pipeline for running experiments on a variety of datasets to compare the accuracies of baseline (the model trained on dirty data) and cleanlab (the model trained on clean data).

What is coming up next?

As a next step, I will be running experiments on a variety of datasets to benchmark the accuracy.

Did you get stuck anywhere?

Not really, it was a bit tricky to set up a pipeline in a way that's reproducible. I managed to overcome this by fixing the seeds.

This week, I've been focusing on the high-level overview of data-centric strategies. I used Roman MNIST Dataset that was provided in the Data-Centric AI Competition. For all experiments, I used fixed Resnet50 and applied resizing and normalization to the original dataset. I also used a fixed seed to be able to replicate the runs. Here are some of the metrics I’ve been getting:

1. Fixing Labels with Cleanlab.

• Baseline Accuracy: 0.7318
• + CleanLearning Accuracy: 0.8290

1. Automatic Augmentations with AutoAlbument (uses Faster AutoAugment algorithm)

• Baseline Accuracy: 0.7318
• + AutoAlbument Augmentations: 0.7404

1. Augmentations with Basic Augmentations and Pre-trained Torch Policies

• Baseline Accuracy: 0.7318
• Basic Baseline Augmentations: 0.8696
• ImageNet Pre-Trained Policy: 0.8560

Here are my findings for these strategies:

• Fixing Labels with Cleanlab.
  • I’m now trying out smaller k values for cross-validation to see to which extent it impacts the accuracy and improves the training speed. I will also try out more epoch ranges to see if this has impact on the accuracy. I’ve noticed that cleanlab performs well only when we already have a robust baseline model. For this specific dataset, I’ve noticed that if the accuracy of the baseline model is less than 0.7, then cleanlab actually has negative effect on the accuracy. I believe this is because of the confident learning algorithm, as it needs to get as accurate confidence scores for each label as possible.
  • For now, the labels fixing and augmentations are applied separately. I think it would be also interesting to see how the accuracy changes after applying CleanLearning and then AutoAlbument. But I haven’t found out an easy way to get the corrected labels from CleanLearning to overwrite the initial labels of the dataset. I’ve messaged the cleanlab team to get their help on this.
• Automatic Augmentations with AutoAlbument.
  • I’ve only used epochs = 15 to find optimal augmentation policy. I’ll try out more epochs ranges to see if this can improve the accuracy.
  • I have also used the whole train and validation datasets for finding the optimal augmentation policy. I then applied augmentations only to train dataset and validated it on the test dataset.
• Augmentations with Basic Augmentations and Pre-trained Torch Policies
  • I was surprised by the accuracy improvement after applying basic transformations, such as HorizontalFlip(), RandomCrop(), and RandomErasing().
  • I only applied augmentations to the train dataset.

What did you do this week?

This week, I've been focusing on implementing custom cross-validation algorithm to compute out-of-sample probabilities for Hub Datasets.

What is coming up next?

As a next step, I will be working on a generic high-level pipeline that consists of fixing labels on a particular dataset, applying augmentations to a dataset and finding slices that underperform on a particular dataset.

Did you get stuck anywhere?

Not particularly, I had a few issues with ensuring that my experiments are deterministic with PyTorch but at the end I was able to fix it.