GSoC Blog | Activeloop | Week 5

This week, I've been focusing on the high-level overview of data-centric strategies. I used Roman MNIST Dataset that was provided in the Data-Centric AI Competition. For all experiments, I used fixed Resnet50 and applied resizing and normalization to the original dataset. I also used a fixed seed to be able to replicate the runs. Here are some of the metrics I’ve been getting:

1. Fixing Labels with Cleanlab.

• Baseline Accuracy: 0.7318
• + CleanLearning Accuracy: 0.8290

1. Automatic Augmentations with AutoAlbument (uses Faster AutoAugment algorithm)

• Baseline Accuracy: 0.7318
• + AutoAlbument Augmentations: 0.7404

1. Augmentations with Basic Augmentations and Pre-trained Torch Policies

• Baseline Accuracy: 0.7318
• Basic Baseline Augmentations: 0.8696
• ImageNet Pre-Trained Policy: 0.8560

Here are my findings for these strategies:

• Fixing Labels with Cleanlab.
  • I’m now trying out smaller k values for cross-validation to see to which extent it impacts the accuracy and improves the training speed. I will also try out more epoch ranges to see if this has impact on the accuracy. I’ve noticed that cleanlab performs well only when we already have a robust baseline model. For this specific dataset, I’ve noticed that if the accuracy of the baseline model is less than 0.7, then cleanlab actually has negative effect on the accuracy. I believe this is because of the confident learning algorithm, as it needs to get as accurate confidence scores for each label as possible.
  • For now, the labels fixing and augmentations are applied separately. I think it would be also interesting to see how the accuracy changes after applying CleanLearning and then AutoAlbument. But I haven’t found out an easy way to get the corrected labels from CleanLearning to overwrite the initial labels of the dataset. I’ve messaged the cleanlab team to get their help on this.
• Automatic Augmentations with AutoAlbument.
  • I’ve only used epochs = 15 to find optimal augmentation policy. I’ll try out more epochs ranges to see if this can improve the accuracy.
  • I have also used the whole train and validation datasets for finding the optimal augmentation policy. I then applied augmentations only to train dataset and validated it on the test dataset.
• Augmentations with Basic Augmentations and Pre-trained Torch Policies
  • I was surprised by the accuracy improvement after applying basic transformations, such as HorizontalFlip(), RandomCrop(), and RandomErasing().
  • I only applied augmentations to the train dataset.