This article introduces a re-labelled, controlled dataset designed for uncertainty exploration. We briefly cover the following:
Motivation
Dataset Curation
Dataset Overview
Dataset Description
Quick access via Hugging Face
Multi-modal extension: text
For more details, please check out the paper itself.
Yet another digit dataset , in this day and age?
Deep learning research keeps scaling, but massive datasets can introduce confounding factors that make understanding fundamental principles of deep learning rather challenging. While modern neural networks do really well on clear, unambiguous examples, their behaviour on examples that humans themselves can’t agree upon is still poorly understood. When a datapoint is genuinely ambiguous (e.g. a digit that could be either a 4 or a 9) , how should a model learn and evaluate such data?
image by author
A model predicting a 50/50 split could look like indecision, but it might just be the model confidently reporting that the answer is ambiguous.…