Paper: https://openreview.net/pdf?id=HJz6tiCqYm
Summary
This paper observes that a major flaw in common image-classification networks is their lack of robustness to common corruptions and perturbations. The authors develop and publish two variants of the ImageNet validation dataset, one for corruptions and one for perturbations. They then propose metrics for evaluating several common networks on their new datasets and find that robustness has not improved much from AlexNet to ResNet. They do, however, find several ways to improve performance including using larger networks, using
Quality: The datasets and metrics are very thoroughly treated, and are the key contribution of the paper.
Some questions: What happens if you combine
Clarity: The novel validation sets and reasoning for them are well-explained, as are the evaluation metrics. Some explanation of adversarial logit pairing would be welcome, and some intuition (or speculation) as to why it is so effective at improving robustness.
Originality: Although adversarial robustness is a relatively popular subject, I am not aware of any other work presenting datasets of corrupted/perturbed images.
Significance: The paper highlights a significant weakness in many image-classification networks, provides a
Question: Why do authors do not recommend training on the new datasets?