Abstract: Numerous methods have been proposed as defenses against adversarial examples in question answering (QA) tasks. Despite appealing to various principles, these techniques are often model specific, require retraining of the model, and give only marginal improvements in performance over vanilla models on popular adversarial QA datasets. In this work, we present a simple model-agnostic approach to this problem that can be applied directly to any QA model without any retraining. Our method employs an explicit answer candidate reranking mechanism that scores candidate answers on the basis of their content overlap with the question before making the final prediction. Combined with a strong base QA model, our method can outperform state-of-the-art baselines, calling into question how strong these adversarial testbeds are and how well sophisticated defense techniques are actually doing.