Human Expertise in Algorithmic Prediction
Highlights
-
Medically related, which I always enjoy.
-
The test for seeing whether or not human experts encode "side information" can be applied to a lot of domains.
Summary
This paper introduces a novel framework that selectively integrates human expertise into algorithmic predictions for subsets of data that are algorithmically indistinguishable. Although algorithms often outperform human counterparts on average, human judgement can improve algorithmic predictions on specific instances. Human experts within specific domains (e.g. X-ray diagnosis) can form judgments by drawing on information not encoded into the algorithm's training data. Hence, the author's approach provides a natural test for whether or not experts encode certain "side information" into these judgments.
Key Contributions
-
A novel framework that selectively integrates human expertise into algorithmic predictions when data is algorithmically indistinguishable
-
An operational test for whether human experts encode "side information" not accessible to algorithms.
Strengths
-
Paper is fairly rigorous, I like the incorporation of multicalibration.
-
Empirical validation within a real-world domain. For this paper, they used x-ray diagnosis and visual prediction tasks.
-
Shows robustness to different user compliance patterns
-
Amazingly, they wrote pages of solid mathematical proofs without a single instance of "clearly", "obviously", "it's easy to see" or "trivially".
Weaknesses / Questions
- I wish multicalibration had a few more lines of text about it, though I think it's because I lacked the background in it. I did have to familiarize myself with it a bit for section 3 onwards to make sense.
Related Work
- Multicalibration: https://arxiv.org/abs/1711.08513