Questions posed to the FDA on metrics for AI evaluation

Here’s a letter I wrote recently to an FDA director regarding AI metrics. (I’m not sure he was happy with the question, but ?) Presumably this is public record, so I will share it with the AI/ML/DL community. We’re still at Sensitivity/Specificity and ROC curve / AUC ROC, but concepts of imbalanced classes, etc. are becoming better established. What gets measured gets optimized, so I have been quietly trying to be a proponent of improved metrics, i.e. those that account for desired effects and bias. These are things that I think we need to start thinking about as we move forward on AI adoption in imaging and medicine. Posted for everyone’s enjoyment.