Just to the panelists: FYI - https://tinyurl.com/ML-panel has my thoughts on deploying models along three axes:(1) the philosophy or conceptual steps [i.e. the delivery science],(2) the process or sequence of actions [i.e. the SOP, MINIMAR checklist for training datasets, etc],(3) the mechanics or nuts and bolts [i.e. what database, what API, what server, send email or use Epic API etc].
The first slide will be shown (by Jonathan) when introducing the discussants.
Is it possible to be get the slides or the recording?
Sessions is being recorded. Should be posted online in a few days at https://bmir.stanford.edu/education/colloquia.html
Thanks a lot
Matt: One of the things we have in our data is insurance type e.g. medicare or hmo or captitated. It may be possible to check for equity using that. One of our other collaborators, Stephen Lin, has been asking these questions.
thats a great idea
we have an ongoing project on that topic with mit and emory - lots of interesting findings with our models
tina hernandez boussard
Using insurance is a great start. However, you have to be careful using these proxy variables when you start looking into bias and fairness! ;)
Which parts of this happen on the school side vs the hospital side? Or do you have a joint team across school and hospital
@Jomol, we are moving towards having joint teams. Having our IT unified under TDS has been a good step forward.
tina hernandez boussard
MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health careTina Hernandez-Boussard, Selen Bozkurt, John P A Ioannidis , Nigam H ShahPMID: 32594179 PMCID: PMC7727333
https://tinyurl.com/ML-panel has details on all three axes we talked about (the philosophy, the best practice processes, and the mechanics). The url should be viewable to anyone.
NIH’s AllofUs Research Program is another (explicitly) diverse EHR, survey and genetic database getting off the ground that might be of interesting for some folks
What are some ways to quantify the expected benefit/cost of adding an AI option to an existing workflow?
@Deen — check out https://www.medrxiv.org/content/10.1101/2020.07.10.20149419v1
@Wendi — Developments in public datasets are patchy. For COVID there is N3C by NIH and the Covidresearchdatabase by Datavant.Reg communicating up the org, the use case has to be distilled down to ROI and value statment.Finally, gettting data out from Vendor systems is something ot be handled at contracting. Get data access included in the purchasing.
Dr. Lungren mentioned this worry of model drift, where the relationship between variables that the model is relying upon might change over time. To what extent can these changes be “compensated” for via model retraining? And to make this a bit more concrete, what specific sources of drift might present more difficulties to a scientist trying to build a model that will stand the test of time?
great point distribution
yes a realistic view
@Priya — GOSSIS is a nice (and free!) alternative to things such as the APACHE IV score.
Great step, and in the right direction.
Joseph Paul Cohen
If the issue is shift then why not define things to prediction that we know won’t shift over time?