Just to the panelists: FYI - https://tinyurl.com/ML-panel has my thoughts on deploying models along three axes:(1) the philosophy or conceptual steps [i.e. the delivery science],(2) the process or sequence of actions [i.e. the SOP, MINIMAR checklist for training datasets, etc],(3) the mechanics or nuts and bolts [i.e. what database, what API, what server, send email or use Epic API etc].

The first slide will be shown (by Jonathan) when introducing the discussants.

THanks!

Is it possible to be get the slides or the recording?

Sessions is being recorded. Should be posted online in a few days at https://bmir.stanford.edu/education/colloquia.html

Thanks a lot

Matt: One of the things we have in our data is insurance type e.g. medicare or hmo or captitated. It may be possible to check for equity using that. One of our other collaborators, Stephen Lin, has been asking these questions.

thats a great idea

we have an ongoing project on that topic with mit and emory - lots of interesting findings with our models

Using insurance is a great start. However, you have to be careful using these proxy variables when you start looking into bias and fairness! ;)

Which parts of this happen on the school side vs the hospital side? Or do you have a joint team across school and hospital

@Jomol, we are moving towards having joint teams. Having our IT unified under TDS has been a good step forward.

MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health careTina Hernandez-Boussard, Selen Bozkurt, John P A Ioannidis , Nigam H ShahPMID: 32594179 PMCID: PMC7727333

https://tinyurl.com/ML-panel has details on all three axes we talked about (the philosophy, the best practice processes, and the mechanics). The url should be viewable to anyone.

NIH’s AllofUs Research Program is another (explicitly) diverse EHR, survey and genetic database getting off the ground that might be of interesting for some folks

What are some ways to quantify the expected benefit/cost of adding an AI option to an existing workflow?

yes Thanks!!

@Deen — check out https://www.medrxiv.org/content/10.1101/2020.07.10.20149419v1

@Wendi — Developments in public datasets are patchy. For COVID there is N3C by NIH and the Covidresearchdatabase by Datavant.Reg communicating up the org, the use case has to be distilled down to ROI and value statment.Finally, gettting data out from Vendor systems is something ot be handled at contracting. Get data access included in the purchasing.

Dr. Lungren mentioned this worry of model drift, where the relationship between variables that the model is relying upon might change over time. To what extent can these changes be “compensated” for via model retraining? And to make this a bit more concrete, what specific sources of drift might present more difficulties to a scientist trying to build a model that will stand the test of time?

great point distribution

yes a realistic view

@Priya — GOSSIS is a nice (and free!) alternative to things such as the APACHE IV score.

Great step, and in the right direction.

If the issue is shift then why not define things to prediction that we know won’t shift over time?

thanks all!