interpretml/interpret

InterpretML

In the beginning machines learned in darkness, and data scientists struggled in the void to explain them.

Let there be light.

InterpretML is an open-source package that incorporates state-of-the-art machine learning interpretability techniques under one roof. With this package, you can train interpretable glassbox models and explain blackbox systems. InterpretML helps you understand your model's global behavior, or understand the reasons behind individual predictions.

Interpretability is essential for:

• Model debugging - Why did my model make this mistake?
• Feature Engineering - How can I improve my model?
• Detecting fairness issues - Does my model discriminate?
• Human-AI cooperation - How can I understand and trust the model's decisions?
• Regulatory compliance - Does my model satisfy legal requirements?
• High-risk applications - Healthcare, finance, judicial, ...

Installation

Python 3.10+ | Linux, Mac, Windows

pip install interpret
# OR
conda install -c conda-forge interpret

Introducing the Explainable Boosting Machine (EBM)

EBM is an interpretable model developed at Microsoft Research^*. It uses modern machine learning techniques like bagging, gradient boosting, and automatic interaction detection to breathe new life into traditional GAMs (Generalized Additive Models). This makes EBMs as accurate as state-of-the-art techniques like random forests and gradient boosted trees. However, unlike these blackbox models, EBMs produce exact explanations and are editable by domain experts.

Notebook for reproducing table

Supported Techniques

Train a glassbox model

Let's fit an Explainable Boosting Machine

from interpret.glassbox import ExplainableBoostingClassifier

ebm = ExplainableBoostingClassifier()
ebm.fit(X_train, y_train)

# or substitute with LogisticRegression, DecisionTreeClassifier, RuleListClassifier, ...
# EBM supports pandas dataframes, numpy arrays, and handles "string" data natively.

Understand the model

from interpret import show

ebm_global = ebm.explain_global()
show(ebm_global)

Understand individual predictions

ebm_local = ebm.explain_local(X_test, y_test)
show(ebm_local)

And if you have multiple model explanations, compare them

show([logistic_regression_global, decision_tree_global])

If you need to keep your data private, use Differentially Private EBMs (see DP-EBMs)

from interpret.privacy import DPExplainableBoostingClassifier, DPExplainableBoostingRegressor

dp_ebm = DPExplainableBoostingClassifier(epsilon=1, delta=1e-5) # Specify privacy parameters
dp_ebm.fit(X_train, y_train)

show(dp_ebm.explain_global()) # Identical function calls to standard EBMs

For more information, see the documentation.

EBMs include pairwise interactions by default. For 3-way interactions and higher see this notebook: https://interpret.ml/docs/python/examples/custom-interactions.html

Interpret EBMs can be fit on datasets with 100 million samples in several hours. For larger workloads consider using distributed EBMs on Azure SynapseML: classification EBMs and regression EBMs

Acknowledgements

InterpretML was originally created by (equal contributions): Samuel Jenkins, Harsha Nori, Paul Koch, and Rich Caruana

EBMs are fast derivative of GA2M, invented by: Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker

Many people have supported us along the way. Check out ACKNOWLEDGEMENTS.md!

We also build on top of many great packages. Please check them out!

plotly | dash | scikit-learn | lime | shap | salib | skope-rules | treeinterpreter | gevent | joblib | pytest | jupyter

Citations

@article{nori2019interpretml,
  title={InterpretML: A Unified Framework for Machine Learning Interpretability},
  author={Nori, Harsha and Jenkins, Samuel and Koch, Paul and Caruana, Rich},
  journal={arXiv preprint arXiv:1909.09223},
  year={2019}
}
    

@inproceedings{caruana2015intelligible,
  title={Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission},
  author={Caruana, Rich and Lou, Yin and Gehrke, Johannes and Koch, Paul and Sturm, Marc and Elhadad, Noemie},
  booktitle={Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining},
  pages={1721--1730},
  year={2015},
  organization={ACM}
}
    

@inproceedings{lou2013accurate,
  title={Accurate intelligible models with pairwise interactions},
  author={Lou, Yin and Caruana, Rich and Gehrke, Johannes and Hooker, Giles},
  booktitle={Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining},
  pages={623--631},
  year={2013},
  organization={ACM}
}
    

@inproceedings{lou2012intelligible,
  title={Intelligible models for classification and regression},
  author={Lou, Yin and Caruana, Rich and Gehrke, Johannes},
  booktitle={Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining},
  pages={150--158},
  year={2012},
  organization={ACM}
}
    

@article{wang2022interpretability,
  title={Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values},
  author={Wang, Zijie J and Kale, Alex and Nori, Harsha and Stella, Peter and Nunnally, Mark E and Chau, Duen Horng and Vorvoreanu, Mihaela and Vaughan, Jennifer Wortman and Caruana, Rich},
  journal={arXiv preprint arXiv:2206.15465},
  year={2022}
}
    

@inproceedings{zhang2019axiomatic,
  title={Axiomatic Interpretability for Multiclass Additive Models},
  author={Zhang, Xuezhou and Tan, Sarah and Koch, Paul and Lou, Yin and Chajewska, Urszula and Caruana, Rich},
  booktitle={Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining},
  pages={226--234},
  year={2019},
  organization={ACM}
}
    

@inproceedings{tan2018distill,
  title={Distill-and-compare: auditing black-box models using transparent model distillation},
  author={Tan, Sarah and Caruana, Rich and Hooker, Giles and Lou, Yin},
  booktitle={Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society},
  pages={303--310},
  year={2018},
  organization={ACM}
}
    

@article{lengerich2019purifying,
  title={Purifying Interaction Effects with the Functional ANOVA: An Efficient Algorithm for Recovering Identifiable Additive Models},
  author={Lengerich, Benjamin and Tan, Sarah and Chang, Chun-Hao and Hooker, Giles and Caruana, Rich},
  journal={arXiv preprint arXiv:1911.04974},
  year={2019}
}
    

@inproceedings{kaur2020interpreting,
  title={Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning},
  author={Kaur, Harmanpreet and Nori, Harsha and Jenkins, Samuel and Caruana, Rich and Wallach, Hanna and Wortman Vaughan, Jennifer},
  booktitle={Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems},
  pages={1--14},
  year={2020}
}
    

@article{chang2020interpretable,
  title={How Interpretable and Trustworthy are GAMs?},
  author={Chang, Chun-Hao and Tan, Sarah and Lengerich, Ben and Goldenberg, Anna and Caruana, Rich},
  journal={arXiv preprint arXiv:2006.06466},
  year={2020}
}
    

@inproceedings{pmlr-v139-nori21a,
  title = 	 {Accuracy, Interpretability, and Differential Privacy via Explainable Boosting},
  author =       {Nori, Harsha and Caruana, Rich and Bu, Zhiqi and Shen, Judy Hanwen and Kulkarni, Janardhan},
  booktitle = 	 {Proceedings of the 38th International Conference on Machine Learning},
  pages = 	 {8227--8237},
  year = 	 {2021},
  volume = 	 {139},
  series = 	 {Proceedings of Machine Learning Research},
  publisher =    {PMLR}
}
    

@inproceedings{ribeiro2016should,
  title={Why should i trust you?: Explaining the predictions of any classifier},
  author={Ribeiro, Marco Tulio and Singh, Sameer and Guestrin, Carlos},
  booktitle={Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining},
  pages={1135--1144},
  year={2016},
  organization={ACM}
}
    

@incollection{NIPS2017_7062,
 title = {A Unified Approach to Interpreting Model Predictions},
 author = {Lundberg, Scott M and Lee, Su-In},
 booktitle = {Advances in Neural Information Processing Systems 30},
 editor = {I. Guyon and U. V. Luxburg and S. Bengio and H. Wallach and R. Fergus and S. Vishwanathan and R. Garnett},
 pages = {4765--4774},
 year = {2017},
 publisher = {Curran Associates, Inc.},
 url = {https://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf}
}
    

@article{lundberg2018consistent,
  title={Consistent individualized feature attribution for tree ensembles},
  author={Lundberg, Scott M and Erion, Gabriel G and Lee, Su-In},
  journal={arXiv preprint arXiv:1802.03888},
  year={2018}
}
    

@article{lundberg2018explainable,
  title={Explainable machine-learning predictions for the prevention of hypoxaemia during surgery},
  author={Lundberg, Scott M and Nair, Bala and Vavilala, Monica S and Horibe, Mayumi and Eisses, Michael J and Adams, Trevor and Liston, David E and Low, Daniel King-Wai and Newman, Shu-Fang and Kim, Jerry and others},
  journal={Nature Biomedical Engineering},
  volume={2},
  number={10},
  pages={749},
  year={2018},
  publisher={Nature Publishing Group}
}
    

@article{herman2017salib,
  title={SALib: An open-source Python library for Sensitivity Analysis.},
  author={Herman, Jonathan D and Usher, Will},
  journal={J. Open Source Software},
  volume={2},
  number={9},
  pages={97},
  year={2017}
}
    

@article{morris1991factorial,
  title={},
  author={Morris, Max D},
  journal={Technometrics},
  volume={33},
  number={2},
  pages={161--174},
  year={1991},
  publisher={Taylor \& Francis Group}
}
    

@article{friedman2001greedy,
  title={Greedy function approximation: a gradient boosting machine},
  author={Friedman, Jerome H},
  journal={Annals of statistics},
  pages={1189--1232},
  year={2001},
  publisher={JSTOR}
}
    

@article{pedregosa2011scikit,
  title={Scikit-learn: Machine learning in Python},
  author={Pedregosa, Fabian and Varoquaux, Ga{\"e}l and Gramfort, Alexandre and Michel, Vincent and Thirion, Bertrand and Grisel, Olivier and Blondel, Mathieu and Prettenhofer, Peter and Weiss, Ron and Dubourg, Vincent and others},
  journal={Journal of machine learning research},
  volume={12},
  number={Oct},
  pages={2825--2830},
  year={2011}
}
    

@online{plotly, 
  author = {Plotly Technologies Inc.}, 
  title = {Collaborative data science}, 
  publisher = {Plotly Technologies Inc.}, 
  address = {Montreal, QC}, 
  year = {2015}, 
  url = {https://plot.ly}
}
    

@article{varoquaux2009joblib,
  title={Joblib: running python function as pipeline jobs},
  author={Varoquaux, Ga{\"e}l and Grisel, O},
  journal={packages. python. org/joblib},
  year={2009}
}
    

Videos

• The Science Behind InterpretML: Explainable Boosting Machine
• How to Explain Models with InterpretML Deep Dive
• Black-Box and Glass-Box Explanation in Machine Learning
• Explainable AI explained! By-design interpretable models with Microsofts InterpretML
• Interpreting Machine Learning Models with InterpretML
• Machine Learning Model Interpretability using AzureML & InterpretML (Explainable Boosting Machine)
• A Case Study of Using Explainable Boosting Machines
• From SHAP to EBM: Explain your Gradient Boosting Models in Python
• Rich Caruana – Friends Don’t Let Friends Deploy Black-Box Models

External links

• Machine Learning Interpretability in Banking: Why It Matters and How Explainable Boosting Machines Can Help
• Interpretable Machine Learning – Increase Trust and Eliminate Bias
• Enhancing Trust in Credit Risk Models: A Comparative Analysis of EBMs and GBMs
• Explainable AI: unlocking value in FEC operations
• Interpretable or Accurate? Why Not Both?
• The Explainable Boosting Machine. As accurate as gradient boosting, as interpretable as linear regression.
• Exploring explainable boosting machines
• Performance And Explainability With EBM
• InterpretML: Another Way to Explain Your Model
• A gentle introduction to GA2Ms, a white box model
• Model Interpretation with Microsoft’s Interpret ML
• Explaining Model Pipelines With InterpretML
• Explain Your Model with Microsoft’s InterpretML
• On Model Explainability: From LIME, SHAP, to Explainable Boosting
• Dealing with Imbalanced Data (Mortgage loans defaults)
• The Art of Sprezzatura for Machine Learning
• MCTS EDA which makes sense
• Explainable Boosting machines for Tabular data
• Out with the opaque: achieving both model performance and transparency

Papers that use or compare EBMs

• Challenging the Performance-Interpretability Trade-off: An Evaluation of Interpretable Machine Learning Models
• The hidden risk of round numbers and sharp thresholds in clinical practice
• Improving Credit Card Fraud Detection with an Optimized Explainable Boosting Machine
• TabArena: A Living Benchmark for Machine Learning on Tabular Data
• Classifier Calibration at Scale: An Empirical Study of Model-Agnostic Post-Hoc Methods
• Statistical Inference for Explainable Boosting Machines
• Explainable Boosting Machine for Predicting Claim Severity and Frequency in Car Insurance
• Unveiling the drivers of the Baryon Cycles with Interpretable Multi-step Machine Learning and Simulations
• Modelling Container Transhipment Throughput and Analysing Dynamics During Significant External Events: A Case Study of the Port of Busan and the China-to-U.S. Trade Route
• The Most Important Features in Generalized Additive Models Might Be Groups of Features
• Explainable AI with EDA for V2I path loss prediction
• A Comparative Analysis of Interpretable Machine Learning Methods
• ParamBoost: Gradient Boosted Piecewise Cubic Polynomials
• GAMFORMER: In-context Learning for Generalized Additive Models
• Automatic Piecewise Linear Regression
• Accuracy is not enough: explainable boosting machine model and identification of candidate biomarkers for real-time sepsis risk assessment in the emergency department
• Explainable Boosting Machine approach identifies risk factors for acute renal failure
• Domain-informed explainable boosting machines for trustworthy lateral spread predictions
• An explainable boosting machine model for identifying artifacts caused by formalin-fixed paraffin embedding
• Glass Box Machine Learning and Corporate Bond Returns
• Data Science with LLMs and Interpretable Models
• DimVis: Interpreting Visual Clusters in Dimensionality Reduction With Explainable Boosting Machine
• Distill knowledge of additive tree models into generalized linear models
• Explainable Boosting Machines with Sparsity - Maintaining Explainability in High-Dimensional Settings
• Cost of Explainability in AI: An Example with Credit Scoring Models
• iTARGET: Interpretable Tailored Age Regression for Grouped Epigenetic Traits
• Vehicle Fuel Optimization Under Real-World Driving Conditions: An Explainable Artificial Intelligence Approach
• Explainable Boosting Machine for Structural Health Assessment: An Interpretable Approach to Data-Driven Structural Assessment
• Interpretable Machine Learning Leverages Proteomics to Improve Cardiovascular Disease Risk Prediction and Biomarker Identification
• Interpretable Additive Tabular Transformer Networks
• Interpretable Survival Analysis for Heart Failure Risk Prediction
• Investigating Protective and Risk Factors and Predictive Insights for Aboriginal Perinatal Mental Health: Explainable Artificial Intelligence Approach
• Explainable Learning Framework for the Assessment and Prediction of Wind Shear-Induced Aviation Turbulence
• HearteXplain: explainable prediction of acute heart failure and identification of hematologic biomarkers using EBMs and Morris sensitivity analysis
• Identification of a Novel Lipidomic Biomarker for Hepatocyte Carcinoma Diagnosis: Advanced Boosting Machine Learning Techniques Integrated with Explainable Artificial Intelligence
• Interpretable machine learning for precision cognitive aging
• LLMs Understand Glass-Box Models, Discover Surprises, and Suggest Repairs
• Model Interpretability in Credit Insurance
• Enhancing ML Interpretability for Credit Scoring
• Transparent and Fair Profiling in Employment Services: Evidence from Switzerland
• Federated Boosted Decision Trees with Differential Privacy
• Differentially private and explainable boosting machine with enhanced utility
• Balancing Explainability and Privacy in Bank Failure Prediction: A Differentially Private Glass-Box Approach
• GAM(E) changer or not? An evaluation of interpretable machine learning models based on additive model constraints
• GAM Coach: Towards Interactive and User-centered Algorithmic Recourse
• Missing Values and Imputation in Healthcare Data: Can Interpretable Machine Learning Help?
• Revealing the Galaxy-Halo Connection Through Machine Learning
• How the Galaxy–Halo Connection Depends on Large-Scale Environment
• A diagnostic support system based on interpretable machine learning and oscillometry for accurate diagnosis of respiratory dysfunction in silicosis
• Using Explainable Boosting Machines (EBMs) to Detect Common Flaws in Data
• Differentially Private Gradient Boosting on Linear Learners for Tabular Data Analysis
• Concrete compressive strength prediction using an explainable boosting machine model
• When Interpretability Meets Generalization: Delta-GAM for Robust Extrapolation in Out-of-Distribution Settings
• Mitigating Cognitive Biases in Predicting Student Dropout: Global and Local Explainability with Explainable Boosting Machine
• Proxy endpoints - bridging clinical trials and real world data
• Machine Learning Model Reveals Determinators for Admission to Acute Mental Health Wards From Emergency Department Presentations
• Interpretable Machine Learning Models for Predicting Perioperative Myocardial Injury in Non-Cardiac Surgery
• Predicting Robotic Hysterectomy Incision Time: Optimizing Surgical Scheduling with Machine Learning
• Performance of explainable artificial intelligence in guiding the management of patients with a pancreatic cyst
• Explainable boosting machine for structural health assessment of reinforced concrete beams using crack width measurements
• Predicting Blood Pressure Variability in Hemodialysis Using an Explainable Boosting Machine Model
• Validating Explainer Methods: A Functionally Grounded Approach for Numerical Forecasting
• Using explainable machine learning and fitbit data to investigate predictors of adolescent obesity
• Estimate Deformation Capacity of Non-Ductile RC Shear Walls Using Explainable Boosting Machine
• Targeting resources efficiently and justifiably by combining causal machine learning and theory
• Extractive Text Summarization Using Generalized Additive Models with Interactions for Sentence Selection
• Post-Hoc Interpretation of Transformer Hyperparameters with Explainable Boosting Machines
• Interpretable machine learning for predicting pathologic complete response in patients treated with chemoradiation therapy for rectal adenocarcinoma
• Exploring the Balance between Interpretability and Performance with carefully designed Constrainable Neural Additive Models
• Interpretable machine learning algorithms to predict leaf senescence date of deciduous trees
• Cross Feature Selection to Eliminate Spurious Interactions and Single Feature Dominance Explainable Boosting Machines
• Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Models
• Assessing wind field characteristics along the airport runway glide slope: an explainable boosting machine-assisted wind tunnel study
• Explainable Modeling for Wind Power Forecasting: A Glass-Box Approach with High Accuracy
• Trustworthy Academic Risk Prediction with Explainable Boosting Machines
• Binary ECG Classification Using Explainable Boosting Machines for IoT Edge Devices
• Explainable artificial intelligence toward usable and trustworthy computer-aided diagnosis of multiple sclerosis from Optical Coherence Tomography
• An Interpretable Machine Learning Model with Deep Learning-based Imaging Biomarkers for Diagnosis of Alzheimer’s Disease
• Explainable Boosting Machine for Predicting Alzheimer’s Disease from MRI Hippocampal Subfields
• Explainable Artificial Intelligence for Cotton Yield Prediction With Multisource Data
• Preoperative detection of extraprostatic tumor extension in patients with primary prostate cancer utilizing [68Ga]Ga-PSMA-11 PET/MRI
• Monotone Tree-Based GAMI Models by Adapting XGBoost
• Extending Explainable Boosting Machines to Scientific Image Data
• Exploring explanation deficits in subclinical mastitis detection with explainable boosting machines
• Pest Presence Prediction Using Interpretable Machine Learning
• Key Thresholds and Relative Contributions of Knee Geometry, Anteroposterior Laxity, and Body Weight as Risk Factors for Noncontact ACL Injury
• Explainable Boosting Machines for Slope Failure Spatial Predictive Modeling
• Leveraging interpretable machine learning in intensive care
• Using Interpretable Machine Learning to Predict Maternal and Fetal Outcomes
• Neural Additive Models: Interpretable Machine Learning with Neural Nets
• Improving Neural Additive Models with Bayesian Principles
• NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning
• Scalable Interpretability via Polynomials
• Neural Basis Models for Interpretability
• ILMART: Interpretable Ranking with Constrained LambdaMART
• Integrating Co-Clustering and Interpretable Machine Learning for the Prediction of Intravenous Immunoglobulin Resistance in Kawasaki Disease
• Explainable Gradient Boosting for Corporate Crisis Forecasting in Italian Businesses
• Investigating Trust in Human-Machine Learning Collaboration: A Pilot Study on Estimating Public Anxiety from Speech
• pureGAM: Learning an Inherently Pure Additive Model
• GAMI-Net: An Explainable Neural Network based on Generalized Additive Models with Structured Interactions
• Interpretable Machine Learning based on Functional ANOVA Framework: Algorithms and Comparisons
• Using Model-Based Trees with Boosting to Fit Low-Order Functional ANOVA Models
• Interpretable generalized additive neural networks
• Explainable machine learning with pairwise interactions for the classification of Parkinson’s disease and SWEDD from clinical and imaging features
• In Pursuit of Interpretable, Fair and Accurate Machine Learning for Criminal Recidivism Prediction
• From Shapley Values to Generalized Additive Models and back
• Developing A Visual-Interactive Interface for Electronic Health Record Labeling: An Explainable Machine Learning Approach
• Development and Validation of an Interpretable 3-day Intensive Care Unit Readmission Prediction Model Using Explainable Boosting Machines
• EPS: An Explainable Post-Shot Expected Goal Metric for Evaluating Goalkeepers and Attackers
• Prediction of surface rougness of additively manufactured and machined parts via machine learning
• Development of Explainable Machine Learning Models to Predict Outcomes After Platelet-Rich Plasma Injections for Knee Osteoarthritis
• Knowledge-Guided Machine Learning: Illustrating the use of Explainable Boosting Machines to Identify Overshooting Tops in Satellite Imagery
• Building a predictive model to identify clinical indicators for COVID-19 using machine learning method
• Using Innovative Machine Learning Methods to Screen and Identify Predictors of Congenital Heart Diseases
• Explainable Boosting Machine: A Contemporary Glass-Box Model to Analyze Work Zone-Related Road Traffic Crashes
• Efficient and Interpretable Traffic Destination Prediction using Explainable Boosting Machines
• Explainable Artificial Intelligence Paves the Way in Precision Diagnostics and Biomarker Discovery for the Subclass of Diabetic Retinopathy in Type 2 Diabetics
• A proposed tree-based explainable artificial intelligence approach for the prediction of angina pectoris
• Explainable Boosting Machine: A Contemporary Glass-Box Strategy for the Assessment of Wind Shear Severity in the Runway Vicinity Based on the Doppler Light Detection and Ranging Data
• On the Physical Nature of Lyα Transmission Spikes in High Redshift Quasar Spectra
• GRAND-SLAMIN’ Interpretable Additive Modeling with Structural Constraints
• Identification of groundwater potential zones in data-scarce mountainous region using explainable machine learning
• Explainable Classification Techniques for Quantum Dot Device Measurements

Books that cover EBMs

• Machine Learning for High-Risk Applications
• Explainable AI with Python
• Interpretable Machine Learning with Python
• Explainable Artificial Intelligence: An Introduction to Interpretable Machine Learning
• Applied Machine Learning Explainability Techniques
• The eXplainable A.I.: With Python examples
• Platform and Model Design for Responsible AI: Design and build resilient, private, fair, and transparent machine learning models
• Explainable AI Recipes
• Ensemble Methods for Machine Learning
• Interpretability and Explainability in AI Using Python

External tools

• R package for building EBMs through reticulate
• EBM to Onnx converter by SoftAtHome
• EBM to SQL converter - ML 2 SQL
• EBM to PMML converter - SkLearn2PMML
• EBM visual editor - GAM Changer
• Interpreting Visual Clusters in Dimensionality Reduction - DimVis

Contact us

There are multiple ways to get in touch:

• Email us at [email protected]
• Or, feel free to raise a GitHub issue

If a tree fell in your random forest, would anyone notice?