Two popular open source Model Explainability frameworks are ELI5 and SHAP. Here is a bit more information about each one.