# Model Explainability Amazon SageMaker Clarify provides tools to help explain how machine learning (ML) models make predictions. These tools can help ML modelers and developers and other internal stakeholders understand model characteristics as a whole prior to deployment and to debug predictions provided by the model after it's deployed. + To obtain explanations for your datasets and models, see [Fairness, model explainability and bias detection with SageMaker Clarify](clarify-configure-processing-jobs.md). + To obtain explanations in real-time from a SageMaker AI endpoint, see [Online explainability with SageMaker Clarify](clarify-online-explainability.md). Transparency about how ML models arrive at their predictions is also critical to consumers and regulators. They need to trust the model predictions if they are going to accept the decisions based on them. SageMaker Clarify uses a model-agnostic feature attribution approach. You can use this to understand why a model made a prediction after training, and to provide per-instance explanation during inference. The implementation includes a scalable and efficient implementation of [SHAP](https://papers.nips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf). This is based on the concept of a Shapley value, from the field of cooperative game theory, that assigns each feature an importance value for a particular prediction. Clarify produces partial dependence plots (PDPs) that show the marginal effect features have on the predicted outcome of a machine learning model. Partial dependence helps explain target response given a set of input features. It also supports both computer vision (CV) and natural language processing (NLP) explainability using the same Shapley values (SHAP) algorithm as used for tabular data explanations. What is the function of an explanation in the machine learning context? An explanation can be thought of as the answer to a *Why question* that helps humans understand the cause of a prediction. In the context of an ML model, you might be interested in answering questions such as: + Why did the model predict a negative outcome such as a loan rejection for a given applicant? + How does the model make predictions? + Why did the model make an incorrect prediction? + Which features have the largest influence on the behavior of the model? You can use explanations for auditing and meeting regulatory requirements, building trust in the model and supporting human decision-making, and debugging and improving model performance. The need to satisfy the demands for human understanding about the nature and outcomes of ML inference is key to the sort of explanation needed. Research from philosophy and cognitive science disciplines has shown that people care especially about contrastive explanations, or explanations of why an event X happened instead of some other event Y that did not occur. Here, X could be an unexpected or surprising event that happened and Y corresponds to an expectation based on their existing mental model referred to as a *baseline*. Note that for the same event X, different people might seek different explanations depending on their point of view or mental model Y. In the context of explainable AI, you can think of X as the example being explained and Y as a baseline that is typically chosen to represent an uninformative or average example in the dataset. Sometimes, for example in the case of ML modeling of images, the baseline might be implicit, where an image whose pixels are all the same color can serves as a baseline. ## Sample Notebooks Amazon SageMaker Clarify provides the following sample notebook for model explainability: + [Amazon SageMaker Clarify Processing](https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-clarify/index.html#sagemaker-clarify-processing) – Use SageMaker Clarify to create a processing job for the detecting bias and explaining model predictions with feature attributions. Examples include using CSV and JSON Lines data formats, bringing your own container, and running processing jobs with Spark. + [Explaining Image Classification with SageMaker Clarify](https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-clarify/computer_vision/image_classification/explainability_image_classification.ipynb) – SageMaker Clarify provides you with insights into how your computer vision models classify images. + [Explaining object detection models with SageMaker Clarify ](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-clarify/computer_vision/object_detection/object_detection_clarify.ipynb) – SageMaker Clarify provides you with insights into how your computer vision models detect objects. This notebook has been verified to run in Amazon SageMaker Studio only. If you need instructions on how to open a notebook in Amazon SageMaker Studio, see [Create or Open an Amazon SageMaker Studio Classic Notebook](notebooks-create-open.md). If you're prompted to choose a kernel, choose **Python 3 (Data Science)**. **Topics** + [Sample Notebooks](#clarify-model-explainability-sample-notebooks) + [Feature Attributions that Use Shapley Values](clarify-shapley-values.md) + [Asymmetric Shapley Values](clarify-feature-attribute-shap-asymm.md) + [SHAP Baselines for Explainability](clarify-feature-attribute-shap-baselines.md) # Feature Attributions that Use Shapley Values SageMaker Clarify provides feature attributions based on the concept of [Shapley value](https://en.wikipedia.org/wiki/Shapley_value). You can use Shapley values to determine the contribution that each feature made to model predictions. These attributions can be provided for specific predictions and at a global level for the model as a whole. For example, if you used an ML model for college admissions, the explanations could help determine whether the GPA or the SAT score was the feature most responsible for the model’s predictions, and then you can determine how responsible each feature was for determining an admission decision about a particular student. SageMaker Clarify has taken the concept of Shapley values from game theory and deployed it in a machine learning context. The Shapley value provides a way to quantify the contribution of each player to a game, and hence the means to distribute the total gain generated by a game to its players based on their contributions. In this machine learning context, SageMaker Clarify treats the prediction of the model on a given instance as the *game* and the features included in the model as the *players*. For a first approximation, you might be tempted to determine the marginal contribution or effect of each feature by quantifying the result of either *dropping* that feature from the model or *dropping* all other features from the model. However, this approach does not take into account that features included in a model are often not independent from each other. For example, if two features are highly correlated, dropping either one of the features might not alter the model prediction significantly. To address these potential dependencies, the Shapley value requires that the outcome of each possible combination (or coalition) of features must be considered to determine the importance of each feature. Given *d* features, there are 2d such possible feature combinations, each corresponding to a potential model. To determine the attribution for a given feature *f*, consider the marginal contribution of including *f* in all feature combinations (and associated models) that do not contain *f*, and take the average. It can be shown that Shapley value is the unique way of assigning the contribution or importance of each feature that satisfies certain desirable properties. In particular, the sum of Shapley values of each feature corresponds to the difference between the predictions of the model and a dummy model with no features. However, even for reasonable values of *d*, say 50 features, it is computationally prohibitive and impractical to train 2d possible models. As a result, SageMaker Clarify needs to make use of various approximation techniques. For this purpose, SageMaker Clarify uses Shapley Additive exPlanations (SHAP), which incorporates such approximations and devised a scalable and efficient implementation of the Kernel SHAP algorithm through additional optimizations. For additional information on Shapley values, see [A Unified Approach to Interpreting Model Predictions](https://papers.nips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdf). # Asymmetric Shapley Values The SageMaker Clarify time series forecasting model explanation solution is a feature attribution method rooted in [cooperative game theory](https://en.wikipedia.org/wiki/Cooperative_game_theory), similar in spirit to SHAP. Specifically, Clarify uses [random order group values](http://www.library.fa.ru/files/Roth2.pdf#page=121), also known as [asymmetric Shapley values](https://proceedings.neurips.cc/paper/2020/file/0d770c496aa3da6d2c3f2bd19e7b9d6b-Paper.pdf) in machine learning and explainability. ## Background The goal is to compute attributions for input features to a given forecasting model *f*. The forecasting model takes the following inputs: + Past time series *(target TS)*. For example, this could be past daily train passengers in the Paris-Berlin route, denoted by *xt​*. + (Optional) A covariate time series. For example, this could be festivities and weather data, denoted by *zt* ​∈ RS. When used, covariate TS could be available only for the past time steps or also for the future ones (included in the festivity calendar). + (Optional) Static covariates, such as quality of service (like 1st or 2nd class), denoted by *u* ∈ RE. Static covariates, dynamic covariates, or both can be omitted, depending on the specific application scenario. Given a prediction horizon K ≥ 0 (e.g. K=30 days) the model prediction can be characterized by the formula: *f(x[1:T], z[1:T\$1K], u) = x[T\$11:T \$1K\$11]*. The following diagram shows a dependency structure for a typical forecasting model. The prediction at time *t\$11* depends on the three types of inputs previously mentioned. ![\[Dependency structure for a typical forecasting model.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/clarify/clarify-forecast-dependency.png) ## Method Explanations are computed by querying the time series model *f* on a series of points derived by the original input. Following game theoretic constructions, Clarify averages differences in predictions led by obfuscating (that is, setting to a baseline value) parts of the inputs iteratively. The temporal structure can be navigated in a chronological or anti-chronological order or both. Chronological explanations are built by iteratively adding information from the first time step, while anti-chronological from the last step. The latter mode may be more appropriate in the presence of recency bias, such as when forecasting stock prices. One important property of the computed explanations is that they sum to the original model output if the model provides deterministic outputs. ## Resulting attributions Resulting attributions are scores that mark individual contributions of specific time steps or input features toward the final forecast at each forecasted time step. Clarify offers the following two granularities for explanations: + Timewise explanations are inexpensive and provide information about specific time steps only, such as how much the information of the 19th day in the past contributed to the forecasting of the 1st day in the future. These attributions do not explain individually static covariates and aggregate explanations of target and covariate time series. The attributions are a matrix *A* where each *Atk​* is the attribution of time step *t* toward forecasting of time step *T\$1k*. Note that if the model accepts future covariates, *t* can be greater than *T*. + Fine-grained explanations are more computationally intensive and provide a full breakdown of all attributions of the input variables. **Note** Fine-grained explanations only support chronological order. The resulting attributions are a triplet composed of the following: + Matrix *Ax* ∈ RT×K related to the input time series, where *Atkx​* is the attribution of *xt* toward forecasting step *T\$1k* + Tensor *Az* ∈ *RT\$1K×S×K* related to the covariate time series, where *Atskz​* is the attribution of *zts​* (i.e. the sth covariate TS) toward forecasting step *T\$1k* + Matrix *Au* ∈ RE×K related to the static covariates, where *Aeku​* is the attribution of *ue* ​(the eth static covariate) toward forecasting step *T\$1k* Regardless of the granularity, the explanation also contains an offset vector *B* ∈ *RK* that represents the “basic behavior” of the model when all data is obfuscated. # SHAP Baselines for Explainability Explanations are typically contrastive (that is, they account for deviations from a baseline). As a result, for the same model prediction, you can expect to get different explanations with respect to different baselines. Therefore, your choice of a baseline is crucial. In an ML context, the baseline corresponds to a hypothetical instance that can be either *uninformative* or *informative*. During the computation of Shapley values, SageMaker Clarify generates several new instances between the baseline and the given instance, in which the absence of a feature, is modeled by setting the feature value to that of the baseline and the presence of a feature is modeled by setting the feature value to that of the given instance. Thus, the absence of all features corresponds to the baseline and the presence of all features corresponds to the given instance. How can you choose good baselines? Often it is desirable to select a baseline with very low information content. For example, you can construct an average instance from the training dataset by taking either the median or average for numerical features and the mode for categorical features. For the college admissions example, you might be interested in explaining why a particular applicant was accepted as compared to a baseline acceptances based on an average applicant. If not provided, a baseline is calculated automatically by SageMaker Clarify using K-means or K-prototypes in the input dataset. Alternatively, you can choose to generate explanations with respect to informative baselines. For the college admissions scenario, you might want to explain why a particular applicant was rejected when compared with other applicants from similar demographic backgrounds. In this case, you can choose a baseline that represents the applicants of interest, namely those from a similar demographic background. Thus, you can use informative baselines to concentrate the analysis on the specific aspects of a particular model prediction. You can isolate the features for assessment by setting demographic attributes and other features that you can't act on to the same value as in the given instance.