SageMaker Debugger Interactive Report for XGBoost
Receive training reports autogenerated by Debugger. The Debugger reports provide insights into your training jobs and suggest recommendations to improve your model performance.
Note
You can download a Debugger reports while your training job is running or after the job has finished. During training, Debugger concurrently updates the report reflecting the current rules' evaluation status. You can download a complete Debugger report only after the training job has completed.
Important
In the report, plots and and recommendations are provided for informational purposes and are not definitive. You are responsible for making your own independent assessment of the information.
SageMaker Debugger XGBoost Training Report
For SageMaker XGBoost training jobs, use the Debugger CreateXgboostReport rule to receive a comprehensive training
report of the training progress and results. Following this guide, specify the CreateXgboostReport rule while
constructing an XGBoost estimator, download the report using the Amazon SageMaker Python SDK
Important
In the report, plots and and recommendations are provided for informational purposes and are not definitive. You are responsible for making your own independent assessment of the information.
Topics
Construct a SageMaker XGBoost Estimator with the Debugger XGBoost Report Rule
The CreateXgboostReport rule collects the following output tensors from your training job:
-
hyperparameters
– Saves at the first step. -
metrics
– Saves loss and accuracy every 5 steps. -
feature_importance
– Saves every 5 steps. -
predictions
– Saves every 5 steps. -
labels
– Saves every 5 steps.
The output tensors are saved at a default S3 bucket. For example,
s3://sagemaker-
.<region>
-<12digit_account_id>
/<base-job-name>
/debug-output/
When you construct a SageMaker estimator for an XGBoost training job, specify the rule as shown in the following example code.
Download the Debugger XGBoost Training Report
Download the Debugger XGBoost training report while your training job is running or
after the job has finished using the Amazon SageMaker Python SDK
Debugger XGBoost Training Report Walkthrough
This section walks you through the Debugger XGBoost training report. The report is automatically aggregated depending on the output tensor regex, recognizing what type of your training job is among binary classification, multiclass classification, and regression.
Important
In the report, plots and and recommendations are provided for informational purposes and are not definitive. You are responsible for making your own independent assessment of the information.
Topics
- Distribution of True Labels of the Dataset
- Loss versus Step Graph
- Feature Importance
- Confusion Matrix
- Evaluation of the Confusion Matrix
- Accuracy Rate of Each Diagonal Element Over Iteration
- Receiver Operating Characteristic Curve
- Distribution of Residuals at the Last Saved Step
- Absolute Validation Error per Label Bin Over Iteration
Distribution of True Labels of the Dataset
This histogram shows the distribution of labeled classes (for classification) or values (for regression) in your original dataset. Skewness in your dataset could contribute to inaccuracies. This visualization is available for the following model types: binary classification, multiclassification, and regression.
![An example of a distribution of true labels of the dataset graph.](/images/sagemaker/latest/dg/images/debugger/debugger-training-xgboost-report-walkthrough-dist-label.png)
Loss versus Step Graph
This is a line chart that shows the progression of loss on training data and validation data throughout training steps. The loss is what you defined in your objective function, such as mean squared error. You can gauge whether the model is overfit or underfit from this plot. This section also provides insights that you can use to determine how to resolve the overfit and underfit problems. This visualization is available for the following model types: binary classification, multiclassification, and regression.
![An example of a loss versus step graph.](/images/sagemaker/latest/dg/images/debugger/debugger-training-xgboost-report-walkthrough-loss-vs-step.png)
Feature Importance
There are three different types of feature importance visualizations provided: Weight, Gain and Coverage. We provide detailed definitions for each of the three in the report. Feature importance visualizations help you learn what features in your training dataset contributed to the predictions. Feature importance visualizations are available for the following model types: binary classification, multiclassification, and regression.
![An example of a feature importance graph.](/images/sagemaker/latest/dg/images/debugger/debugger-training-xgboost-report-walkthrough-feature-importance.png)
Confusion Matrix
This visualization is only applicable to binary and multiclass classification models. Accuracy alone might not be sufficient for evaluating the model performance. For some use cases, such as healthcare and fraud detection, it’s also important to know the false positive rate and false negative rate. A confusion matrix gives you the additional dimensions for evaluating your model performance.
![An example of confusion matrix.](/images/sagemaker/latest/dg/images/debugger/debugger-training-xgboost-report-walkthrough-confusion-matrix.png)
Evaluation of the Confusion Matrix
This section provides you with more insights on the micro, macro, and weighted metrics on precision, recall, and F1-score for your model.
![Evaluation of the confusion matrix.](/images/sagemaker/latest/dg/images/debugger/debugger-training-xgboost-report-walkthrough-eval-conf-matrix.png)
Accuracy Rate of Each Diagonal Element Over Iteration
This visualization is only applicable to binary classification and multiclass classification models. This is a line chart that plots the diagonal values in the confusion matrix throughout the training steps for each class. This plot shows you how the accuracy of each class progresses throughout the training steps. You can identify the under-performing classes from this plot.
![An example of an accuracy rate of each diagonal element over iteration graph.](/images/sagemaker/latest/dg/images/debugger/debugger-training-xgboost-report-walkthrough-accuracy-rate.gif)
Receiver Operating Characteristic Curve
This visualization is only applicable to binary classification models. The Receiver Operating Characteristic curve is commonly used to evaluate binary classification model performance. The y-axis of the curve is True Positive Rate (TPF) and x-axis is false positive rate (FPR). The plot also displays the value for the area under the curve (AUC). The higher the AUC value, the more predictive your classifier. You can also use the ROC curve to understand the trade-off between TPR and FPR and identify the optimum classification threshold for your use case. The classification threshold can be adjusted to tune the behavior of the model to reduce more of one or another type of error (FP/FN).
![An example a receiver operating characteristic curve graph.](/images/sagemaker/latest/dg/images/debugger/debugger-training-xgboost-report-walkthrough-rec-op-char.png)
Distribution of Residuals at the Last Saved Step
This visualization is a column chart that shows the residual distributions in the last step Debugger captures. In this visualization, you can check whether the residual distribution is close to normal distribution that’s centered at zero. If the residuals are skewed, your features may not be sufficient for predicting the labels.
![An example of a distribution of residuals at the last saved step graph.](/images/sagemaker/latest/dg/images/debugger/debugger-training-xgboost-report-walkthrough-dist-residual.png)
Absolute Validation Error per Label Bin Over Iteration
This visualization is only applicable to regression models. The actual target values are split into 10 intervals. This visualization shows how validation errors progress for each interval throughout the training steps in line plots. Absolute validation error is the absolute value of difference between prediction and actual during validation. You can identify the underperforming intervals from this visualization.
![An example an absolute validation error per label bin over iteration graph.](/images/sagemaker/latest/dg/images/debugger/debugger-training-xgboost-report-walkthrough-val-error-per-label-bin.png)