MLOE-10: Profile data to improve quality

Use the built-in data preparation capability of Amazon SageMaker AI Studio Notebook - This allows you to visually review data characteristics and remediate data-quality problems directly in your notebook environment. When you display a data frame (that is, a tabular representation of data) in your notebook, Amazon SageMaker AI Studio Notebook automatically generates charts to help users identify data-quality issues and suggests data transformations to help fix common problems. After you select a data transformation, Amazon SageMaker AI Studio Notebook generates the corresponding code within the notebook so that it can be repeatedly applied every time the notebook is run.

Use Amazon SageMaker AI Data Wrangler - Import, prepare, transform, visualize, and analyze data with SageMaker AI Data Wrangler. You can integrate Data Wrangler into your ML workflows to simplify and streamline data pre-processing and feature engineering with little to no coding. You can also add your own Python scripts and transformations to customize your data preparation workflow. Import data from Amazon S3, Amazon Redshift, or other data sources, and then query the data using Amazon Athena. Use Data Wrangler to create sophisticated machine learning data preparation workflows with built-in and custom data transformations and analysis features. These features include feature target leakage and quick modeling.

Create an automatic data profile and a reporting system - Use AWS Glue Crawler to crawl the data sources and create a data schema. Use AWSAWS Glue Data Catalog to list all the tables and schemas. Use Amazon Athena for serverless SQL querying to constantly profile data and then use Amazon QuickSight dashboards for visualization of the data.

Create a baseline dataset with SageMaker AI Model Monitor – The training dataset used to train the model is usually a good baseline dataset. The training dataset data schema and the inference dataset schema should exactly match (the number and order of the features).

Amazon SageMaker AI Notebooks

Data Wrangler – Getting Started

SageMaker AI Model Monitor – Create baseline

Next Generation SageMaker Notebooks – Now with Built-in Data Preparation, Real-Time Collaboration, and Notebook Automation

Introducing Amazon SageMaker AI Data Wrangler, a Visual Interface to Prepare Data for Machine Learning

Exploratory data analysis, feature engineering, and operationalizing your data flow into your ML with Amazon SageMaker AI Data Wrangler

Prepare data for predicting credit risk using Amazon SageMaker AI Data Wrangler and Amazon SageMaker AI Clarify

Prepare data from Snowflake for machine learning with Amazon SageMaker AI Data Wrangler

Develop and deploy ML models using Amazon SageMaker AI Data Wrangler and Amazon SageMaker AI Autopilot

Build an automatic data profiling and reporting solution with Amazon EMR, AWS Glue, and Amazon QuickSight

Prepare image data with Amazon SageMaker AI Data Wrangler