Data model mappings for scheduled queries
Timestream for LiveAnalytics supports flexible modeling of data in its tables and this same flexibility applies to results of scheduled queries that are materialized into another Timestream for LiveAnalytics table. With scheduled queries, you can query any table, whether it has data in multi-measure records or single-measure records and write the query results using either multi-measure or single-measure records.
You use the TargetConfiguration in the specification of a scheduled query to map the query results to the appropriate columns in the destination derived table. The following sections describe the different ways of specifying this TargetConfiguration to achieve different data models in the derived table. Specifically, you will see:
-
How to write to multi-measure records when the query result does not have a measure name and you specify the target measure name in the TargetConfiguration.
-
How you use measure name in the query result to write multi-measure records.
-
How you can define a model to write multiple records with different multi-measure attributes.
-
How you can define a model to write to single-measure records in the derived table.
-
How you can query single-measure records and/or multi-measure records in a scheduled query and have the results materialized to either a single-measure record or a multi-measure record, which allows you to choose the flexibility of data models.
Example: Target measure name for multi-measure records
In this example, you will see that the query is reading data from a table with multi-measure data and is writing the results into another table using multi-measure records. The scheduled query result does not have a natural measure name column. Here, you specify the measure name in the derived table using the TargetMultiMeasureName property in the TargetConfiguration.TimestreamConfiguration.
{ "Name" : "CustomMultiMeasureName", "QueryString" : "SELECT region, bin(time, 1h) as hour, AVG(memory_cached) as avg_mem_cached_1h, MIN(memory_free) as min_mem_free_1h, MAX(memory_used) as max_mem_used_1h, SUM(disk_io_writes) as sum_1h, AVG(disk_used) as avg_disk_used_1h, AVG(disk_free) as avg_disk_free_1h, MAX(cpu_user) as max_cpu_user_1h, MIN(cpu_idle) as min_cpu_idle_1h, MAX(cpu_system) as max_cpu_system_1h FROM raw_data.devops_multi WHERE time BETWEEN bin(@scheduled_runtime, 1h) - 14h AND bin(@scheduled_runtime, 1h) - 2h AND measure_name = 'metrics' GROUP BY region, bin(time, 1h)", "ScheduleConfiguration" : { "ScheduleExpression" : "cron(0 0/1 * * ? *)" }, "NotificationConfiguration" : { "SnsConfiguration" : { "TopicArn" : "******" } }, "ScheduledQueryExecutionRoleArn": "******", "TargetConfiguration": { "TimestreamConfiguration": { "DatabaseName" : "derived", "TableName" : "dashboard_metrics_1h_agg_1", "TimeColumn" : "hour", "DimensionMappings" : [ { "Name": "region", "DimensionValueType" : "VARCHAR" } ], "MultiMeasureMappings" : { "TargetMultiMeasureName": "dashboard-metrics", "MultiMeasureAttributeMappings" : [ { "SourceColumn" : "avg_mem_cached_1h", "MeasureValueType" : "DOUBLE", "TargetMultiMeasureAttributeName" : "avgMemCached" }, { "SourceColumn" : "min_mem_free_1h", "MeasureValueType" : "DOUBLE" }, { "SourceColumn" : "max_mem_used_1h", "MeasureValueType" : "DOUBLE" }, { "SourceColumn" : "sum_1h", "MeasureValueType" : "DOUBLE", "TargetMultiMeasureAttributeName" : "totalDiskWrites" }, { "SourceColumn" : "avg_disk_used_1h", "MeasureValueType" : "DOUBLE" }, { "SourceColumn" : "avg_disk_free_1h", "MeasureValueType" : "DOUBLE" }, { "SourceColumn" : "max_cpu_user_1h", "MeasureValueType" : "DOUBLE", "TargetMultiMeasureAttributeName" : "CpuUserP100" }, { "SourceColumn" : "min_cpu_idle_1h", "MeasureValueType" : "DOUBLE" }, { "SourceColumn" : "max_cpu_system_1h", "MeasureValueType" : "DOUBLE", "TargetMultiMeasureAttributeName" : "CpuSystemP100" } ] } } }, "ErrorReportConfiguration": { "S3Configuration" : { "BucketName" : "******", "ObjectKeyPrefix": "errors", "EncryptionOption": "SSE_S3" } } }
The mapping in this example creates one multi-measure record with measure name dashboard-metrics and attribute names avgMemCached, min_mem_free_1h, max_mem_used_1h, totalDiskWrites, avg_disk_used_1h, avg_disk_free_1h, CpuUserP100, min_cpu_idle_1h, CpuSystemP100. Notice the optional use of TargetMultiMeasureAttributeName to rename the query output columns to a different attribute name used for result materialization.
The following is the schema for the destination table once this scheduled query is
materialized. As you can see from the Timestream for LiveAnalytics attribute type in the following result,
the results are materialized into a multi-measure record with a single-measure name
dashboard-metrics
, as shown in the measure schema.
Column | Type | Timestream for LiveAnalytics attribute type |
---|---|---|
region |
varchar |
DIMENSION |
measure_name |
varchar |
MEASURE_NAME |
time |
timestamp |
TIMESTAMP |
CpuSystemP100 |
double |
MULTI |
avgMemCached |
double |
MULTI |
min_cpu_idle_1h |
double |
MULTI |
avg_disk_free_1h |
double |
MULTI |
avg_disk_used_1h |
double |
MULTI |
totalDiskWrites |
double |
MULTI |
max_mem_used_1h |
double |
MULTI |
min_mem_free_1h |
double |
MULTI |
CpuUserP100 |
double |
MULTI |
The following are the corresponding measures obtained with a SHOW MEASURES query.
measure_name | data_type | Dimensions |
---|---|---|
dashboard-metrics |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
Example: Using measure name from scheduled query in multi-measure records
In this example, you will see a query reading from a table with single-measure records and materializing the results into multi-measure records. In this case, the scheduled query result has a column whose values can be used as measure names in the target table where the results of the scheduled query is materialized. Then you can specify the measure name for the multi-measure record in the derived table using the MeasureNameColumn property in TargetConfiguration.TimestreamConfiguration.
{ "Name" : "UsingMeasureNameFromQueryResult", "QueryString" : "SELECT region, bin(time, 1h) as hour, measure_name, AVG(CASE WHEN measure_name IN ('memory_cached', 'disk_used', 'disk_free') THEN measure_value::double ELSE NULL END) as avg_1h, MIN(CASE WHEN measure_name IN ('memory_free', 'cpu_idle') THEN measure_value::double ELSE NULL END) as min_1h, SUM(CASE WHEN measure_name IN ('disk_io_writes') THEN measure_value::double ELSE NULL END) as sum_1h, MAX(CASE WHEN measure_name IN ('memory_used', 'cpu_user', 'cpu_system') THEN measure_value::double ELSE NULL END) as max_1h FROM raw_data.devops WHERE time BETWEEN bin(@scheduled_runtime, 1h) - 14h AND bin(@scheduled_runtime, 1h) - 2h AND measure_name IN ('memory_free', 'memory_used', 'memory_cached', 'disk_io_writes', 'disk_used', 'disk_free', 'cpu_user', 'cpu_system', 'cpu_idle') GROUP BY region, measure_name, bin(time, 1h)", "ScheduleConfiguration" : { "ScheduleExpression" : "cron(0 0/1 * * ? *)" }, "NotificationConfiguration" : { "SnsConfiguration" : { "TopicArn" : "******" } }, "ScheduledQueryExecutionRoleArn": "******", "TargetConfiguration": { "TimestreamConfiguration": { "DatabaseName" : "derived", "TableName" : "dashboard_metrics_1h_agg_2", "TimeColumn" : "hour", "DimensionMappings" : [ { "Name": "region", "DimensionValueType" : "VARCHAR" } ], "MeasureNameColumn" : "measure_name", "MultiMeasureMappings" : { "MultiMeasureAttributeMappings" : [ { "SourceColumn" : "avg_1h", "MeasureValueType" : "DOUBLE" }, { "SourceColumn" : "min_1h", "MeasureValueType" : "DOUBLE", "TargetMultiMeasureAttributeName": "p0_1h" }, { "SourceColumn" : "sum_1h", "MeasureValueType" : "DOUBLE" }, { "SourceColumn" : "max_1h", "MeasureValueType" : "DOUBLE", "TargetMultiMeasureAttributeName": "p100_1h" } ] } } }, "ErrorReportConfiguration": { "S3Configuration" : { "BucketName" : "******", "ObjectKeyPrefix": "errors", "EncryptionOption": "SSE_S3" } } }
The mapping in this example will create multi-measure records with attributes avg_1h, p0_1h, sum_1h, p100_1h and will use the values of the measure_name column in the query result as the measure name for the multi-measure records in the destination table. Additionally note that the previous examples optionally use the TargetMultiMeasureAttributeName with a subset of the mappings to rename the attributes. For instance, min_1h was renamed to p0_1h and max_1h is renamed to p100_1h.
The following is the schema for the destination table once this scheduled query is materialized. As you can see from the Timestream for LiveAnalytics attribute type in the following result, the results are materialized into a multi-measure record. If you look at the measure schema, there were nine different measure names that were ingested which correspond to the values seen in the query results.
Column | Type | Timestream for LiveAnalytics attribute type |
---|---|---|
region |
varchar |
DIMENSION |
measure_name |
varchar |
MEASURE_NAME |
time |
timestamp |
TIMESTAMP |
sum_1h |
double |
MULTI |
p100_1h |
double |
MULTI |
p0_1h |
double |
MULTI |
avg_1h |
double |
MULTI |
The following are corresponding measures obtained with a SHOW MEASURES query.
measure_name | data_type | Dimensions |
---|---|---|
cpu_idle |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
cpu_system |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
cpu_user |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
disk_free |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
disk_io_writes |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
disk_used |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
memory_cached |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
memory_free |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
memory_free |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
Example: Mapping results to different multi-measure records with different attributes
The following example shows how you can map different columns in your query result
into different multi-measure records with different measure names. If you see the
following scheduled query definition, the result of the query has the following
columns: region, hour, avg_mem_cached_1h, min_mem_free_1h, max_mem_used_1h,
total_disk_io_writes_1h, avg_disk_used_1h, avg_disk_free_1h, max_cpu_user_1h,
max_cpu_system_1h, min_cpu_system_1h. region
is mapped to dimension,
and hour
is mapped to the time column.
The MixedMeasureMappings property in TargetConfiguration.TimestreamConfiguration specifies how to map the measures to multi-measure records in the derived table.
In this specific example, avg_mem_cached_1h, min_mem_free_1h, max_mem_used_1h are used in one multi-measure record with measure name of mem_aggregates, total_disk_io_writes_1h, avg_disk_used_1h, avg_disk_free_1h are used in another multi-measure record with measure name of disk_aggregates, and finally max_cpu_user_1h, max_cpu_system_1h, min_cpu_system_1h are used in another multi-measure record with measure name cpu_aggregates.
In these mappings, you can also optionally use TargetMultiMeasureAttributeName to rename the query result column to have a different attribute name in the destination table. For instance, the result column avg_mem_cached_1h gets renamed to avgMemCached, total_disk_io_writes_1h gets renamed to totalIOWrites, etc.
When you're defining the mappings for multi-measure records, Timestream for LiveAnalytics inspects every row in the query results and automatically ignores the column values that have NULL values. As a result, in the case of mappings with multiple measures names, if all the column values for that group in the mapping are NULL for a given row, then no value for that measure name is ingested for that row.
For example, in the following mapping, avg_mem_cached_1h, min_mem_free_1h, and max_mem_used_1h are mapped to measure name mem_aggregates. If for a given row of the query result, all these of the column values are NULL, Timestream for LiveAnalytics won't ingest the measure mem_aggregates for that row. If all nine columns for a given row are NULL, then you will see an user error reported in your error report.
{ "Name" : "AggsInDifferentMultiMeasureRecords", "QueryString" : "SELECT region, bin(time, 1h) as hour, AVG(CASE WHEN measure_name = 'memory_cached' THEN measure_value::double ELSE NULL END) as avg_mem_cached_1h, MIN(CASE WHEN measure_name = 'memory_free' THEN measure_value::double ELSE NULL END) as min_mem_free_1h, MAX(CASE WHEN measure_name = 'memory_used' THEN measure_value::double ELSE NULL END) as max_mem_used_1h, SUM(CASE WHEN measure_name = 'disk_io_writes' THEN measure_value::double ELSE NULL END) as total_disk_io_writes_1h, AVG(CASE WHEN measure_name = 'disk_used' THEN measure_value::double ELSE NULL END) as avg_disk_used_1h, AVG(CASE WHEN measure_name = 'disk_free' THEN measure_value::double ELSE NULL END) as avg_disk_free_1h, MAX(CASE WHEN measure_name = 'cpu_user' THEN measure_value::double ELSE NULL END) as max_cpu_user_1h, MAX(CASE WHEN measure_name = 'cpu_system' THEN measure_value::double ELSE NULL END) as max_cpu_system_1h, MIN(CASE WHEN measure_name = 'cpu_idle' THEN measure_value::double ELSE NULL END) as min_cpu_system_1h FROM raw_data.devops WHERE time BETWEEN bin(@scheduled_runtime, 1h) - 14h AND bin(@scheduled_runtime, 1h) - 2h AND measure_name IN ('memory_cached', 'memory_free', 'memory_used', 'disk_io_writes', 'disk_used', 'disk_free', 'cpu_user', 'cpu_system', 'cpu_idle') GROUP BY region, bin(time, 1h)", "ScheduleConfiguration" : { "ScheduleExpression" : "cron(0 0/1 * * ? *)" }, "NotificationConfiguration" : { "SnsConfiguration" : { "TopicArn" : "******" } }, "ScheduledQueryExecutionRoleArn": "******", "TargetConfiguration": { "TimestreamConfiguration": { "DatabaseName" : "derived", "TableName" : "dashboard_metrics_1h_agg_3", "TimeColumn" : "hour", "DimensionMappings" : [ { "Name": "region", "DimensionValueType" : "VARCHAR" } ], "MixedMeasureMappings" : [ { "MeasureValueType" : "MULTI", "TargetMeasureName" : "mem_aggregates", "MultiMeasureAttributeMappings" : [ { "SourceColumn" : "avg_mem_cached_1h", "MeasureValueType" : "DOUBLE", "TargetMultiMeasureAttributeName": "avgMemCached" }, { "SourceColumn" : "min_mem_free_1h", "MeasureValueType" : "DOUBLE" }, { "SourceColumn" : "max_mem_used_1h", "MeasureValueType" : "DOUBLE", "TargetMultiMeasureAttributeName": "maxMemUsed" } ] }, { "MeasureValueType" : "MULTI", "TargetMeasureName" : "disk_aggregates", "MultiMeasureAttributeMappings" : [ { "SourceColumn" : "total_disk_io_writes_1h", "MeasureValueType" : "DOUBLE", "TargetMultiMeasureAttributeName": "totalIOWrites" }, { "SourceColumn" : "avg_disk_used_1h", "MeasureValueType" : "DOUBLE" }, { "SourceColumn" : "avg_disk_free_1h", "MeasureValueType" : "DOUBLE" } ] }, { "MeasureValueType" : "MULTI", "TargetMeasureName" : "cpu_aggregates", "MultiMeasureAttributeMappings" : [ { "SourceColumn" : "max_cpu_user_1h", "MeasureValueType" : "DOUBLE" }, { "SourceColumn" : "max_cpu_system_1h", "MeasureValueType" : "DOUBLE" }, { "SourceColumn" : "min_cpu_idle_1h", "MeasureValueType" : "DOUBLE", "TargetMultiMeasureAttributeName": "minCpuIdle" } ] } ] } }, "ErrorReportConfiguration": { "S3Configuration" : { "BucketName" : "******", "ObjectKeyPrefix": "errors", "EncryptionOption": "SSE_S3" } } }
The following is the schema for the destination table once this scheduled query is materialized.
Column | Type | Timestream for LiveAnalytics attribute type |
---|---|---|
region |
varchar |
DIMENSION |
measure_name |
varchar |
MEASURE_NAME |
time |
timestamp |
TIMESTAMP |
minCpuIdle |
double |
MULTI |
max_cpu_system_1h |
double |
MULTI |
max_cpu_user_1h |
double |
MULTI |
avgMemCached |
double |
MULTI |
maxMemUsed |
double |
MULTI |
min_mem_free_1h |
double |
MULTI |
avg_disk_free_1h |
double |
MULTI |
avg_disk_used_1h |
double |
MULTI |
totalIOWrites |
double |
MULTI |
The following are the corresponding measures obtained with a SHOW MEASURES query.
measure_name | data_type | Dimensions |
---|---|---|
cpu_aggregates |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
disk_aggregates |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
mem_aggregates |
multi |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
Example: Mapping results to single-measure records with measure name from query results
The following is an example of a scheduled query whose results are materialized into single-measure records. In this example, the query result has the measure_name column whose values will be used as measure names in the target table. You use the MixedMeasureMappings attribute in the TargetConfiguration.TimestreamConfiguration to specify the mapping of the query result column to the scalar measure in the target table.
In the following example definition, the query result is expected to nine distinct measure_name values. You list out all these measure names in the mapping and specify which column to use for the single-measure value for that measure name. For example, in this mapping, if measure name of memory_cached is seen for a given result row, then the value in the avg_1h column is used as the value for the measure when the data is written to the target table. You can optionally use TargetMeasureName to provide a new measure name for this value.
{ "Name" : "UsingMeasureNameColumnForSingleMeasureMapping", "QueryString" : "SELECT region, bin(time, 1h) as hour, measure_name, AVG(CASE WHEN measure_name IN ('memory_cached', 'disk_used', 'disk_free') THEN measure_value::double ELSE NULL END) as avg_1h, MIN(CASE WHEN measure_name IN ('memory_free', 'cpu_idle') THEN measure_value::double ELSE NULL END) as min_1h, SUM(CASE WHEN measure_name IN ('disk_io_writes') THEN measure_value::double ELSE NULL END) as sum_1h, MAX(CASE WHEN measure_name IN ('memory_used', 'cpu_user', 'cpu_system') THEN measure_value::double ELSE NULL END) as max_1h FROM raw_data.devops WHERE time BETWEEN bin(@scheduled_runtime, 1h) - 14h AND bin(@scheduled_runtime, 1h) - 2h AND measure_name IN ('memory_free', 'memory_used', 'memory_cached', 'disk_io_writes', 'disk_used', 'disk_free', 'cpu_user', 'cpu_system', 'cpu_idle') GROUP BY region, bin(time, 1h), measure_name", "ScheduleConfiguration" : { "ScheduleExpression" : "cron(0 0/1 * * ? *)" }, "NotificationConfiguration" : { "SnsConfiguration" : { "TopicArn" : "******" } }, "ScheduledQueryExecutionRoleArn": "******", "TargetConfiguration": { "TimestreamConfiguration": { "DatabaseName" : "derived", "TableName" : "dashboard_metrics_1h_agg_4", "TimeColumn" : "hour", "DimensionMappings" : [ { "Name": "region", "DimensionValueType" : "VARCHAR" } ], "MeasureNameColumn" : "measure_name", "MixedMeasureMappings" : [ { "MeasureName" : "memory_cached", "MeasureValueType" : "DOUBLE", "SourceColumn" : "avg_1h", "TargetMeasureName" : "AvgMemCached" }, { "MeasureName" : "disk_used", "MeasureValueType" : "DOUBLE", "SourceColumn" : "avg_1h" }, { "MeasureName" : "disk_free", "MeasureValueType" : "DOUBLE", "SourceColumn" : "avg_1h" }, { "MeasureName" : "memory_free", "MeasureValueType" : "DOUBLE", "SourceColumn" : "min_1h", "TargetMeasureName" : "MinMemFree" }, { "MeasureName" : "cpu_idle", "MeasureValueType" : "DOUBLE", "SourceColumn" : "min_1h" }, { "MeasureName" : "disk_io_writes", "MeasureValueType" : "DOUBLE", "SourceColumn" : "sum_1h", "TargetMeasureName" : "total-disk-io-writes" }, { "MeasureName" : "memory_used", "MeasureValueType" : "DOUBLE", "SourceColumn" : "max_1h", "TargetMeasureName" : "maxMemUsed" }, { "MeasureName" : "cpu_user", "MeasureValueType" : "DOUBLE", "SourceColumn" : "max_1h" }, { "MeasureName" : "cpu_system", "MeasureValueType" : "DOUBLE", "SourceColumn" : "max_1h" } ] } }, "ErrorReportConfiguration": { "S3Configuration" : { "BucketName" : "******", "ObjectKeyPrefix": "errors", "EncryptionOption": "SSE_S3" } } }
The following is the schema for the destination table once this scheduled query is materialized. As you can see from the schema, the table is using single-measure records. If you list the measure schema for the table, you will see the nine measures written to based on the mapping provided in the specification.
Column | Type | Timestream for LiveAnalytics attribute type |
---|---|---|
region |
varchar |
DIMENSION |
measure_name |
varchar |
MEASURE_NAME |
time |
timestamp |
TIMESTAMP |
measure_value::double |
double |
MEASURE_VALUE |
The following are the corresponding measures obtained with a SHOW MEASURES query.
measure_name | data_type | Dimensions |
---|---|---|
AvgMemCached |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
MinMemFree |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
cpu_idle |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
cpu_system |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
cpu_user |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
disk_free |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
disk_used |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
maxMemUsed |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
total-disk-io-writes |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
Example: Mapping results to single-measure records with query result columns as measure names
In this example, you have a query whose results do not have a measure name column. Instead, you want the query result column name as the measure name when mapping the output to single-measure records. Earlier there was an example where a similar result was written to a multi-measure record. In this example, you will see how to map it to single-measure records if that fits your application scenario.
Again, you specify this mapping using the MixedMeasureMappings property in TargetConfiguration.TimestreamConfiguration. In the following example, you see that the query result has nine columns. You use the result columns as measure names and the values as the single-measure values.
For example, for a given row in the query result, the column name avg_mem_cached_1h is used as the column name and value associated with column, and avg_mem_cached_1h is used as the measure value for the single-measure record. You can also use TargetMeasureName to use a different measure name in the target table. For instance, for values in column sum_1h, the mapping specifies to use total_disk_io_writes_1h as the measure name in the target table. If any column's value is NULL, then the corresponding measure is ignored.
{ "Name" : "SingleMeasureMappingWithoutMeasureNameColumnInQueryResult", "QueryString" : "SELECT region, bin(time, 1h) as hour, AVG(CASE WHEN measure_name = 'memory_cached' THEN measure_value::double ELSE NULL END) as avg_mem_cached_1h, AVG(CASE WHEN measure_name = 'disk_used' THEN measure_value::double ELSE NULL END) as avg_disk_used_1h, AVG(CASE WHEN measure_name = 'disk_free' THEN measure_value::double ELSE NULL END) as avg_disk_free_1h, MIN(CASE WHEN measure_name = 'memory_free' THEN measure_value::double ELSE NULL END) as min_mem_free_1h, MIN(CASE WHEN measure_name = 'cpu_idle' THEN measure_value::double ELSE NULL END) as min_cpu_idle_1h, SUM(CASE WHEN measure_name = 'disk_io_writes' THEN measure_value::double ELSE NULL END) as sum_1h, MAX(CASE WHEN measure_name = 'memory_used' THEN measure_value::double ELSE NULL END) as max_mem_used_1h, MAX(CASE WHEN measure_name = 'cpu_user' THEN measure_value::double ELSE NULL END) as max_cpu_user_1h, MAX(CASE WHEN measure_name = 'cpu_system' THEN measure_value::double ELSE NULL END) as max_cpu_system_1h FROM raw_data.devops WHERE time BETWEEN bin(@scheduled_runtime, 1h) - 14h AND bin(@scheduled_runtime, 1h) - 2h AND measure_name IN ('memory_free', 'memory_used', 'memory_cached', 'disk_io_writes', 'disk_used', 'disk_free', 'cpu_user', 'cpu_system', 'cpu_idle') GROUP BY region, bin(time, 1h)", "ScheduleConfiguration" : { "ScheduleExpression" : "cron(0 0/1 * * ? *)" }, "NotificationConfiguration" : { "SnsConfiguration" : { "TopicArn" : "******" } }, "ScheduledQueryExecutionRoleArn": "******", "TargetConfiguration": { "TimestreamConfiguration": { "DatabaseName" : "derived", "TableName" : "dashboard_metrics_1h_agg_5", "TimeColumn" : "hour", "DimensionMappings" : [ { "Name": "region", "DimensionValueType" : "VARCHAR" } ], "MixedMeasureMappings" : [ { "MeasureValueType" : "DOUBLE", "SourceColumn" : "avg_mem_cached_1h" }, { "MeasureValueType" : "DOUBLE", "SourceColumn" : "avg_disk_used_1h" }, { "MeasureValueType" : "DOUBLE", "SourceColumn" : "avg_disk_free_1h" }, { "MeasureValueType" : "DOUBLE", "SourceColumn" : "min_mem_free_1h" }, { "MeasureValueType" : "DOUBLE", "SourceColumn" : "min_cpu_idle_1h" }, { "MeasureValueType" : "DOUBLE", "SourceColumn" : "sum_1h", "TargetMeasureName" : "total_disk_io_writes_1h" }, { "MeasureValueType" : "DOUBLE", "SourceColumn" : "max_mem_used_1h" }, { "MeasureValueType" : "DOUBLE", "SourceColumn" : "max_cpu_user_1h" }, { "MeasureValueType" : "DOUBLE", "SourceColumn" : "max_cpu_system_1h" } ] } }, "ErrorReportConfiguration": { "S3Configuration" : { "BucketName" : "******", "ObjectKeyPrefix": "errors", "EncryptionOption": "SSE_S3" } } }
The following is the schema for the destination table once this scheduled query is materialized. As you can see that the target table is storing records with single-measure values of type double. Similarly, the measure schema for the table shows the nine measure names. Also notice that the measure name total_disk_io_writes_1h is present since the mapping renamed sum_1h to total_disk_io_writes_1h.
Column | Type | Timestream for LiveAnalytics attribute type |
---|---|---|
region |
varchar |
DIMENSION |
measure_name |
varchar |
MEASURE_NAME |
time |
timestamp |
TIMESTAMP |
measure_value::double |
double |
MEASURE_VALUE |
The following are the corresponding measures obtained with a SHOW MEASURES query.
measure_name | data_type | Dimensions |
---|---|---|
avg_disk_free_1h |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
avg_disk_used_1h |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
avg_mem_cached_1h |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
max_cpu_system_1h |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
max_cpu_user_1h |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
max_mem_used_1h |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
min_cpu_idle_1h |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
min_mem_free_1h |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |
total-disk-io-writes |
double |
[{'dimension_name': 'region', 'data_type': 'varchar'}] |