View a markdown version of this page

Collect Slurm metrics with a managed Prometheus collector - AWS PCS
PrerequisitesConfigure security groups for the collectorCreate the managed collectorFind the cluster controller endpointSuggested scrape configurationVerify metrics deliveryQuery Slurm metrics with PromQL

Collect Slurm metrics with a managed Prometheus collector

After you enable Slurm metrics on your AWS PCS cluster (see Slurm metrics in AWS PCS), you can use a managed Prometheus collector to automatically scrape the metrics endpoints and deliver the data for PromQL querying. The managed collector creates elastic network interfaces in your VPC subnets to reach the cluster controller's metrics endpoints on port 6817.

The collector can deliver metrics to either of the following destinations:

  • Amazon Managed Service for Prometheus workspace – A dedicated Prometheus-compatible metrics store with configurable retention (150 days by default). You can query through Prometheus-compatible APIs or Grafana.

  • CloudWatch dataset – Your account's default CloudWatch dataset with 15 months of included retention. You can query through CloudWatch Query Studio or the Prometheus-compatible HTTP API.

The cluster-side configuration (Slurm settings, controller target, security groups, and scrape configuration) is the same regardless of which destination you choose. Only the destination block in the create-scraper request and the query endpoint differ.

You create the collector with the aws amp create-scraper command. Although this command belongs to the amp CLI namespace, it supports both destinations.

For more information about VPC-connected managed collectors, see Set up a VPC-connected managed collector in the Amazon CloudWatch User Guide.

Prerequisites

Before you configure the managed collector, verify the following:

  • Slurm metrics enabled – The metrics endpoint must be active on your cluster. You enable it by setting the MetricsType and CommunicationParameters custom Slurm settings. For instructions on enabling Slurm metrics, see Slurm metrics in AWS PCS. For more information about custom Slurm settings, see Configuring custom Slurm settings in AWS PCS.

  • Slurm version 25.11 or higher – The cluster must run Slurm 25.11 or higher to expose the metrics endpoint.

  • Destination resource – Create the destination for your metrics:

    • Amazon Managed Service for Prometheus workspace – Create a workspace and wait for it to reach ACTIVE status. For instructions on creating a workspace, see Create a workspace in the Amazon Managed Service for Prometheus User Guide.

    • CloudWatch dataset – Every account has a default dataset in each Region. You do not need to create it.

  • VPC subnets and networking – You need at least two subnets in different Availability Zones within the same VPC as the cluster controller. Include the Availability Zone where the controller's network interface resides. The VPC must have DNS support and DNS hostnames enabled. For detailed networking requirements, see Set up a VPC-connected managed collector in the Amazon CloudWatch User Guide.

  • Security groups – A dedicated security group for the collector is required. For instructions on configuring security groups, see Configure security groups for the collector.

  • IAM permissions – The IAM principal that creates the scraper needs aps:CreateScraper and iam:CreateServiceLinkedRole permissions. The service automatically creates a service-linked role (AWSServiceRoleForAmazonPrometheusScraper). This role grants the collector permission to access your VPC resources and write to your chosen destination. No manual role setup is required. For more information, see Using service-linked roles in the Amazon Managed Service for Prometheus User Guide.

  • Internet or VPC endpoint – The collector subnets must be able to reach the destination service. If your subnets have no internet access, create an interface VPC endpoint in the same VPC and subnets. Use com.amazonaws.region.aps-workspaces for an Amazon Managed Service for Prometheus workspace destination, or com.amazonaws.region.monitoring for a CloudWatch dataset destination.

Configure security groups for the collector

We recommend creating a dedicated security group for the managed collector rather than reusing an existing security group. A dedicated group gives you explicit, auditable rules that follow the principle of least privilege.

Important

The enable_http setting exposes an unauthenticated HTTP endpoint on port 6817. Restrict inbound access on this port to the collector's security group only. Do not allow broad network access (such as a CIDR range) to this port.

The following procedure uses shell variables for security group IDs. Set these variables before you run the commands:

  • VPC_ID – The ID of the VPC where your AWS PCS cluster resides.

  • CLUSTER_SG_ID – The ID of the security group attached to your cluster (the one you specified when you created the cluster).

  • VPCE_SG_ID – The ID of the security group attached to your interface VPC endpoint. This variable is only needed if your collector subnets reach the destination service through an interface VPC endpoint rather than through internet or NAT egress. The endpoint is com.amazonaws.region.aps-workspaces for an Amazon Managed Service for Prometheus workspace destination or com.amazonaws.region.monitoring for a CloudWatch dataset destination.

To configure security groups for the managed Prometheus collector

  1. Create a dedicated security group for the collector and capture the group ID:

    COLLECTOR_SG_ID=$(aws ec2 create-security-group \
    --group-name "pcs-prometheus-collector" \
    --description "Security group for managed Prometheus collector" \
    --vpc-id "$VPC_ID" \
    --query 'GroupId' \
    --output text)

  2. Add an inbound rule to the cluster's security group that allows TCP traffic on port 6817 from the collector's security group. This rule permits the collector to scrape the Slurm metrics endpoint on the controller.

    Use the --ip-permissions form to include a rule description for auditability:

    aws ec2 authorize-security-group-ingress \
    --group-id "$CLUSTER_SG_ID" \
    --ip-permissions \
    'IpProtocol=tcp,FromPort=6817,ToPort=6817,UserIdGroupPairs=[{GroupId='"$COLLECTOR_SG_ID"',Description="Prometheus collector scrapes Slurm metrics"}]'

    Alternatively, you can use the shorter form without a rule description:

    aws ec2 authorize-security-group-ingress \
    --group-id "$CLUSTER_SG_ID" \
    --protocol tcp \
    --port 6817 \
    --source-group "$COLLECTOR_SG_ID"

  3. (Optional) Lock down outbound traffic on the collector's security group. By default, a newly created security group allows all outbound traffic. If you want to enforce explicit-only egress for a higher security posture, revoke the default allow-all rule and add only the egress rules that the collector requires.

    Revoke the default allow-all egress rule:

    aws ec2 revoke-security-group-egress \
    --group-id "$COLLECTOR_SG_ID" \
    --ip-permissions '[{"IpProtocol":"-1","IpRanges":[{"CidrIp":"0.0.0.0/0"}]}]'

    Add an explicit egress rule to allow the collector to reach the controller on TCP 6817:

    aws ec2 authorize-security-group-egress \
    --group-id "$COLLECTOR_SG_ID" \
    --ip-permissions \
    'IpProtocol=tcp,FromPort=6817,ToPort=6817,UserIdGroupPairs=[{GroupId='"$CLUSTER_SG_ID"',Description="Egress to Slurm controller for metrics scraping"}]'

    Add an explicit egress rule for HTTPS (TCP 443) to reach the metrics delivery destination (Amazon Managed Service for Prometheus or CloudWatch):

    aws ec2 authorize-security-group-egress \
    --group-id "$COLLECTOR_SG_ID" \
    --ip-permissions \
    'IpProtocol=tcp,FromPort=443,ToPort=443,IpRanges=[{CidrIp=0.0.0.0/0,Description="HTTPS egress for metrics delivery"}]'

    For tighter control in subnets that use a VPC endpoint for delivery, replace the CIDR range with the VPC endpoint's security group:

    aws ec2 authorize-security-group-egress \
    --group-id "$COLLECTOR_SG_ID" \
    --ip-permissions \
    'IpProtocol=tcp,FromPort=443,ToPort=443,UserIdGroupPairs=[{GroupId='"$VPCE_SG_ID"',Description="HTTPS egress to VPC endpoint for metrics delivery"}]'
    Note

    If you did not revoke the default egress rule, you can skip this step. The default rule already permits all outbound traffic, including traffic to port 6817 and port 443.

  4. (Isolated clusters) If your collector subnets have no internet access and use an interface VPC endpoint to reach the destination service, add an inbound rule to the VPC endpoint's security group. This rule allows the collector's HTTPS traffic to reach the endpoint network interfaces. This is a commonly missed step.

    aws ec2 authorize-security-group-ingress \
    --group-id "$VPCE_SG_ID" \
    --ip-permissions \
    'IpProtocol=tcp,FromPort=443,ToPort=443,UserIdGroupPairs=[{GroupId='"$COLLECTOR_SG_ID"',Description="Managed collector reaches service endpoint"}]'
Note

As a simpler but less restrictive alternative, you can attach the controller's existing security group to the collector if that security group contains a self-referencing rule that allows traffic from itself. This satisfies the connectivity requirement without creating a dedicated group.

For example, if your cluster's security group (sg-0abc1234def56789a) already allows all TCP traffic from itself, pass that security group ID in the --security-group-ids parameter when you create the scraper:

aws amp create-scraper \
    --source '{"vpcConfiguration":{"subnetIds":["subnet-id-1","subnet-id-2"],"securityGroupIds":["sg-0abc1234def56789a"]}}' \
    ...

However, the dedicated security group approach described in the preceding procedure provides a higher security bar with explicit, auditable rules.

For more information about security group rules, see Security group rules in the Amazon VPC User Guide.

Create the managed collector

Use the AWS CLI to create a VPC-connected managed collector that scrapes your cluster controller and delivers metrics to your chosen destination.

To create a managed collector for Slurm metrics

  1. Save the scrape configuration to a local YAML file named scrape-config.yaml. For a suggested configuration, see Suggested scrape configuration.

  2. Create a JSON input file named create-scraper-input.json with the following structure. Choose the destination block that matches your target.

    {
  "source": {
    "vpcConfiguration": {
      "subnetIds": ["subnet-1", "subnet-2"],
      "securityGroupIds": ["sg-collector"]
    }
  },
  "destination": { ... },
  "scrapeConfiguration": {
    "configurationBlob": "raw-yaml-contents"
  }
}

    Replace:

    • subnet-1 and subnet-2 – At least two subnet IDs in different Availability Zones within the same VPC as the cluster controller.

    • sg-collector – The security group ID for the managed collector.

    • raw-yaml-contents – The full text of your scrape-config.yaml file, pasted as a single JSON string value. The AWS CLI base64-encodes the value on the wire automatically.

    For the destination field, use one of the following:

    Destination: Amazon Managed Service for Prometheus workspace

    "destination": {
  "ampConfiguration": {
    "workspaceArn": "arn:aws:aps:region:account-id:workspace/workspace-id"
  }
}

    Destination: CloudWatch dataset

    "destination": {
  "cloudWatchConfiguration": {
    "datasetArn": "arn:aws:cloudwatch:region:account-id:dataset/default"
  }
}

    For the full set of parameters, see create-scraper in the AWS CLI Command Reference.

  3. Create the scraper:

    aws amp create-scraper --cli-input-json file://create-scraper-input.json

    The command returns a scraperId and a status of CREATING.

  4. Wait for the scraper status to change to ACTIVE (typically 5–15 minutes):

    aws amp describe-scraper --scraper-id scraper-id

    Replace scraper-id with the ID returned in the previous step. You cannot delete a scraper until it reaches ACTIVE status.

Find the cluster controller endpoint

The scrape configuration requires the private IP address of your AWS PCS cluster controller. Use one of the following methods to find it.

AWS Management Console

  1. Open the AWS PCS console at https://console.aws.amazon.com/pcs/.

  2. Choose your cluster from the list.

  3. In the cluster configuration details, locate the Endpoints section.

  4. Note the private IP address and port for Slurm controller (slurmctld). The port is 6817.

AWS CLI

  1. Run the following command. Replace cluster-identifier with your cluster name or ID.

    aws pcs get-cluster --cluster-identifier cluster-identifier

    In the response, locate the SLURMCTLD entry in the endpoints array. The privateIpAddress value is the controller endpoint you need for the scrape configuration. Here is an example:

    "endpoints": [
    {
        "type": "SLURMCTLD",
        "privateIpAddress": "192.0.2.1",
        "port": "6817"
    },
    {
        "type": "SLURMRESTD",
        "privateIpAddress": "192.0.2.1",
        "port": "6820"
    }
]

    Use the privateIpAddress from the SLURMCTLD entry (port 6817) as the controller-endpoint value in your scrape configuration.

  2. Alternatively, extract only the controller IP address directly:

    aws pcs get-cluster --cluster-identifier cluster-identifier \
    --query 'cluster.endpoints[?type==`SLURMCTLD`].privateIpAddress' \
    --output text

Suggested scrape configuration

The following YAML configuration scrapes four Slurm metrics endpoints (jobs, nodes, scheduler, and partitions) from the cluster controller. Each endpoint is defined as a separate job so that you can identify metrics by source in your queries.

global:
  scrape_interval: 60s
  scrape_timeout: 30s

scrape_configs:
  - job_name: 'slurm-jobs'
    metrics_path: /metrics/jobs
    static_configs:
      - targets:
          - 'controller-endpoint:6817'
    relabel_configs:
      - target_label: cluster
        replacement: 'my-cluster-name'

  - job_name: 'slurm-nodes'
    metrics_path: /metrics/nodes
    static_configs:
      - targets:
          - 'controller-endpoint:6817'
    relabel_configs:
      - target_label: cluster
        replacement: 'my-cluster-name'

  - job_name: 'slurm-scheduler'
    metrics_path: /metrics/scheduler
    static_configs:
      - targets:
          - 'controller-endpoint:6817'
    relabel_configs:
      - target_label: cluster
        replacement: 'my-cluster-name'

  - job_name: 'slurm-partitions'
    metrics_path: /metrics/partitions
    static_configs:
      - targets:
          - 'controller-endpoint:6817'
    relabel_configs:
      - target_label: cluster
        replacement: 'my-cluster-name'

Replace:

  • controller-endpoint – The private IP address of your AWS PCS cluster controller. For instructions on finding this value, see Find the cluster controller endpoint.

  • my-cluster-name – A label that identifies your cluster. The relabel_configs block stamps a cluster label on every metric from this scraper. Always filter queries on the cluster label. Series from a scraper that did not stamp the label appear as duplicate unlabeled series until they age out.

The minimum scrape_interval for a managed collector is 30 seconds. This configuration uses 60 seconds because scraping places load on the Slurm controller (slurmctld). Querying metrics acquires internal locks and reads in-memory data structures. This activity can affect scheduler performance on busy clusters. The Slurm Metrics Guide recommends a scrape interval of 60–120 seconds to minimize performance impact.

Note

This configuration does not include the /metrics/jobs-users-accts endpoint. Slurm documentation warns that this endpoint produces an unbounded number of series and is unsuitable for stored monitoring. Do not scrape the bare /metrics index either, because it combines all sub-endpoint data into a single response.

For more information about supported scrape configuration options, see Scraper configuration in the Amazon CloudWatch User Guide.

Verify metrics delivery

After the scraper reaches ACTIVE status, the first datapoints appear approximately one scrape interval plus delivery time later. Use the verification method that matches your destination.

Verify delivery to an Amazon Managed Service for Prometheus workspace

Send a SigV4-signed request to list available metric names from your workspace:

awscurl --service aps --region region \
    "https://aps-workspaces.region.amazonaws.com/workspaces/workspace-id/api/v1/label/__name__/values"

The calling principal needs aps:QueryMetrics and aps:GetLabels permissions (or the AmazonPrometheusQueryAccess managed policy).

Verify delivery to a CloudWatch dataset

Send a SigV4-signed request to list available metric names from your CloudWatch dataset:

awscurl --service monitoring --region region \
    "https://monitoring.region.amazonaws.com/api/v1/label/__name__/values"

The calling principal needs cloudwatch:GetMetricData and cloudwatch:ListMetrics permissions.

Important

Metrics delivered to a CloudWatch dataset are stored as OpenTelemetry (OTel) metrics. They do not appear in the classic CloudWatch Metrics namespace browser or in the output of aws cloudwatch list-metrics. You must query them with PromQL.

For either destination, look for metric names that begin with slurm_, such as slurm_nodes, slurm_jobs_running, or slurm_node_cpus. If no Slurm metrics appear, verify the following:

  • The scraper status is ACTIVE.

  • Security group rules allow the collector to reach port 6817 on the controller.

  • The Slurm metrics endpoint is enabled on the cluster.

Query Slurm metrics with PromQL

You query the collected Slurm metrics using PromQL. The same queries work against either destination. The query method depends on where you delivered the metrics.

Query an Amazon Managed Service for Prometheus workspace

  • HTTP API – Send SigV4-signed requests (service name aps) to https://aps-workspaces.region.amazonaws.com/workspaces/workspace-id/api/v1/query or /api/v1/query_range.

  • Grafana – Add a Prometheus data source with SigV4 authentication and service name aps. For instructions on querying with Grafana, see Query using Grafana in the Amazon Managed Service for Prometheus User Guide.

The following example uses awscurl to query running jobs from an Amazon Managed Service for Prometheus workspace:

awscurl --service aps --region region \
    -X POST "https://aps-workspaces.region.amazonaws.com/workspaces/workspace-id/api/v1/query" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "query=slurm_jobs_running"

The calling principal needs aps:QueryMetrics, aps:GetMetricMetadata, aps:GetSeries, and aps:GetLabels permissions (or the AmazonPrometheusQueryAccess managed policy).

Query a CloudWatch dataset

  • CloudWatch console – Open CloudWatch, choose Query Studio, and select PromQL from the query language menu.

  • HTTP API – Send SigV4-signed requests (service name monitoring) to https://monitoring.region.amazonaws.com/api/v1/query or /api/v1/query_range.

  • Grafana – Add a Prometheus data source with URL https://monitoring.region.amazonaws.com, SigV4 authentication, and service name monitoring. For instructions on querying CloudWatch metrics with PromQL in Grafana, see Query CloudWatch metrics with PromQL in Grafana in the Amazon Managed Grafana User Guide.

The following example uses awscurl to query running jobs from a CloudWatch dataset:

awscurl --service monitoring --region region \
    -X POST "https://monitoring.region.amazonaws.com/api/v1/query" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    -d "query=slurm_jobs_running"

The calling principal needs cloudwatch:GetMetricData and cloudwatch:ListMetrics permissions.

Example PromQL queries

The following queries work against either destination. Replace my-cluster-name with the value you set in your scrape configuration's cluster relabel.

CPU utilization percentage
100 * slurm_node_cpus_alloc{cluster="my-cluster-name"} / slurm_node_cpus{cluster="my-cluster-name"}
Pending jobs (queue backlog)
slurm_jobs_pending{cluster="my-cluster-name"}
Running jobs
slurm_jobs_running{cluster="my-cluster-name"}
Job throughput
slurm_jobs_completed{cluster="my-cluster-name"}

For more information about available metrics and scraping configuration, see the Metrics Guide on the Slurm website.