input:
path input_file
val name
val threshold
output:
path 'results.txt', emit: results
script:
"""
my_tool --name ${name} --threshold ${threshold} ${input_file}
"""
}
workflow MY_WORKFLOW {
my_task(
params.input_file,
params.name,
params.threshold
)
}
workflow {
MY_WORKFLOW()
}
```
------
#### [ CWL ]
```
cwlVersion: v1.2
class: Workflow
requirements:
InlineJavascriptRequirement: {}
inputs:
input_file: File
name: string
threshold: int
outputs:
result:
type: ...
outputSource: ...
steps:
my_task:
run:
class: CommandLineTool
baseCommand: my_tool
requirements:
...
inputs:
name:
type: string
inputBinding:
prefix: "--name"
threshold:
type: int
inputBinding:
prefix: "--threshold"
input_file:
type: File
inputBinding: {}
outputs:
results:
type: File
outputBinding:
glob: results.txt
```
------
# Parameter template files for HealthOmics workflows
Parameter templates define the input parameters for a workflow. You can define input parameters to make your workflow more flexible and versatile. For example, you can define a parameter for the Amazon S3 location of the reference genome files. Parameter templates can be provided through a Git-based repository service or your local drive. Users can then run the workflow using various data sets.
You can create the parameter template for your workflow, or HealthOmics can generate the parameter template for you.
The parameter template is a JSON file. In the file, each input parameter is a named object that must match the name of the workflow input. When you start a run, if you don't provide values for all the required parameters, the run fails.
The input parameter object includes the following attributes:
+ **description** – This required attribute is a string that the console displays in the **Start run** page. This description is also retained as run metadata.
+ **optional** – This optional attribute indicates whether the input parameter is optional. If you don't specify the **optional** field, the input parameter is required.
The following example parameter template shows how to specify the input parameters.
```
{
"myRequiredParameter1": {
"description": "this parameter is required",
},
"myRequiredParameter2": {
"description": "this parameter is also required",
"optional": false
},
"myOptionalParameter": {
"description": "this parameter is optional",
"optional": true
}
}
```
## Generating parameter templates
HealthOmics generates the parameter template by parsing the workflow definition to detect input parameters. If you provide a parameter template file for a workflow, the parameters in your file override the parameters detected in the workflow definition.
There are slight differences between the parsing logic of the CWL, WDL, and Nextflow engines, as described in the following sections.
**Topics**
+ [Parameter detection for CWL](#parameter-parsing-cwl)
+ [Parameter detection for WDL](#parameter-parsing-wdl)
+ [Parameter detection for Nextflow](#parameter-parsing-nextflow)
### Parameter detection for CWL
In the CWL workflow engine, the parsing logic makes the following assumptions:
+ Any nullable supported types are marked as optional input parameters.
+ Any non-null supported types are marked as required input parameters.
+ Any parameters with default values are marked as optional input parameters.
+ Descriptions are extracted from the `label` section from the `main` workflow definition. If `label` is not specified, the description will be blank (an empty string).
The following tables show CWL interpolation examples. For each example, the parameter name is `x`. If the parameter is required, you must provide a value for the parameter. If the parameter is optional, you don't need to provide a value.
This table shows CWL interpolation examples for primitive types.
| Input | Example input/output | Required |
| --- | --- | --- |
| x:
type: int
| 1 or 2 or ... | Yes |
| x:
type: int
default: 2
| Default value is 2. Valid input is 1 or 2 or ... | No |
| x:
type: int?
| Valid input is None or 1 or 2 or ... | No |
| x:
type: int?
default: 2
| Default value is 2. Valid input is None or 1 or 2 or ... | No |
The following table shows CWL interpolation examples for complex types. A complex type is a collection of primitive types.
| Input | Example input/output | Required |
| --- | --- | --- |
| x:
type: array
items: int
| [] or [1,2,3] | Yes |
| x:
type: array?
items: int
| None or [] or [1,2,3] | No |
| x:
type: array
items: int?
| [] or [None, 3, None] | Yes |
| x:
type: array?
items: int?
| [None] or None or [1,2,3] or [None, 3] but not [] | No |
### Parameter detection for WDL
In the WDL workflow engine, the parsing logic makes the following assumptions:
+ Any nullable supported types are marked as optional input parameters.
+ For non-nullable supported types:
+ Any input variable with assignment of literals or expression are marked as optional parameters. For example:
```
Int x = 2
Float f0 = 1.0 + f1
```
+ If no values or expressions have been been assigned to the input parameters, they will be marked as required parameters.
+ Descriptions are extracted from `parameter_meta` in the `main` workflow definition. If `parameter_meta` is not specified, the description will be blank (an empty string). For more information, see the WDL specification for [Parameter metadata](https://github.com/openwdl/wdl/blob/wdl-1.2/SPEC.md#metadata-sections).
The following tables show WDL interpolation examples. For each example, the parameter name is `x`. If the parameter is required, you must provide a value for the parameter. If the parameter is optional, you don't need to provide a value.
This table shows WDL interpolation examples for primitive types.
| Input | Example input/output | Required |
| --- | --- | --- |
| Int x | 1 or 2 or ... | Yes |
| Int x = 2 | 2 | No |
| Int x = 1\$12 | 3 | No |
| Int x = y\$1z | y\$1z | No |
| Int? x | None or 1 or 2 or ... | Yes |
| Int? x = 2 | None or 2 | No |
| Int? x = 1\$12 | None or 3 | No |
| Int? x = y\$1z | None or y\$1z | No |
The following table shows WDL interpolation examples for complex types. A complex type is a collection of primitive types.
| Input | Example input/output | Required |
| --- | --- | --- |
| Array[Int] x | [1,2,3] or [] | Yes |
| Array[Int]\$1 x | [1], but not [] | Yes |
| Array[Int]? x | None or [] or [1,2,3] | No |
| Array[Int?] x | [] or [None, 3, None] | Yes |
| Array[Int?]=? x | [None] or None or [1,2,3] or [None, 3] but not [] | No |
| Struct sample \$1String a, Int y\$1 later in inputs: Sample mySample | String a = mySample.a
Int y = mySample.y
| Yes |
| Struct sample \$1String a, Int y\$1 later in inputs: Sample? mySample | if (defined(mySample)) {
String a = mySample.a
Int y = mySample.y
} | No |
### Parameter detection for Nextflow
For Nextflow, HealthOmics generates the parameter template by parsing the `nextflow_schema.json` file. If the workflow definition doesn't include a schema file, HealthOmics parses the main workflow definition file.
**Topics**
+ [Parsing the schema file](#parameter-parsing-nextflow-schema)
+ [Parsing the main file](#parameter-parsing-nextflow-main)
+ [Nested parameters](#parameter-parsing-nextflow-nested)
+ [Examples of Nextflow interpolation](#parameter-parsing-nextflow-examples)
#### Parsing the schema file
For parsing to work correctly, make sure the schema file meets the following requirements:
+ The schema file is named `nextflow_schema.json` and is located in the same directory as the main workflow file.
+ The schema file is valid JSON as defined in either of the following schemas:
+ [json-schema.org/draft/2020-12/schema](https://json-schema.org/draft/2020-12/schema).
+ [json-schema.org/draft-07/schema](https://json-schema.org/draft-07/schema).
HealthOmics parses the `nextflow_schema.json` file to generate the parameter template:
+ Extracts all **properties** that are defined in the schema.
+ Includes the property **description** if available for the property.
+ Identifies whether each parameter is optional or required, based on the **required** field of the property.
The following example shows a definition file and the generated parameter file.
```
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"$defs": {
"input_options": {
"title": "Input options",
"type": "object",
"required": ["input_file"],
"properties": {
"input_file": {
"type": "string",
"format": "file-path",
"pattern": "^s3://[a-z0-9.-]{3,63}(?:/\\S*)?$",
"description": "description for input_file"
},
"input_num": {
"type": "integer",
"default": 42,
"description": "description for input_num"
}
}
},
"output_options": {
"title": "Output options",
"type": "object",
"required": ["output_dir"],
"properties": {
"output_dir": {
"type": "string",
"format": "file-path",
"description": "description for output_dir",
}
}
}
},
"properties": {
"ungrouped_input_bool": {
"type": "boolean",
"default": true
}
},
"required": ["ungrouped_input_bool"],
"allOf": [
{ "$ref": "#/$defs/input_options" },
{ "$ref": "#/$defs/output_options" }
]
}
```
The generated parameter template:
```
{
"input_file": {
"description": "description for input_file",
"optional": False
},
"input_num": {
"description": "description for input_num",
"optional": True
},
"output_dir": {
"description": "description for output_dir",
"optional": False
},
"ungrouped_input_bool": {
"description": None,
"optional": False
}
}
```
#### Parsing the main file
If the workflow definition doesn't include a `nextflow_schema.json` file, HealthOmics parses the main workflow definition file.
HealthOmics analyzes the `params` expressions found in the main workflow definition file and in the `nextflow.config` file. All `params` with default values are marked as optional.
For parsing to work correctly, note the following requirements:
+ HealthOmics parses only the main workflow definition file. To ensure all parameters are captured, we recommend that you wire all **params** through to any submodules and imported workflows.
+ The config file is optional. If you define one, name it `nextflow.config` and place it in the same directory as the main workflow definition file.
The following example shows a definition file and the generated parameter template.
```
params.input_file = "default.txt"
params.threads = 4
params.memory = "8GB"
workflow {
if (params.version) {
println "Using version: ${params.version}"
}
}
```
The generated parameter template:
```
{
"input_file": {
"description": None,
"optional": True
},
"threads": {
"description": None,
"optional": True
},
"memory": {
"description": None,
"optional": True
},
"version": {
"description": None,
"optional": False
}
}
```
For default values that are defined in nextflow.config, HealthOmics collects `params` assignments and parameters declared within `params {}`, as shown in the following example. In assignment statements, `params` must appear in the left side of the statement.
```
params.alpha = "alpha"
params.beta = "beta"
params {
gamma = "gamma"
delta = "delta"
}
env {
// ignored, as this assignment isn't in the params block
VERSION = "TEST"
}
// ignored, as params is not on the left side
interpolated_image = "${params.cli_image}"
```
The generated parameter template:
```
{
// other params in your main workflow defintion
"alpha": {
"description": None,
"optional": True
},
"beta": {
"description": None,
"optional": True
},
"gamma": {
"description": None,
"optional": True
},
"delta": {
"description": None,
"optional": True
}
}
```
#### Nested parameters
Both `nextflow_schema.json` and `nextflow.config` allow nested parameters. However, the HealthOmics parameter template requires only the top-level parameters. If your workflow uses a nested parameter, you must provide a JSON object as the input for that parameter.
##### Nested parameters in schema files
HealthOmics skips nested **params** when parsing a `nextflow_schema.json` file. For example, if you define the following `nextflow_schema.json` file:
```
{
"properties": {
"input": {
"properties": {
"input_file": { ... },
"input_num": { ... }
}
},
"input_bool": { ... }
}
}
```
HealthOmics ignores `input_file` and `input_num` when it generates the parameter template:
```
{
"input": {
"description": None,
"optional": True
},
"input_bool": {
"description": None,
"optional": True
}
}
```
When you run this workflow, HealthOmics expects an `input.json` file similar to the following:
```
{
"input": {
"input_file": "s3://bucket/obj",
"input_num": 2
},
"input_bool": false
}
```
##### Nested parameters in config files
HealthOmics doesn't collect nested **params** in a `nextflow.config` file, and skips them during parsing. For example, if you define the following `nextflow.config` file:
```
params.alpha = "alpha"
params.nested.beta = "beta"
params {
gamma = "gamma"
group {
delta = "delta"
}
}
```
HealthOmics ignores `params.nested.beta` and `params.group.delta` when it generates the parameter template:
```
{
"alpha": {
"description": None,
"optional": True
},
"gamma": {
"description": None,
"optional": True
}
}
```
#### Examples of Nextflow interpolation
The following table shows Nextflow interpolation examples for params in the main file.
| Parameters | Required |
| --- | --- |
| params.input\$1file | Yes |
| params.input\$1file = "s3://bucket/data.json" | No |
| params.nested.input\$1file | N/A |
| params.nested.input\$1file = "s3://bucket/data.json" | N/A |
The following table shows Nextflow interpolation examples for params in the `nextflow.config` file.
| Parameters | Required |
| --- | --- |
| params.input_file = "s3://bucket/data.json"
| No |
| params {
input_file = "s3://bucket/data.json"
} | No |
| params {
nested {
input_file = "s3://bucket/data.json"
}
} | N/A |
| input_file = params.input_file
| N/A |
# Container images for private workflows
HealthOmics supports container images hosted in Amazon ECR private repositories. You can create container images and upload them to the private repository. You can also use your Amazon ECR private registry as a pull through cache to synchronize the contents of upstream registries.
Your Amazon ECR repository must reside in the same AWS Region as the account calling the service. A different AWS account can own the container image, as long as the source image repository provides appropriate permissions. For more information, see [Policies for cross-account Amazon ECR access](permissions-ecr.md#permissions-cross-account).
We recommend that you define your Amazon ECR container image URIs as parameters in your workflow so that access can be verified before the run begins. It also makes it easier to run a workflow in a new Region by changing the Region parameter.
**Note**
HealthOmics doesn't support ARM containers and doesn't support access to public repositories.
For information about configuring IAM permissions for HealthOmics to access Amazon ECR, see [HealthOmics Resource permissions](permissions-resource.md).
**Topics**
+ [Synchronizing with third-party container registries](#ecr-pull-through)
+ [General considerations for Amazon ECR container images](#ecr-considerations)
+ [Environment variables for HealthOmics workflows](#ecr-env-vars)
+ [Using Java in Amazon ECR container images](#ecr-java-considerations)
+ [Add task inputs to an Amazon ECR container image](#ecr-tasks)
## Synchronizing with third-party container registries
You can use Amazon ECR pull through cache rules to synchronize repositories in a supported upstream registry with your Amazon ECR private repositories. For more information, see [Sync an upstream registry](https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache.html) in the *Amazon ECR User Guide*.
The pull through cache automatically creates the image repository in your private registry when you create the cache, and it automatically synchronizes with the cached image when there are changes to the upstream image.
HealthOmics supports pull through cache for the following upstream registries:
+ Amazon ECR Public
+ Kubernetes container image registry
+ Quay
+ Docker Hub
+ Microsoft Azure Container Registry
+ GitHub Container Registry
+ GitLab Container Registry
HealthOmics doesn't support pull through cache for an upstream Amazon ECR private repository.
Benefits of using Amazon ECR pull through cache include:
1. You avoid having to manually migrate container images to Amazon ECR or to synchronize updates from the third party repository.
1. Workflows access the synchronized container images in your private repository, which is more reliable than downloading content at run time from a public registry.
1. Because Amazon ECR pull through caches use a predictable URI structure, the HealthOmics service can automatically map the Amazon ECR private URI with the upstream registry URI. You aren't required to update and replace URI values in the workflow definition.
**Topics**
+ [Configuring pull through cache](#ecr-pull-through-configure)
+ [Registry mappings](#ecr-pull-through-registry-mapping)
+ [Image mappings](#ecr-pull-through-mapping-format)
### Configuring pull through cache
Amazon ECR provides a registry for your AWS account in each Region. Make sure you create the Amazon ECR configuration in the same region where you plan to run the workflow.
The following sections describe the configuration tasks for pull through cache.
**Topics**
+ [Create a pull through cache rule](#create-ecr-ptc)
+ [Registry permissions for upstream registry](#reg-ecr-ptc)
+ [Repository creation templates](#repo-create-templates-ptc)
+ [Creating the workflow](#reg-mapping-ecr-ptc)
#### Create a pull through cache rule
Create an Amazon ECR pull through cache rule for each upstream registry that has images you want to cache. A rule specifies a mapping between an upstream registry and the Amazon ECR private repository.
For an upstream registry that requires authentication, you provide your credentials using AWS Secrets Manager.
**Note**
Don't change a pull through cache rule while an active run is using the private repository. The run could fail or, more critically, result in your pipeline using unexpected images.
For more information, see [Creating a pull through cache rule](https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache-creating-rule.html) in the *Amazon Elastic Container Registry User Guide*.
##### Create a pull through cache rule using the console
To configure pull through cache, follow these steps using the Amazon ECR console:
1. Open the Amazon ECR console : https://console.aws.amazon.com/ecr
1. From the left menu, under **Private registry**, expand **Features & Settings**. then choose **Pull through cache**.
1. From the **Pull through cache** page, choose **Add rule**.
1. In the **Upstream registry** panel, choose the upstream registry to sync with your private registry, then choose **Next**.
1. If the upstream registry requires authentication, the console opens a new page where you specify the SageMaker AI secret that contains your credentials. Choose **Next**.
1. Under **Specify namespaces**, in the **Cache namespace** panel, choose whether to create the private repositories using a specific repository prefix or with no prefix. If you choose to use a prefix, specify the prefix name in **Cache repository prefix**.
1. In the **Upstream namespace** panel, choose whether to pull from upstream repositories using a specific repository prefix or with no prefix. If you choose to use a prefix, specify the prefix name in **Upstream repository prefix**.
The **Namespace example** panel shows an example pull request, upstream URL, and the URL of the cache repository that is created.
1. Choose **Next**.
1. Review the configuration and choose **Create** to create the rule.
For more information, see [ Create a pull through cache rule (AWS Management Console)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache-creating-rule.html#pull-through-cache-creating-rule-console).
##### Create a pull through cache rule using the CLI
Use the Amazon ECR **create-pull-through-cache-rule** command to create a pull through cache rule. For upstream registries that require authentication, store the credentials in an Secrets Manager secret.
The following sections provide examples for each supported upstream registry.
##### For Amazon ECR Public
The following example creates a pull through cache rule for the Amazon ECR Public registry. It specifies a repository prefix of `ecr-public`, which results in each repository created using the pull through cache rule to have the naming scheme of `ecr-public/upstream-repository-name`.
```
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix ecr-public \
--upstream-registry-url public.ecr.aws \
--region us-east-1
```
##### For Kubernetes Container Registry
The following example creates a pull through cache rule for the Kubernetes public registry. It specifies a repository prefix of `kubernetes`, which results in each repository created using the pull through cache rule to have the naming scheme of `kubernetes/upstream-repository-name`.
```
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix kubernetes \
--upstream-registry-url registry.k8s.io \
--region us-east-1
```
##### For Quay
The following example creates a pull through cache rule for the Quay public registry. It specifies a repository prefix of `quay`, which results in each repository created using the pull through cache rule to have the naming scheme of `quay/upstream-repository-name`.
```
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix quay \
--upstream-registry-url quay.io \
--region us-east-1
```
##### For Docker Hub
The following example creates a pull through cache rule for the Docker Hub registry. It specifies a repository prefix of `docker-hub`, which results in each repository created using the pull through cache rule to have the naming scheme of `docker-hub/upstream-repository-name`. You must specify the full Amazon Resource Name (ARN) of the secret containing your Docker Hub credentials.
```
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix docker-hub \
--upstream-registry-url registry-1.docker.io \
--credential-arn arn:aws:secretsmanager:us-east-1:111122223333:secret:ecr-pullthroughcache/example1234 \
--region us-east-1
```
##### For GitHub Container Registry
The following example creates a pull through cache rule for the GitHub Container Registry. It specifies a repository prefix of `github`, which results in each repository created using the pull through cache rule to have the naming scheme of `github/upstream-repository-name`. You must specify the full Amazon Resource Name (ARN) of the secret containing your GitHub Container Registry credentials.
```
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix github \
--upstream-registry-url ghcr.io \
--credential-arn arn:aws:secretsmanager:us-east-1:111122223333:secret:ecr-pullthroughcache/example1234 \
--region us-east-1
```
##### For Microsoft Azure Container Registry
The following example creates a pull through cache rule for the Microsoft Azure Container Registry. It specifies a repository prefix of `azure`, which results in each repository created using the pull through cache rule to have the naming scheme of `azure/upstream-repository-name`. You must specify the full Amazon Resource Name (ARN) of the secret containing your Microsoft Azure Container Registry credentials.
```
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix azure \
--upstream-registry-url myregistry.azurecr.io \
--credential-arn arn:aws:secretsmanager:us-east-1:111122223333:secret:ecr-pullthroughcache/example1234 \
--region us-east-1
```
##### For GitLab Container Registry
The following example creates a pull through cache rule for the GitLab Container Registry. It specifies a repository prefix of `gitlab`, which results in each repository created using the pull through cache rule to have the naming scheme of `gitlab/upstream-repository-name`. You must specify the full Amazon Resource Name (ARN) of the secret containing your GitLab Container Registry credentials.
```
aws ecr create-pull-through-cache-rule \
--ecr-repository-prefix gitlab \
--upstream-registry-url registry.gitlab.com \
--credential-arn arn:aws:secretsmanager:us-east-1:111122223333:secret:ecr-pullthroughcache/example1234 \
--region us-east-1
```
For more information, see [ Create a pull through cache rule (CLI)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache-creating-rule.html#pull-through-cache-creating-rule-cli) in the *Amazon ECR User Guide*.
You can use the **get-run-task** CLI command to retrieve information about the container image used for a specific task:
```
aws omics get-run-task --id 1234567 --task-id
```
The output includes the following information about the container image:
```
"imageDetails": {
"image": "string",
"imageDigest": "string",
"sourceImage": "string",
...
}
```
#### Registry permissions for upstream registry
Use registry permissions to allow HealthOmics to use the pull through cache and to pull the container images into the Amazon ECR private registry. Add an Amazon ECR Registry policy to the registry that provides the containers used in runs.
The following policy grants permission for the HealthOmics service to create repositories with the specified pull through cache prefix(es) and to initiate upstream pulls into these repositories.
1. From the Amazon ECR console, open the left menu, under **Private registry**, expand **Registry permissions**. then choose **Generate statement**.
1. On the top right side, choose JSON. Enter a policy similar to the following:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "AllowPTCinRegPermissions",
"Effect": "Allow",
"Principal": {
"Service": "omics.amazonaws.com"
},
"Action": [
"ecr:CreateRepository",
"ecr:BatchImportUpstreamImage"
],
"Resource": [
"arn:aws:ecr:us-east-1:123456789012:repository/ecr-public/*",
"arn:aws:ecr:us-east-1:123456789012:repository/docker-hub/*"
]
}
]
}
```
------
#### Repository creation templates
To use pull through caching in HealthOmics, the Amazon ECR repository must have a repository creation template. The template defines configuration settings for when you or Amazon ECR create a private repository for an upstream registry.
Each template contains a repository namespace prefix, which Amazon ECR uses to match new repositories to a specific template. Templates specify the configuration for all repository settings including resource-based access policies, tag immutability, encryption, and lifecycle policies.
For more information, see [Repository creation templates](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-creation-templates.html) in the *Amazon Elastic Container Registry User Guide*.
How to create a repository creation template:
1. From the Amazon ECR console, open the left menu, under **Private registry**, expand **Features and settings**. then choose **Repository creation templates**.
1. Choose **Create template**.
1. In **Template details**, choose **Pull through cache**.
1. Choose whether to apply this template to a specific prefix or to all repositories that don't match another template.
If you choose **A specific prefix**, enter the namespace prefix value in **Prefix**. You specified this prefix when you created the PTC rule.
1. Choose **Next**.
1. In **Add repository creation configuration** page, enter **Repository permissions**. Use one of the sample policy statements, or enter one similar to the following example:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "PTCRepoCreationTemplate",
"Effect": "Allow",
"Principal": {
"Service": "omics.amazonaws.com"
},
"Action": [
"ecr:BatchGetImage",
"ecr:GetDownloadUrlForLayer"
],
"Resource": "*"
}
]
}
```
------
1. Optionally, you can add repository settings such as lifecycle policy and tags. Amazon ECR applies these rules for all container images created for pull through cache that use the specified prefix.
1. Choose **Next**.
1. Review the configuration and choose **Next**.
#### Creating the workflow
When you create a new workflow or workflow version, review the registry mappings and update them if required. For details, see [Create a private workflow](create-private-workflow.md).
### Registry mappings
You define registry mappings to map between prefixes in your private Amazon ECR registry and the upstream registry names.
For more information about Amazon ECR registry mappings, see [ Creating a pull through cache rule in Amazon ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache-creating-rule.html).
The following example shows registry mappings to Docker Hub, Quay, and Amazon ECR Public.
```
{
"registryMappings": [
{
"upstreamRegistryUrl": "registry-1.docker.io",
"ecrRepositoryPrefix": "docker-hub"
},
{
"upstreamRegistryUrl": "quay.io",
"ecrRepositoryPrefix": "quay"
},
{
"upstreamRegistryUrl": "public.ecr.aws",
"ecrRepositoryPrefix": "ecr-public"
}
]
}
```
### Image mappings
You define image mappings to map between the image names as defined in your private Amazon ECR workflows and the image names in the upstream registry.
You can use image mappings with registries that support pull through cache. You can also use image mappings with upstream registries where HealthOmics doesn't support pull through cache. You need to manually synchronize the upstream registry with your private repository.
For more information about Amazon ECR image mappings, see [ Creating a pull through cache rule in Amazon ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache-creating-rule.html).
The following example shows mappings from private Amazon ECR images to a public genomics image and the latest Ubuntu image.
```
{
"imageMappings": [
{
"sourceImage": "public.ecr.aws/aws-genomics/broadinstitute/gatk:4.6.0.2",
"destinationImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/broadinstitute/gatk:4.6.0.2"
},
{
"sourceImage": "ubuntu:latest",
"destinationImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/custom/ubuntu:latest",
}
]
}
```
## General considerations for Amazon ECR container images
+ Architecture
HealthOmics supports x86\$164 containers. If your local machine is ARM-based, such as Apple Mac, use a command such as the following to build an x86\$164 container image:
```
docker build --platform amd64 -t my_tool:latest .
```
+ Entrypoint and shell
HealthOmics workflow engines inject bash scripts as a command override to the container images used by workflow tasks. Thus, container images should be built without a specified ENTRYPOINT such that a bash shell is the default.
+ Mounted paths
A shared filesystem is mounted to container tasks at /tmp. Any data or tooling built into the container image at this location will be overridden.
The workflow definition is available to tasks via a read-only mount at /mnt/workflow.
+ Image size
See [HealthOmics workflow fixed size quotas](fixed-quotas.md#fixed-quotas-workflows) for the maximum container image sizes.
## Environment variables for HealthOmics workflows
HealthOmics provides environment variables that have information about the workflow running in the container. You can use the values of these variables in the logic of your workflow tasks.
All HealthOmics workflow variables start with the `AWS_WORKFLOW_` prefix. This prefix is a protected environment variable prefix. Don't use this prefix for your own variables in workflow containers.
HealthOmics provides the following workflow environment variables:
**AWS\$1REGION**
This variable is the region where the container is running.
**AWS\$1WORKFLOW\$1RUN**
This variable is the name of the current run.
**AWS\$1WORKFLOW\$1RUN\$1ID**
This variable is the run identifier of the current run.
**AWS\$1WORKFLOW\$1RUN\$1UUID**
This variable is the run UUID of the current run.
**AWS\$1WORKFLOW\$1TASK**
This variable is the name of the current task.
**AWS\$1WORKFLOW\$1TASK\$1ID**
This variable is the task identifier of the current task.
**AWS\$1WORKFLOW\$1TASK\$1UUID**
This variable is the task UUID of the current task.
The following example shows typical values for each environment variable:
```
AWS Region: us-east-1
Workflow Run: arn:aws:omics:us-east-1:123456789012:run/6470304
Workflow Run ID: 6470304
Workflow Run UUID: f4d9ed47-192e-760e-f3a8-13afedbd4937
Workflow Task: arn:aws:omics:us-east-1:123456789012:task/4192063
Workflow Task ID: 4192063
Workflow Task UUID: f0c9ed49-652c-4a38-7646-60ad835e0a2e
```
## Using Java in Amazon ECR container images
If a workflow task uses a Java application such as GATK, consider the following memory requirements for the container:
+ Java applications use stack memory and heap memory. By default, the maximum heap memory is a percentage of the total available memory in the container. This default depends on the specific JVM distribution and JVM version, so consult the relevant documentation for your JVM or explicitly set the heap memory maximum using Java command line options (such as `-Xmx`).
+ Don't set the maximum heap memory to be 100% of the container's memory allocation, because the JVM stack also requires memory. Memory is also required for the JVM garbage collector and any other operating system processes running in the container.
+ Some Java applications, such as GATK, can use native method invocations or other optimizations such as memory mapping files. These techniques require memory allocations that are performed “off heap”, which aren't controlled by the JVM maximum heap parameter.
If you know (or suspect) that your Java application allocates off-heap memory, make sure your task memory allocation includes the off-heap memory requirements.
If these off-heap allocations cause the container to run out of memory, you typically won't see a Java **OutOfMemory** error, because the JVM doesn't control this memory.
## Add task inputs to an Amazon ECR container image
Add all executables, libraries, and scripts needed to run a workflow task into the Amazon ECR image that's used to run the task.
It's best practice to avoid using scripts, binaries, and libraries that are external to a tasks container image. This is especially important when using `nf-core` workflows that use a `bin` directory as part of the workflow package. While this directory will be available to the workflow task, it's mounted as a read-only directory. Required resources in this directory should be copied into the task image and made available at runtime or when building the container image used for the task.
See [HealthOmics workflow fixed size quotas](fixed-quotas.md#fixed-quotas-workflows) for the maximum size of container image that HealthOmics supports.
# HealthOmics Workflow README files
You can upload a README.md file containing instructions, diagrams, and essential information for your workflow. Each workflow version supports one README file, which you can update at any time.
**README requirements include:**
+ README file must be in markdown (.md) format
+ Maximum file size: 500 KiB
**Topics**
+ [Use an existing README](#workflows-add-readme)
+ [Rendering conditions](#workflows-rendering-readme)
## Use an existing README
READMEs exported from Git repositories contain relative links that typically do not work outside the repository. HealthOmics Git integration automatically converts these to absolute links for proper rendering in the console, eliminating the need for manual URL updates.
For READMEs imported from Amazon S3 or local drives, images and links must either use public URLs or have their relative paths updated for proper rendering.
**Note**
Images must be publicly hosted to display in the HealthOmics console. Images stored in GitHub Enterprise Server or GitLab Self-Managed repositories cannot be rendered.
## Rendering conditions
The HealthOmics console interpolates publicly accessible images and links using absolute paths. To render URLs from private repositories, the user must have access to the repository. For GitHub Enterprise Server or GitLab Self-Managed repositories, which use custom domains, HealthOmics cannot resolve relative links or render images stored in these private repositories.
The following table shows the markdown elements that are supported by the AWS console README view.
| Element | AWS console |
| --- | --- |
| Alerts | Yes, but without icons |
| Badges | Yes |
| Basic text formatting | Yes |
| [Code blocks](https://www.markdownguide.org/basic-syntax/#code-blocks) | Yes, but does not have [syntax highlight](https://www.markdownguide.org/extended-syntax/#syntax-highlighting) and copy button functionality |
| Collapsible sections | Yes |
| [Headings](https://www.markdownguide.org/basic-syntax/#headings) | Yes |
| [Image formats](https://www.markdownguide.org/basic-syntax/#images-1) | Yes |
| [Image (clickable)](https://www.markdownguide.org/basic-syntax/#linking-images) | Yes |
| [Line breaks](https://www.markdownguide.org/basic-syntax/#line-breaks) | Yes |
| Mermaid diagram | Only can open graph, move graph position, and copy code |
| Quotes | Yes |
| [Subscript](https://www.markdownguide.org/extended-syntax/#subscript) and [superscript](https://www.markdownguide.org/extended-syntax/#superscript) | Yes |
| [Tables](https://www.markdownguide.org/extended-syntax/#tables) | Yes, but does not support text alignment |
| Text alignment | Yes |
### Using image and link URLs
Depending on your source provider, structure your base URLs for pages and images in the following formats.
+ `{username}`: The username where the repository is hosted.
+ `{repo}`: The repository name.
+ `{ref}`: The source reference (branch, tag, and commit ID).
+ `{path}`: The file path to the page or image in the repository.
| Source provider | Page URL | Image URL |
| --- | --- | --- |
| GitHub | https://github.com/\$1username\$1/\$1repo\$1/blob/\$1ref\$1/\$1path\$1 | `https://github.com/{username}/{repo}/blob/{ref}/{path}?raw=true` `https://raw.githubusercontent.com/{username}/{repo}/{ref}/{path}` |
| GitLab | https://gitlab.com/\$1username\$1/\$1repo\$1/-/blob/\$1ref\$1/\$1path\$1 | https://gitlab.com/\$1username\$1/\$1repo\$1/-/raw/\$1ref\$1/\$1path\$1 |
| Bitbucket | https://bitbucket.org/\$1username\$1/\$1repo\$1/src/\$1ref\$1/\$1path\$1 | https://bitbucket.org/\$1username\$1/\$1repo\$1/raw/\$1ref\$1/\$1path\$1 |
GitHub, GitLab, and Bitbucket support both page and image URLs that link to a public repository. The following table shows each source provider’s support for rendering image and link URLs for private repositories.
| Private repository support | Source provider | Page URL | Image URL |
| --- | --- | --- | --- |
| GitHub | Only with access to repository | No |
| GitLab | Only with access to repository | No |
| Bitbucket | Only with access to repository | No |
# Requesting Sentieon licenses for private workflows
If your private workflow uses Sentieon software, you need a Senieon license. Follow these steps to request and set up a license for the Sentieon software:
+ Request a Sentieon license
+ Send an email to the Sentieon support group (support@sentieon.com) to request a software license.
+ Provide your AWS Canonical User ID in the email.
+ Find your AWS Canonical User ID by following [ these instructions](https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-identifiers.html#FindCanonicalId).
+ Update your HealthOmics service role to grant it access to the Sentieon licensing server proxy and Sentieon Omics bucket in your Region. The following example grants access in `us-east-1`. If required, replace this text with your Region.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObjectAcl",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::omics-ap-us-east-1/*",
"arn:aws:s3:::sentieon-omics-license-us-east-1/*"
]
}
]
}
```
------
+ Generate an AWS support case to get access to the Sentieon license server proxy.
+ To create a support case, navigate to [ support.console.aws.amazon.com.](https://support.console.aws.amazon.com)
+ Provide your AWS account and Region in the support case. Your account is added to the allowlist for the licensing server proxy.
+ Build your private workflow using the Sentieon container and the Sentieon license script.
+ For additional instructions on using Sentieon tools inside private workflows, see [Sentieon-Amazon-Omics](https://github.com/Sentieon/sentieon-amazon-omics) in GitHub.
+ Sentieon software version 202112.07 and higher support the HealthOmics licensing server proxy. To use Sentieon software versions earlier than 202112.07, contact Sentieon support.
# Workflow linters in HealthOmics
After you create a workflow, we recommend that you run a linter on the workflow before you start the first run. The linter detects errors that can cause the run to fail.
For WDL, HealthOmics automatically runs a linter when you create the workflow. The linter output is available in the `statusMessage` field of the **get-workflow** response. Use the following CLI command to retrieve the status output (use the workflow ID of the WDL workflow that you created):
```
aws omics get-workflow
—id 123456
—query 'statusMessage'
```
HealthOmics provides linters that you can run on the workflow defnition before you create the workflow. Run these linters on existing pipelines that you're migrating to HealthOmics.
+ **WDL** – A public Amazon ECR image to run a [WDL linter](https://gallery.ecr.aws/aws-genomics/healthomics-linter).
+ **Nextflow** – A public Amazon ECR image to run [Linter rules for Nextflow]( https://gallery.ecr.aws/aws-genomics/linter-rules-for-nextflow). You can access the source code for this linter from [GitHub](https://github.com/awslabs/linter-rules-for-nextflow).
+ **CWL** – not available
# HealthOmics workflow operations
To create a private workflow, you need:
+ **Workflow definition file:** A workflow definition file written in WDL, Nextflow, or CWL. The workflow definition specifies the inputs and outputs for runs that use the workflow. It also includes specifications for the runs and run tasks for your workflow, including compute and memory requirements. The workflow definition file must be in `.zip` format. For more information, see [Workflow definition files](workflow-definition-files.md) in HealthOmics.
+ You can use [Kiro CLI](https://docs.aws.amazon.com/kiro/latest/userguide/what-is.html) to build and validate your workflow definition files in WDL, Nextflow, and CWL. For more information, see [Example prompts for Kiro CLI](getting-started.md#omics-kiro-prompts) and the [HealthOmics Agentic generative AI tutorial](https://github.com/aws-samples/aws-healthomics-tutorials/tree/main/generative-ai) on GitHub.
+ **(Optional) Parameter template file:** A parameter template file written in JSON. Create the file to define the run parameters, or HealthOmics generates the parameter template for you. For more information, see [Parameter template files for HealthOmics workflows](parameter-templates.md).
+ **Amazon ECR container images:** Create private Amazon ECR repositories for each container used in the workflow. Create container images for the workflow and store them in a private repository, or synchronize the contents of a supported upstream registry with your ECR private repository.
+ **(Optional) Sentieon licenses:** Request a Sentieon license to use the Sentieon software in private workflows.
For workflow definition files larger than 4 MiB (zipped), choose one of these options during workflow creation:
+ Upload to an Amazon Simple Storage Service folder and specify the location.
+ Upload to an external repository (max size 1 GiB) and specify the repository details.
After you create a workflow, you can update the following workflow information with the `UpdateWorkflow` operation:
+ Name
+ Description
+ Default storage type
+ Default storage capacity (with workflow ID)
+ README.md file
To change other information in the workflow, create a new workflow or workflow version.
Use workflow versioning to organize and structure you workflows. Versions also help you to manage the introduction of iterative workflow updates. For more information about versions, see [Create a workflow version](workflows-version-create.md).
**Topics**
+ [Create a private workflow](create-private-workflow.md)
+ [Update a private workflow](update-private-workflow.md)
+ [Delete a private workflow](delete-private-workflow.md)
+ [Verify the workflow status](using-get-workflow.md)
+ [Referencing genome files from a workflow definition](create-ref-files.md)
# Create a private workflow
Create a workflow using the HealthOmics console, AWS CLI commands, or one of the AWS SDKs.
**Note**
Don’t include any personally identifiable information (PII) in workflow names. These names are visible in CloudWatch logs.
When you create a workflow, HealthOmics assigns a universally unique identifier (UUID) to the workflow. The workflow UUID is a Globally Unique Identifier (guid) that's unique across workflows and workflow versions. For data provenance purposes, we recommend that you use the workflow UUID to uniquely identify workflows.
If your workflow tasks use any external tools (executables, libraries, or scripts), you build these tools into a container image. You have the following options for hosting the container image:
+ Host the container image in the ECR private registry. Prerequisites for this option:
+ Create an ECR private repository, or choose an existing repository.
+ Configure the ECR resource policy as described in [Amazon ECR permissions](permissions-ecr.md).
+ Upload your container image to the private repository.
+ Synchronize the container image with the contents of a supported third-party registry. Prerequisites for this option:
+ In the ECR private registry, configure a pull through cache rule for each upstream registry. For more information, see [Image mappings](workflows-ecr.md#ecr-pull-through-mapping-format).
+ Configure the ECR resource policy as described in [Amazon ECR permissions](permissions-ecr.md).
+ Create repository creation templates. The template defines settings for when Amazon ECR creates the private repository for an upstream registry.
+ Create prefix mappings to remap container image references in the workflow definition to ECR cache namespaces.
When you create a workflow, you provide a workflow definition that contains information about the workflow, runs, and tasks. HealthOmics can retrieve the workflow definition as a .zip archive stored locally or in an Amazon S3 bucket, or from a supported Git-based repository.
**Topics**
+ [Creating a workflow using the console](#console-create-workflows)
+ [Creating a workflow using the CLI](#api-create-workflows)
+ [Creating a workflow using an SDK](#sdk-create-workflows)
## Creating a workflow using the console
**Steps to create a workflow**
1. Open the [HealthOmics console](https://console.aws.amazon.com/omics/).
1. If required, open the left navigation pane (≡). Choose **Private workflows**.
1. On the **Private workflows** page, choose **Create workflow**.
1. On the **Define workflow** page, provide the following information:
1. **Workflow name**: A distinctive name for this workflow. We recommend setting workflow names to organize your runs in the AWS HealthOmics console and CloudWatch logs.
1. **Description** (optional): A description of this workflow.
1. In the **Workflow definition** panel, provide the following information:
1. **Workflow language** (optional): Select the specification language of the workflow. Otherwise, HealthOmics determines the language from the workflow definition.
1. For **Workflow definition source**, choose to import the definition folder from a Git-based repository, an Amazon S3 location, or from a local drive.
1. For **Import from a repository service**:
**Note**
HealthOmics supports public and private repositories for GitHub, GitLab, Bitbucket, GitHub self-managed, GitLab self-managed.
1. Choose a **Connection** to connect your AWS resources to the external repository. To create a connection, see [Connect with external code repositories](setting-up-new.md#setting-up-omics-repository).
**Note**
Customers in the TLV region need to create a connection in the IAD (us-east-1) region to create a workflow.
1. In **Full repository ID**, enter your repository ID as user-name/repo-name. Verify you have access to the files in this repository.
1. In **Source reference** (optional), enter a repository source reference (branch, tag, or commit ID). HealthOmics uses the default branch if no source reference is specified.
1. In **Exclude file patterns**, enter the file patterns to exclude specific folders, files, or extensions. This helps manage data size when importing repository files. There is a max of 50 patterns, and the patters must follow the [glob pattern syntax](https://fossil-scm.org/home/doc/tip/www/globs.md). For example:
1. `tests/`
1. `*.jpeg`
1. `large_data.zip`
1. For **Select definition folder from S3**:
1. Enter the Amazon S3 location that contains the zipped workflow definition folder. The Amazon S3 bucket must be in the same region as the workflow.
1. If your account doesn't own the Amazon S3 bucket, enter the bucket owner's AWS account ID in the **S3 bucket owner's account ID**. This information is required so that HealthOmics can verify the bucket ownership.
1. For **Select definition folder from a local source**:
1. Enter the local drive location of the zipped workflow definition folder.
1. **Main workflow definition file path** (optional): Enter the file path from the zipped workflow definition folder or repository to the `main` file. This parameter is not required if there is only one file in the workflow definition folder, or if the main file is named "main".
1. In the **README file** (optional) panel, select the **Source of the README file** and provide the following information:
+ For **Import from a repository service**, in **README file path**, enter the path to the README file within the repository.
+ For **Select file from S3**, in **README file in S3**, enter the Amazon S3 URI for the README file.
+ For **Select file from a local source**: in **README - optional**, chose **Choose file** to select the markdown (.md) file to upload.
1. In the **Default run storage configuration** panel, provide the default run storage type and capacity for runs that use this workflow:
1. **Run storage type**: Choose whether to use static or dynamic storage as the default for the temporary run storage. The default is static storage.
1. **Run storage capacity** (optional): For static run storage type, you can enter the default amount of run storage required for this workflow. The default value for this parameter is 1200 GiB. You can override these default values when you start a run.
1. **Tags** (optional): You can associate up to 50 tags with this workflow.
1. Choose **Next**.
1. On the **Add workflow parameters** (optional) page, select the **Parameter source**:
1. For **Parse from workflow definition file**, HealthOmics will automatically parse the workflow parameters from the workflow definition file.
1. For **Provide parameter template from Git repository**, use the path to the parameter template file from your repository.
1. For **Select JSON file from local source**, upload a JSON file from a local source that specifies the parameters.
1. For **Manually enter workflow parameters**, manually enter parameter names and descriptions.
1. In the **Parameter preview** panel, you can review or change the parameters for this workflow version. If you restore the JSON file, you lose any local changes that you made.
1. Choose **Next**.
1. On the **Container URI remapping** page, in the **Mapping rules** panel, you can define URI mapping rules for your workflow.
For **Source of mapping file**, select one of the following options:
+ **None** – No mapping rules required.
+ **Select JSON file from S3** – Specify the S3 location for the mapping file.
+ **Select JSON file from a local source** – Specify the mapping file location on your local device.
+ **Manually enter mappings** – enter the registry mappings and image mappings in the **Mappings** panel.
1. The console displays the **Mappings** panel. If you chose a mapping source file, the console displays the values from the file.
1. In **Registry mappings**, you can edit the mappings or add mappings (maximum of 20 registry mappings).
Each registry mapping contains the following fields:
+ **Upstream registry URL** – The URI of the upstream registry.
+ **ECR repository prefix** – The repository prefix to use in the Amazon ECR private repository.
+ (Optional) **Upstream repository prefix** – The prefix of the repository in the upstream registry.
+ (Optional) **ECR account ID** – Account ID of the account that owns the upstream container image.
1. In **Image mappings**, you can edit the image mappings or add mappings (maximum of 100 image mappings).
Each image mapping contains the following fields:
+ **Source image** – Specifies the URI of the source image in the upstream registry.
+ **Destination image** – Specifies the URI of the corresponding image in the private Amazon ECR registry.
1. Choose **Next**.
1. Review the workflow configuration, then choose **Create workflow**.
## Creating a workflow using the CLI
If your workflow files and the parameter template file are on your local machine, you can create a workflow using the following CLI command.
```
aws omics create-workflow \
--name "my_workflow" \
--definition-zip fileb://my-definition.zip \
--parameter-template file://my-parameter-template.json
```
The `create-workflow` operation returns the following response:
```
{
"arn": "arn:aws:omics:us-west-2:....",
"id": "1234567",
"status": "CREATING",
"tags": {
"resourceArn": "arn:aws:omics:us-west-2:...."
},
"uuid": "64c9a39e-8302-cc45-0262-2ea7116d854f"
}
```
### Optional parameters to use when creating a workflow
You can specify any of the optional parameters when you create a workflow. For syntax details, see [CreateWorkflow](https://docs.aws.amazon.com/omics/latest/api/API_CreateWorkflow.html) in the AWS HealthOmics API Reference.
**Topics**
+ [Specify the workflow definition Amazon S3 location](#create-defn-uri-parameter)
+ [Use the workflow definition from a Git-based repository](#create-defn-uri-git)
+ [Specify a Readme file](#specify-readme-file)
+ [Specify the **main** definition file](#create-main-parameter)
+ [Specify the run storage type](#create-run-storage-parameter)
+ [Specify the GPU configuration](#create-accelerator-parameter)
+ [Configure pull through cache mapping parameters](#create-prefix-mapping-parameters)
#### Specify the workflow definition Amazon S3 location
If your workflow definition file is located in an Amazon S3 folder, specify the location using the `definition-uri` parameter, as shown in the following example. If your account does not own the Amazon S3 bucket, provide the owner's AWS account ID.
```
aws omics create-workflow \
--name Test \
--definition-uri s3://omics-bucket/workflow-definition/ \
--owner-id 123456789012
...
```
#### Use the workflow definition from a Git-based repository
To use the workflow definition from a supported Git-based repository, use the `definition-repository` parameter in your request. Don’t provide any other `definition` parameter, as a request fails if it includes more than one input source.
The `definition-respository` parameter contains the following fields:
+ **connectionArn** – ARN of the Code Connection that connects your AWS resources to the external repository.
+ **fullRepositoryId** – Enter the repository ID as `owner-name/repo-name`. Verify you have access to the files in this repository.
+ **sourceReference** (Optional) – Enter a repository reference type (BRANCH, TAG, or COMMIT) and a value.
HealthOmics uses the latest commit on the default branch if you don't specify a source reference.
+ **excludeFilePatterns** (Optional) – Enter the file patterns to exclude specific folders, files, or extensions. This helps manage data size when importing repository files. Provide a maximum of 50 patterns. the patterns must follow the [ glob pattern syntax](https://fossil-scm.org/home/doc/tip/www/globs.md). For example:
+ `tests/`
+ `*.jpeg`
+ `large_data.zip`
When you specify the workflow definition from a Git-based repository, use `parameter-template-path` to specify the parameter template file. If you don’t provide this parameter, HealthOmics creates the workflow without a parameter template.
The following example shows the parameters related to content from a Git-based private repository:
```
aws omics create-workflow \
--name custom-variant \
--description "Custom variant calling pipeline" \
--engine "WDL" \
--definition-repository '{
"connectionArn": "arn:aws:codeconnections:us-east-1:123456789012:connection/abcd1234-5678-90ab-cdef-1234567890ab",
"fullRepositoryId": "myorg/my-genomics-workflows",
"sourceReference": {
"type": "BRANCH",
"value": "main"
},
"excludeFilePatterns": ["tests/**", "*.log"]
}' \
--main "workflows/variant-calling/main.wdl" \
--parameter-template-path "parameters/variant-calling-params.json" \
--readme-path "docs/variant-calling-README.md" \
--storage-type "DYNAMIC" \
```
For more examples, see the blog post [ How To Create an AWS HealthOmics Workflows from Content in Git](https://repost.aws/articles/ARCEN7AjhaRSmteczRoc_QsA/how-to-create-an-aws-healthomics-workflows-from-content-in-git).
#### Specify a Readme file
You can specify the README file location using one of the following parameters:
+ **readme-markdown** – String input or a file on your local machine.
+ **readme-uri** – The URI of a file stored on S3.
+ **readme-path ** – The path to the README file in the repository.
Use readme-path only in conjunction with **definition-respository**. If you don’t specify any README parameter, HealthOmics imports the root level README.md file in the repository (if it exists).
The following examples show how specify the README file location using readme-path and readme-uri.
```
# Using README from repository
aws omics create-workflow \
--name "documented-workflow" \
--definition-repository '...' \
--readme-path "docs/workflow-guide.md"
# Using README from S3
aws omics create-workflow \
--name "s3-readme-workflow" \
--definition-repository '...' \
--readme-uri "s3://my-bucket/workflow-docs/readme.md"
```
For more information, see [HealthOmics Workflow README files](workflows-readme.md).
#### Specify the **main** definition file
If you are including multiple workflow definition files, use the `main` parameter to specify the main definition file for your workflow.
```
aws omics create-workflow \
--name Test \
--main multi_workflow/workflow2.wdl \
...
```
#### Specify the run storage type
You can specify the default run storage type (DYNAMIC or STATIC) and run storage capacity (required for static storage). For more information about run storage types, see [Run storage types in HealthOmics workflows](workflows-run-types.md).
```
aws omics create-workflow \
--name my_workflow \
--definition-zip fileb://my-definition.zip \
--parameter-template file://my-parameter-template.json \
--storage-type 'STATIC' \
--storage-capacity 1200 \
```
#### Specify the GPU configuration
Use the accelerators parameter to create a workflow that runs on an accelerated-compute instance. The following example shows how to use the `accelerators` parameter. You specify the GPU configuration in the workflow definition. See [Accelerated-computing instances](memory-and-compute-tasks.md#workflow-task-accelerated-computing-instances).
```
aws omics create-workflow --name workflow name \
--definition-uri s3://amzn-s3-demo-bucket1/GPUWorkflow.zip \
--accelerators GPU
```
#### Configure pull through cache mapping parameters
If you're using the Amazon ECR pull through cache mapping feature, you can override the default mappings. For more information about the container setup parameters, see [Container images for private workflows](workflows-ecr.md).
In the following example, file `mappings.json` contains this content:
```
{
"registryMappings": [
{
"upstreamRegistryUrl": "registry-1.docker.io",
"ecrRepositoryPrefix": "docker-hub"
},
{
"upstreamRegistryUrl": "quay.io",
"ecrRepositoryPrefix": "quay",
"accountId": "123412341234"
},
{
"upstreamRegistryUrl": "public.ecr.aws",
"ecrRepositoryPrefix": "ecr-public"
}
],
"imageMappings": [{
"sourceImage": "docker.io/library/ubuntu:latest",
"destinationImage": "healthomics-docker-2/custom/ubuntu:latest",
"accountId": "123412341234"
},
{
"sourceImage": "nvcr.io/nvidia/k8s/dcgm-exporter",
"destinationImage": "healthomics-nvidia/k8s/dcgm-exporter"
}
]
}
```
Specify the mapping parameters in the create-workflow command:
```
aws omics create-workflow \
...
--container-registry-map-file file://mappings.json
...
```
You can also specify the S3 location of the mapping parameters file:
```
aws omics create-workflow \
...
--container-registry-map-uri s3://amzn-s3-demo-bucket1/test.zip
...
```
## Creating a workflow using an SDK
You can create a workflow using one of the SDKs. The following example shows how to create a workflow using the Python SDK
```
import boto3
omics = boto3.client('omics')
with open('definition.zip', 'rb') as f:
definition = f.read()
response = omics.create_workflow(
name='my_workflow',
definitionZip=definition,
parameterTemplate={ ... }
)
```
# Update a private workflow
You can update a workflow using the HealthOmics console, AWS CLI commands, or one of the AWS SDKs.
**Note**
Don’t include any personally identifiable information (PII) in workflow names. These names are visible in CloudWatch logs.
**Topics**
+ [Updating a workflow using the console](#console-update-workflows)
+ [Updating a workflow using the CLI](#api-update-workflows)
+ [Updating a workflow using an SDK](#sdk-update-workflows)
## Updating a workflow using the console
**Steps to update a workflow**
1. Open the [HealthOmics console](https://console.aws.amazon.com/omics/).
1. If required, open the left navigation pane (≡). Choose **Private workflows**.
1. On the **Private workflows** page, choose the workflow to update.
1. On the **Workflow** page:
+ If the workflow has versions, make sure that you select the **Default version**.
+ Choose **Edit selected** from the **Actions** list.
1. On the **Edit workflow** page, you can change any of the following values:
+ **Workflow name**.
+ **Workflow description**.
+ The default **Run storage type** for the workflow.
+ The default **Run storage capacity** (if the run storage type is static storage). For more information about the default run storage configuration, see [Creating a workflow using the console](create-private-workflow.md#console-create-workflows).
1. Choose **Save changes** to apply the changes.
## Updating a workflow using the CLI
As shown in the following example, you can update the workflow name and description. You can also change the default run storage type (STATIC or DYNAMIC) and run storage capacity (for static storage type). For more information about run storage types, see [Run storage types in HealthOmics workflows](workflows-run-types.md).
```
aws omics update-workflow \
--id 1234567 \
--name my_workflow \
--description "updated workflow" \
--storage-type 'STATIC' \
--storage-capacity 1200
```
You don't receive a response to the `update-workflow` request.
## Updating a workflow using an SDK
You can update a workflow using one of the SDKs.
The following example shows how to update a workflow using the Python SDK
```
import boto3
omics = boto3.client('omics')
response = omics.update_workflow(
name='my_workflow',
description='updated workflow'
)
```
# Delete a private workflow
When you no longer need a workflow, you can delete it using the HealthOmics console, AWS CLI commands, or one of the AWS SDKs. You can delete a workflow that meets the following criteria:
+ Its status is ACTIVE or FAILED.
+ It has no active shares.
+ You've deleted all the workflow versions.
Deleting a workflow doesn't affect any ongoing runs that are using the workflow.
**Topics**
+ [Deleting a workflow using the console](#console-delete-workflows)
+ [Deleting a workflow using the CLI](#api-delete-workflows)
+ [Deleting a workflow using an SDK](#sdk-delete-workflows)
## Deleting a workflow using the console
**To delete a workflow**
1. Open the [HealthOmics console](https://console.aws.amazon.com/omics/).
1. If required, open the left navigation pane (≡). Choose **Private workflows**.
1. On the **Private workflows** page, choose the workflow to delete.
1. On the **Workflow** page, choose **Delete selected** from the **Actions** list.
1. In the **Delete workflow** modal, enter "confirm" to confirm the deletion.
1. Choose **Delete**.
## Deleting a workflow using the CLI
The following example shows how you can use the AWS CLI command to delete a workflow. To run the example, replace the `workflow id` with the ID of the workflow you want to delete.
```
aws omics delete-workflow
--id workflow id
```
HealthOmics doesn't send a response to the `delete-workflow` request.
## Deleting a workflow using an SDK
You can delete a workflow using one of the SDKs.
The following example shows how to delete a workflow using the Python SDK.
```
import boto3
omics = boto3.client('omics')
response = omics.delete_workflow(
id='1234567'
)
```
# Verify the workflow status
After you create your workflow, you can verify the status and view other details of the workflow using **get-workflow**, as shown.
```
aws omics get-workflow --id 1234567
```
The response includes workflow details, including the status, as shown.
```
{
"arn": "arn:aws:omics:us-west-2:....",
"creationTime": "2022-07-06T00:27:05.542459"
"id": "1234567",
"engine": "WDL",
"status": "ACTIVE",
"type": "PRIVATE",
"main": "workflow-crambam.wdl",
"name": "workflow_name",
"storageType": "STATIC",
"storageCapacity": "1200",
"uuid": "64c9a39e-8302-cc45-0262-2ea7116d854f"
}
```
You can start a run using this workflow after the status transitions to `ACTIVE`.
# Referencing genome files from a workflow definition
An HealthOmics reference store object can be referred to with a URI like the following. Use your own `account ID`, `reference store ID`, and `reference ID` where indicated.
```
omics://account ID.storage.us-west-2.amazonaws.com/reference store id/reference/id
```
Some workflows will require both the `SOURCE` and `INDEX` files for the reference genome. The previous URI is the default short form and will default to the SOURCE file. In order to specify either file, you can use the long URI form, as follows.
```
omics://account ID.storage.us-west-2.amazonaws.com/reference store id/reference/id/source
omics://account ID.storage.us-west-2.amazonaws.com/reference store id/reference/id/index
```
Using a sequence read set would have a similar pattern, as shown.
```
aws omics create-workflow \
--name workflow name \
--main sample workflow.wdl \
--definition-uri omics://account ID.storage.us-west-2.amazonaws.com/sequence_store_id/readSet/id \
--parameter-template file://parameters_sample_description.json
```
Some read sets, such as those based on FASTQ, can contain paired reads. In the following examples, they're referred to as SOURCE1 and SOURCE2. Formats such as BAM and CRAM will only have a SOURCE1 file. Some read sets will contain INDEX files such as `bai` or `crai` files. The preceding URI is the default short form and will default to the SOURCE1 file. To specify the exact file or index, you can use the long URI form, as follows.
```
omics://123456789012.storage.us-west-2.amazonaws.com//readSet//source1
omics://123456789012.storage.us-west-2.amazonaws.com//readSet//source2
omics://123456789012.storage.us-west-2.amazonaws.com//readSet//index
```
The following is an example of an input JSON file that uses two Omics Storage URIs.
```
{
"input_fasta": "omics://123456789012.storage.us-west-2.amazonaws.com//reference/",
"input_cram": "omics://123456789012.storage.us-west-2.amazonaws.com//readSet/"
}
```
Reference the input JSON file in the AWS CLI by adding `--inputs file://` to your **start-run** request.