Use looping constructs in AWSTOE

This section provides information to help you create looping constructs in the AWSTOE. Looping constructs define a repeated sequence of instructions. You can use the following types of looping constructs in AWSTOE:

Reference iteration variables

To refer to the index and value of the current iteration variable, the reference expression {{ loop.* }} must be used within the input body of a step that contains a looping construct. This expression cannot be used to refer to the iteration variables of the looping construct of another step.

The reference expression consists of the following members:

Loop names

All looping constructs have an optional name field for identification. If a loop name is provided, it can be used to refer to iteration variables in the input body of the step. To refer to the iteration indices and values of a named loop, use {{ <loop_name>.* }} with {{ loop.* }} in the input body of the step. This expression cannot be used to refer to the named looping construct of another step.

The reference expression consists of the following members:

Resolve reference expressions

The AWSTOE resolves reference expressions as follows:

If reference expressions are used within a step that does not contain a loop, then AWSTOE does not resolve the expressions and they appear in the step with no replacement.

Types of looping constructs

This section provides information and examples about looping construct types that can be used in the AWSTOE.

for loop

The for loop iterates on a range of integers specified within a boundary outlined by the start and end of the variables. The iterating values are in the set [start, end] and includes boundary values.

AWSTOE verifies the start, end, and updateBy values to ensure that the combination does not result in an infinite loop.

for loop schema

  - name: "StepName"
    action: "ActionModule"
    loop:
      name: "string"
      for:
        start: int
        end: int
        updateBy: int
inputs:
  ...

for loop input example

  - name: "CalculateFileUploadLatencies"
    action: "ExecutePowerShell"
    loop:
      for:
        start: 100000
        end: 1000000
        updateBy: 100000
    inputs:
      commands:
        - |
          $f = new-object System.IO.FileStream c:\temp\test{{ loop.index }}.txt, Create, ReadWrite
          $f.SetLength({{ loop.value }}MB)
          $f.Close()
        - c:\users\administrator\downloads\latencyTest.exe --file c:\temp\test{{ loop.index }}.txt
        - AWS s3 cp c:\users\administrator\downloads\latencyMetrics.json s3://bucket/latencyMetrics.json
        - |
          Remove-Item -Path c:\temp\test{{ loop.index }}.txt
          Remove-Item -Path c:\users\administrator\downloads\latencyMetrics.json

forEach loop with input list

The forEach loop iterates on an explicit list of values, which can be strings and chained expressions.

forEach loop with input list schema

  - name: "StepName"
    action: "ActionModule"
    loop:
      name: "string"
      forEach:
        - "string"
    inputs:
  ...

forEach loop with input list example 1

  - name: "ExecuteCustomScripts"
    action: "ExecuteBash"
    loop:
      name: BatchExecLoop
      forEach:
        - /tmp/script1.sh
        - /tmp/script2.sh
        - /tmp/script3.sh
    inputs:
      commands:
        - echo "Count {{ BatchExecLoop.index }}"
        - sh "{{ loop.value }}"
        - |
          retVal=$?
          if [ $retVal -ne 0 ]; then
            echo "Failed"
          else
            echo "Passed"
         fi

forEach loop with input list example 2

  - name: "RunMSIWithDifferentArgs"
    action: "ExecuteBinary"
    loop:
      name: MultiArgLoop
      forEach:
        - "ARG1=C:\Users ARG2=1"
        - "ARG1=C:\Users"
        - "ARG1=C:\Users ARG3=C:\Users\Administrator\Documents\f1.txt"
    inputs:
      commands:
        path: "c:\users\administrator\downloads\runner.exe"
        args:
          - "{{ MultiArgLoop.value }}"

forEach loop with input list example 3

  - name: "DownloadAllBinaries"
    action: "S3Download"
    loop:
      name: MultiArgLoop
      forEach:
        - "bin1.exe"
        - "bin10.exe"
        - "bin5.exe"
    inputs:
      - source: "s3://bucket/{{ loop.value }}"
        destination: "c:\temp\{{ loop.value }}"

forEach loop with delimited list

The loop iterates over a string containing values separated by a delimiter. To iterate over the string’s constituents, AWSTOE uses the delimiter to split the string into an array suitable for iteration.

forEach loop with delimited list schema

  - name: "StepName"
    action: "ActionModule"
    loop:
      name: "string"
      forEach:
        list: "string"
        delimiter: ".,;:\n\t -_"
    inputs:
  ...

forEach loop with delimited list example 1

This example uses the following chaining expression pattern to refer to another step's output: <phase_name>.<step_name>.[inputs | outputs].<var_name>.

  - name: "RunMSIs"
    action: "ExecuteBinary"
    loop:
      forEach:
        list: "{{ build.GetAllMSIPathsForInstallation.outputs.stdout }}"
        delimiter: "\n"
    inputs:
      commands:
        path: "{{ loop.value }}"

forEach loop with delimited list example 2

  - name: "UploadMetricFiles"
    action: "S3Upload"
    loop:
      forEach:
        list: "/tmp/m1.txt,/tmp/m2.txt,/tmp/m3.txt,..."
    inputs:
      commands:
        - source: "{{ loop.value }}"
          destination: "s3://bucket/key/{{ loop.value }}"

Step fields

Loops are part of a step. Any field related to the running of a step is not applied to individual iterations. Step fields apply only at the step level, as follows:

Step and iteration outputs

Every iteration contains an output. At the end of a loop run, AWSTOE consolidates all successful iteration outputs in detailedOutput.json. The consolidated outputs are a collation of values that belong to the corresponding output keys as defined in the output schema of the action module. The following example shows how the outputs are consolidated:

Output of ExecuteBash for Iteration 1

{
	"stdout":"Hello"
}

Output of ExecuteBash for Iteration 2

{
	"stdout":"World"
}

Output of ExecuteBash for Step

{
	"stdout":"Hello\nWorld"
}

For example, ExecuteBash, ExecutePowerShell, and ExecuteBinary are action modules which return STDOUT as the action module output. STDOUT messages are joined with the new line character to produce the overall output of the step in detailedOutput.json.

AWSTOE will not consolidate the outputs of unsuccessful iterations.