You can monitor a training job using TrainingJobStatus and
SecondaryStatus returned by DescribeTrainingJob.
The list below shows how TrainingJobStatus and SecondaryStatus values
change depending on the training scenario:
-
Spot instances acquired with no interruption during training
-
InProgress:Starting↠Downloading↠Training↠Uploading
-
-
Spot instances interrupted once. Later, enough spot instances were acquired to finish the training job.
-
InProgress:Starting↠Downloading↠Training↠Interrupted↠Starting↠Downloading↠Training↠Uploading
-
-
Spot instances interrupted twice and
MaxWaitTimeInSecondsexceeded.-
InProgress:Starting↠Downloading↠Training↠Interrupted↠Starting↠Downloading↠Training↠Interrupted↠Downloading↠Training -
Stopping:Stopping -
Stopped:MaxWaitTimeExceeded
-
-
Spot instances were never launched.
-
InProgress:Starting -
Stopping:Stopping -
Stopped:MaxWaitTimeExceeded
-
