# Object Detection - TensorFlow
<a name="object-detection-tensorflow"></a>

The Amazon SageMaker AI Object Detection - TensorFlow algorithm is a supervised learning algorithm that supports transfer learning with many pretrained models from the [TensorFlow Model Garden](https://github.com/tensorflow/models). Use transfer learning to fine-tune one of the available pretrained models on your own dataset, even if a large amount of image data is not available. The object detection algorithm takes an image as input and outputs a list of bounding boxes. Training datasets must consist of images in .`jpg`, `.jpeg`, or `.png` format. This page includes information about Amazon EC2 instance recommendations and sample notebooks for Object Detection - TensorFlow.

**Topics**
+ [How to use the SageMaker AI Object Detection - TensorFlow algorithm](object-detection-tensorflow-how-to-use.md)
+ [Input and output interface for the Object Detection - TensorFlow algorithm](object-detection-tensorflow-inputoutput.md)
+ [Amazon EC2 instance recommendation for the Object Detection - TensorFlow algorithm](#object-detection-tensorflow-instances)
+ [Object Detection - TensorFlow sample notebooks](#object-detection-tensorflow-sample-notebooks)
+ [How Object Detection - TensorFlow Works](object-detection-tensorflow-HowItWorks.md)
+ [TensorFlow Models](object-detection-tensorflow-Models.md)
+ [Object Detection - TensorFlow Hyperparameters](object-detection-tensorflow-Hyperparameter.md)
+ [Tune an Object Detection - TensorFlow model](object-detection-tensorflow-tuning.md)

# How to use the SageMaker AI Object Detection - TensorFlow algorithm
<a name="object-detection-tensorflow-how-to-use"></a>

You can use Object Detection - TensorFlow as an Amazon SageMaker AI built-in algorithm. The following section describes how to use Object Detection - TensorFlow with the SageMaker AI Python SDK. For information on how to use Object Detection - TensorFlow from the Amazon SageMaker Studio Classic UI, see [SageMaker JumpStart pretrained models](studio-jumpstart.md).

The Object Detection - TensorFlow algorithm supports transfer learning using any of the compatible pretrained TensorFlow models. For a list of all available pretrained models, see [TensorFlow Models](object-detection-tensorflow-Models.md). Every pretrained model has a unique `model_id`. The following example uses ResNet50 (`model_id`: `tensorflow-od1-ssd-resnet50-v1-fpn-640x640-coco17-tpu-8`) to fine-tune on a custom dataset. The pretrained models are all pre-downloaded from the TensorFlow Hub and stored in Amazon S3 buckets so that training jobs can run in network isolation. Use these pre-generated model training artifacts to construct a SageMaker AI Estimator.

First, retrieve the Docker image URI, training script URI, and pretrained model URI. Then, change the hyperparameters as you see fit. You can see a Python dictionary of all available hyperparameters and their default values with `hyperparameters.retrieve_default`. For more information, see [Object Detection - TensorFlow Hyperparameters](object-detection-tensorflow-Hyperparameter.md). Use these values to construct a SageMaker AI Estimator.

**Note**  
Default hyperparameter values are different for different models. For example, for larger models, the default number of epochs is smaller. 

This example uses the [https://www.cis.upenn.edu/~jshi/ped_html/#pub1](https://www.cis.upenn.edu/~jshi/ped_html/#pub1) dataset, which contains images of pedestriants in the street. We pre-downloaded the dataset and made it available with Amazon S3. To fine-tune your model, call `.fit` using the Amazon S3 location of your training dataset.

```
from sagemaker import image_uris, model_uris, script_uris, hyperparameters
from sagemaker.estimator import Estimator

model_id, model_version = "tensorflow-od1-ssd-resnet50-v1-fpn-640x640-coco17-tpu-8", "*"
training_instance_type = "ml.p3.2xlarge"

# Retrieve the Docker image
train_image_uri = image_uris.retrieve(model_id=model_id,model_version=model_version,image_scope="training",instance_type=training_instance_type,region=None,framework=None)

# Retrieve the training script
train_source_uri = script_uris.retrieve(model_id=model_id, model_version=model_version, script_scope="training")

# Retrieve the pretrained model tarball for transfer learning
train_model_uri = model_uris.retrieve(model_id=model_id, model_version=model_version, model_scope="training")

# Retrieve the default hyperparameters for fine-tuning the model
hyperparameters = hyperparameters.retrieve_default(model_id=model_id, model_version=model_version)

# [Optional] Override default hyperparameters with custom values
hyperparameters["epochs"] = "5"

# Sample training data is available in this bucket
training_data_bucket = f"jumpstart-cache-prod-{aws_region}"
training_data_prefix = "training-datasets/PennFudanPed_COCO_format/"

training_dataset_s3_path = f"s3://{training_data_bucket}/{training_data_prefix}"

output_bucket = sess.default_bucket()
output_prefix = "jumpstart-example-od-training"
s3_output_location = f"s3://{output_bucket}/{output_prefix}/output"

# Create an Estimator instance
tf_od_estimator = Estimator(
    role=aws_role,
    image_uri=train_image_uri,
    source_dir=train_source_uri,
    model_uri=train_model_uri,
    entry_point="transfer_learning.py",
    instance_count=1,
    instance_type=training_instance_type,
    max_run=360000,
    hyperparameters=hyperparameters,
    output_path=s3_output_location,
)

# Launch a training job
tf_od_estimator.fit({"training": training_dataset_s3_path}, logs=True)
```

For more information about how to use the SageMaker AI Object Detection - TensorFlow algorithm for transfer learning on a custom dataset, see the [Introduction to SageMaker TensorFlow - Object Detection](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/object_detection_tensorflow/Amazon_Tensorflow_Object_Detection.ipynb) notebook.

# Input and output interface for the Object Detection - TensorFlow algorithm
<a name="object-detection-tensorflow-inputoutput"></a>

Each of the pretrained models listed in TensorFlow Models can be fine-tuned to any dataset with any number of image classes. Be mindful of how to format your training data for input to the Object Detection - TensorFlow model.
+ **Training data input format:** Your training data should be a directory with an `images` subdirectory and an `annotations.json` file. 

The following is an example of an input directory structure. The input directory should be hosted in an Amazon S3 bucket with a path similar to the following: `s3://bucket_name/input_directory/`. Note that the trailing `/` is required.

```
input_directory
    |--images
        |--abc.png
        |--def.png
    |--annotations.json
```

The `annotations.json` file should contain information for bounding boxes and their class labels in the form of a dictionary `"images"` and `"annotations"` keys. The value for the `"images"` key should be a list of dictionaries. There should be one dictionary for each image with the following information: `{"file_name": image_name, "height": height, "width": width, "id": image_id}`. The value for the `"annotations"` key should also be a list of dictionaries. There should be one dictionary for each bounding box with the following information: `{"image_id": image_id, "bbox": [xmin, ymin, xmax, ymax], "category_id": bbox_label}`.

After training, a label mapping file and trained model are saved to your Amazon S3 bucket.

## Incremental training
<a name="object-detection-tensorflow-incremental-training"></a>

You can seed the training of a new model with artifacts from a model that you trained previously with SageMaker AI. Incremental training saves training time when you want to train a new model with the same or similar data.

**Note**  
You can only seed a SageMaker AI Object Detection - TensorFlow model with another Object Detection - TensorFlow model trained in SageMaker AI. 

You can use any dataset for incremental training, as long as the set of classes remains the same. The incremental training step is similar to the fine-tuning step, but instead of starting with a pretrained model, you start with an existing fine-tuned model. For more information about how to use incremental training with the SageMaker AI Object Detection - TensorFlow, see the [Introduction to SageMaker TensorFlow - Object Detection](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/object_detection_tensorflow/Amazon_Tensorflow_Object_Detection.ipynb) notebook.

## Inference with the Object Detection - TensorFlow algorithm
<a name="object-detection-tensorflow-inference"></a>

You can host the fine-tuned model that results from your TensorFlow Object Detection training for inference. Any input image for inference must be in `.jpg`, .`jpeg`, or `.png` format and be content type `application/x-image`. The Object Detection - TensorFlow algorithm resizes input images automatically. 

Running inference results in bounding boxes, predicted classes, and the scores of each prediction encoded in JSON format. The Object Detection - TensorFlow model processes a single image per request and outputs only one line. The following is an example of a JSON format response:

```
accept: application/json;verbose

{"normalized_boxes":[[xmin1, xmax1, ymin1, ymax1],....], 
    "classes":[classidx1, class_idx2,...], 
    "scores":[score_1, score_2,...], 
    "labels": [label1, label2, ...], 
    "tensorflow_model_output":<original output of the model>}
```

If `accept` is set to `application/json`, then the model only outputs normalized boxes, classes, and scores. 

## Amazon EC2 instance recommendation for the Object Detection - TensorFlow algorithm
<a name="object-detection-tensorflow-instances"></a>

The Object Detection - TensorFlow algorithm supports all GPU instances for training, including:
+ `ml.p2.xlarge`
+ `ml.p2.16xlarge`
+ `ml.p3.2xlarge`
+ `ml.p3.16xlarge`

We recommend GPU instances with more memory for training with large batch sizes. Both CPU (such as M5) and GPU (P2 or P3) instances can be used for inference. For a comprehensive list of SageMaker training and inference instances across AWS Regions, see [Amazon SageMaker Pricing](https://aws.amazon.com/sagemaker/pricing/).

## Object Detection - TensorFlow sample notebooks
<a name="object-detection-tensorflow-sample-notebooks"></a>

For more information about how to use the SageMaker AI Object Detection - TensorFlow algorithm for transfer learning on a custom dataset, see the [Introduction to SageMaker TensorFlow - Object Detection](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/object_detection_tensorflow/Amazon_Tensorflow_Object_Detection.ipynb) notebook.

For instructions how to create and access Jupyter notebook instances that you can use to run the example in SageMaker AI, see [Amazon SageMaker notebook instances](nbi.md). After you have created a notebook instance and opened it, select the **SageMaker AI Examples** tab to see a list of all the SageMaker AI samples. To open a notebook, choose its **Use** tab and choose **Create copy**.

# How Object Detection - TensorFlow Works
<a name="object-detection-tensorflow-HowItWorks"></a>

The Object Detection - TensorFlow algorithm takes an image as input and predicts bounding boxes and object labels. Various deep learning networks such as MobileNet, ResNet, Inception, and EfficientNet are highly accurate for object detection. There are also deep learning networks that are trained on large image datasets, such as Common Objects in Context (COCO), which has 328,000 images. After a network is trained with COCO data, you can then fine-tune the network on a dataset with a particular focus to perform more specific object detection tasks. The Amazon SageMaker AI Object Detection - TensorFlow algorithm supports transfer learning on many pretrained models that are available in the TensorFlow Model Garden.

According to the number of class labels in your training data, an object detection layer is attached to the pretrained TensorFlow model of your choice. You can then fine-tune either the entire network (including the pretrained model) or only the top classification layer on new training data. With this method of transfer learning, training with smaller datasets is possible.

# TensorFlow Models
<a name="object-detection-tensorflow-Models"></a>

The following pretrained models are available to use for transfer learning with the Object Detection - TensorFlow algorithm. 

The following models vary significantly in size, number of model parameters, training time, and inference latency for any given dataset. The best model for your use case depends on the complexity of your fine-tuning dataset and any requirements that you have on training time, inference latency, or model accuracy.


| Model Name | `model_id` | Source | 
| --- | --- | --- | 
| ResNet50 V1 FPN 640 | `tensorflow-od1-ssd-resnet50-v1-fpn-640x640-coco17-tpu-8` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8.tar.gz) | 
| EfficientDet D0 512 | `tensorflow-od1-ssd-efficientdet-d0-512x512-coco17-tpu-8` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/efficientdet_d0_coco17_tpu-32.tar.gz) | 
| EfficientDet D1 640 | `tensorflow-od1-ssd-efficientdet-d1-640x640-coco17-tpu-8` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/efficientdet_d1_coco17_tpu-32.tar.gz) | 
| EfficientDet D2 768 | `tensorflow-od1-ssd-efficientdet-d2-768x768-coco17-tpu-8` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/efficientdet_d2_coco17_tpu-32.tar.gz) | 
| EfficientDet D3 896 | `tensorflow-od1-ssd-efficientdet-d3-896x896-coco17-tpu-32` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/efficientdet_d3_coco17_tpu-32.tar.gz) | 
| MobileNet V1 FPN 640 | `tensorflow-od1-ssd-mobilenet-v1-fpn-640x640-coco17-tpu-8` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v1_fpn_640x640_coco17_tpu-8.tar.gz) | 
| MobileNet V2 FPNLite 320 | `tensorflow-od1-ssd-mobilenet-v2-fpnlite-320x320-coco17-tpu-8` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.tar.gz) | 
| MobileNet V2 FPNLite 640 | `tensorflow-od1-ssd-mobilenet-v2-fpnlite-640x640-coco17-tpu-8` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8.tar.gz) | 
| ResNet50 V1 FPN 1024 | `tensorflow-od1-ssd-resnet50-v1-fpn-1024x1024-coco17-tpu-8` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_resnet50_v1_fpn_1024x1024_coco17_tpu-8.tar.gz) | 
| ResNet101 V1 FPN 640 | `tensorflow-od1-ssd-resnet101-v1-fpn-640x640-coco17-tpu-8` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_resnet101_v1_fpn_640x640_coco17_tpu-8.tar.gz) | 
| ResNet101 V1 FPN 1024 | `tensorflow-od1-ssd-resnet101-v1-fpn-1024x1024-coco17-tpu-8` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_resnet101_v1_fpn_1024x1024_coco17_tpu-8.tar.gz) | 
| ResNet152 V1 FPN 640 | `tensorflow-od1-ssd-resnet152-v1-fpn-640x640-coco17-tpu-8` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_resnet152_v1_fpn_640x640_coco17_tpu-8.tar.gz) | 
| ResNet152 V1 FPN 1024 | `tensorflow-od1-ssd-resnet152-v1-fpn-1024x1024-coco17-tpu-8` | [TensorFlow Model Garden link](http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_resnet152_v1_fpn_1024x1024_coco17_tpu-8.tar.gz) | 

# Object Detection - TensorFlow Hyperparameters
<a name="object-detection-tensorflow-Hyperparameter"></a>

Hyperparameters are parameters that are set before a machine learning model begins learning. The following hyperparameters are supported by the Amazon SageMaker AI built-in Object Detection - TensorFlow algorithm. See [Tune an Object Detection - TensorFlow model](object-detection-tensorflow-tuning.md) for information on hyperparameter tuning. 


| Parameter Name | Description | 
| --- | --- | 
| batch\$1size |  The batch size for training.  Valid values: positive integer. Default value: `3`.  | 
| beta\$11 |  The beta1 for the `"adam"` optimizer. Represents the exponential decay rate for the first moment estimates. Ignored for other optimizers. Valid values: float, range: [`0.0`, `1.0`]. Default value: `0.9`.  | 
| beta\$12 |  The beta2 for the `"adam"` optimizer. Represents the exponential decay rate for the second moment estimates. Ignored for other optimizers. Valid values: float, range: [`0.0`, `1.0`]. Default value: `0.999`.  | 
| early\$1stopping |  Set to `"True"` to use early stopping logic during training. If `"False"`, early stopping is not used. Valid values: string, either: (`"True"` or `"False"`). Default value: `"False"`.  | 
| early\$1stopping\$1min\$1delta | The minimum change needed to qualify as an improvement. An absolute change less than the value of early\$1stopping\$1min\$1delta does not qualify as improvement. Used only when early\$1stopping is set to "True".Valid values: float, range: [`0.0`, `1.0`].Default value: `0.0`. | 
| early\$1stopping\$1patience |  The number of epochs to continue training with no improvement. Used only when `early_stopping` is set to `"True"`. Valid values: positive integer. Default value: `5`.  | 
| epochs |  The number of training epochs. Valid values: positive integer. Default value: `5` for smaller models, `1` for larger models.  | 
| epsilon |  The epsilon for `"adam"`, `"rmsprop"`, `"adadelta"`, and `"adagrad"` optimizers. Usually set to a small value to avoid division by 0. Ignored for other optimizers. Valid values: float, range: [`0.0`, `1.0`]. Default value: `1e-7`.  | 
| initial\$1accumulator\$1value |  The starting value for the accumulators, or the per-parameter momentum values, for the `"adagrad"` optimizer. Ignored for other optimizers. Valid values: float, range: [`0.0`, `1.0`]. Default value: `0.1`.  | 
| learning\$1rate | The optimizer learning rate. Valid values: float, range: [`0.0`, `1.0`].Default value: `0.001`. | 
| momentum |  The momentum for the `"sgd"` and `"nesterov"` optimizers. Ignored for other optimizers. Valid values: float, range: [`0.0`, `1.0`]. Default value: `0.9`.  | 
| optimizer |  The optimizer type. For more information, see [Optimizers](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers) in the TensorFlow documentation. Valid values: string, any of the following: (`"adam"`, `"sgd"`, `"nesterov"`, `"rmsprop"`,` "adagrad"` , `"adadelta"`). Default value: `"adam"`.  | 
| reinitialize\$1top\$1layer |  If set to `"Auto"`, the top classification layer parameters are re-initialized during fine-tuning. For incremental training, top classification layer parameters are not re-initialized unless set to `"True"`. Valid values: string, any of the following: (`"Auto"`, `"True"` or `"False"`). Default value: `"Auto"`.  | 
| rho |  The discounting factor for the gradient of the `"adadelta"` and `"rmsprop"` optimizers. Ignored for other optimizers.  Valid values: float, range: [`0.0`, `1.0`]. Default value: `0.95`.  | 
| train\$1only\$1on\$1top\$1layer |  If `"True"`, only the top classification layer parameters are fine-tuned. If `"False"`, all model parameters are fine-tuned. Valid values: string, either: (`"True"` or `"False"`). Default value: `"False"`.  | 

# Tune an Object Detection - TensorFlow model
<a name="object-detection-tensorflow-tuning"></a>

*Automatic model tuning*, also known as hyperparameter tuning, finds the best version of a model by running many jobs that test a range of hyperparameters on your dataset. You choose the tunable hyperparameters, a range of values for each, and an objective metric. You choose the objective metric from the metrics that the algorithm computes. Automatic model tuning searches the hyperparameters chosen to find the combination of values that result in the model that optimizes the objective metric.

For more information about model tuning, see [Automatic model tuning with SageMaker AI](automatic-model-tuning.md).

## Metrics computed by the Object Detection - TensorFlow algorithm
<a name="object-detection-tensorflow-metrics"></a>

Refer to the following chart to find which metrics are computed by the Object Detection - TensorFlow algorithm.


| Metric Name | Description | Optimization Direction | Regex Pattern | 
| --- | --- | --- | --- | 
| validation:localization\$1loss | The localization loss for box prediction. | Minimize | `Val_localization=([0-9\\.]+)` | 

## Tunable Object Detection - TensorFlow hyperparameters
<a name="object-detection-tensorflow-tunable-hyperparameters"></a>

Tune an object detection model with the following hyperparameters. The hyperparameters that have the greatest impact on object detection objective metrics are: `batch_size`, `learning_rate`, and `optimizer`. Tune the optimizer-related hyperparameters, such as `momentum`, `regularizers_l2`, `beta_1`, `beta_2`, and `eps` based on the selected `optimizer`. For example, use `beta_1` and `beta_2` only when `adam` is the `optimizer`.

For more information about which hyperparameters are used for each `optimizer`, see [Object Detection - TensorFlow Hyperparameters](object-detection-tensorflow-Hyperparameter.md).


| Parameter Name | Parameter Type | Recommended Ranges | 
| --- | --- | --- | 
| batch\$1size | IntegerParameterRanges | MinValue: 8, MaxValue: 512 | 
| beta\$11 | ContinuousParameterRanges | MinValue: 1e-6, MaxValue: 0.999 | 
| beta\$12 | ContinuousParameterRanges | MinValue: 1e-6, MaxValue: 0.999 | 
| eps | ContinuousParameterRanges | MinValue: 1e-8, MaxValue: 1.0 | 
| learning\$1rate | ContinuousParameterRanges | MinValue: 1e-6, MaxValue: 0.5 | 
| momentum | ContinuousParameterRanges | MinValue: 0.0, MaxValue: 0.999 | 
| optimizer | CategoricalParameterRanges | ['sgd', ‘adam’, ‘rmsprop’, 'nesterov', 'adagrad', 'adadelta'] | 
| regularizers\$1l2 | ContinuousParameterRanges | MinValue: 0.0, MaxValue: 0.999 | 
| train\$1only\$1on\$1top\$1layer | CategoricalParameterRanges | ['True', 'False'] | 
| initial\$1accumulator\$1value | CategoricalParameterRanges | MinValue: 0.0, MaxValue: 0.999 |