IoTSiteWise / Client / create_bulk_import_job

create_bulk_import_job¶

IoTSiteWise.Client.create_bulk_import_job(**kwargs)¶

Defines a job to ingest data to IoT SiteWise from Amazon S3. For more information, see Create a bulk import job (CLI) in the Amazon Simple Storage Service User Guide.

Warning

Before you create a bulk import job that ingests data into time series outside of a workspace, you must enable IoT SiteWise warm tier or IoT SiteWise cold tier. For more information about how to configure storage settings, see PutStorageConfiguration. This requirement doesn’t apply to bulk import jobs that ingest data into a session dataset in a workspace (jobs that specify a workspaceName and datasetId). Those jobs don’t use IoT SiteWise warm or cold tier storage.

Bulk import is designed to store historical data to IoT SiteWise.

Newly ingested data in the hot tier triggers notifications and computations.
After data moves from the hot tier to the warm or cold tier based on retention settings, it does not trigger computations or notifications.
Data older than 7 days does not trigger computations or notifications.

Request Syntax

response = client.create_bulk_import_job(
    jobName='string',
    jobRoleArn='string',
    files=[
        {
            'bucket': 'string',
            'key': 'string',
            'versionId': 'string',
            'alias': 'string',
            'startTime': {
                'timeInSeconds': 123,
                'offsetInNanos': 123
            },
            'fileFormat': {
                'csv': {
                    'columnNames': [
                        'ALIAS'|'ASSET_ID'|'PROPERTY_ID'|'DATA_TYPE'|'TIMESTAMP_SECONDS'|'TIMESTAMP_NANO_OFFSET'|'QUALITY'|'VALUE',
                    ]
                },
                'parquet': {}
                ,
                'mp4': {}
                ,
                'annotation': {}

            }
        },
    ],
    errorReportLocation={
        'bucket': 'string',
        'prefix': 'string'
    },
    jobConfiguration={
        'fileFormat': {
            'csv': {
                'columnNames': [
                    'ALIAS'|'ASSET_ID'|'PROPERTY_ID'|'DATA_TYPE'|'TIMESTAMP_SECONDS'|'TIMESTAMP_NANO_OFFSET'|'QUALITY'|'VALUE',
                ]
            },
            'parquet': {}
            ,
            'mp4': {}
            ,
            'annotation': {}

        }
    },
    adaptiveIngestion=True|False,
    deleteFilesAfterImport=True|False,
    datasetId='string',
    workspaceName='string'
)

Parameters:

jobName (string) –
[REQUIRED]

The unique name that helps identify the job request.
jobRoleArn (string) –
[REQUIRED]

The ARN of the IAM role that allows IoT SiteWise to read Amazon S3 data.
files (list) –
[REQUIRED]

The files in the specified Amazon S3 bucket that contain your data. You can specify up to 100 files for each bulk import job. Each file supports the following size limits:
- Parquet files – Up to 256 MiB.
- Other file formats – Up to 5 GiB.
- (dict) –
  
  The file in Amazon S3 where your data is saved.
  - bucket (string) – [REQUIRED]
    
    The name of the Amazon S3 bucket from which data is imported.
  - key (string) – [REQUIRED]
    
    The key of the Amazon S3 object that contains your data. Each object has a key that is a unique identifier. Each object has exactly one key.
  - versionId (string) –
    
    The version ID to identify a specific version of the Amazon S3 object that contains your data.
  - alias (string) –
    
    The alias associated with the file’s time series.
  - startTime (dict) –
    
    The nanosecond-precision start time for the file data.
    - timeInSeconds (integer) – [REQUIRED]
      
      The timestamp date, in seconds, in the Unix epoch format. Fractional nanosecond data is provided by offsetInNanos.
    - offsetInNanos (integer) –
      
      The nanosecond offset from timeInSeconds.
  - fileFormat (dict) –
    
    The file format of the data in S3.
    - csv (dict) –
      
      The file is in .CSV format.
      - columnNames (list) – [REQUIRED]
        
        The column names specified in the .csv file.
        
        (string) –
    - parquet (dict) –
      
      The file is in parquet format.
    - mp4 (dict) –
      
      The MP4 format configuration.
    - annotation (dict) –
      
      The annotation format configuration.
errorReportLocation (dict) –
[REQUIRED]

The Amazon S3 destination where errors associated with the job creation request are saved.
- bucket (string) – [REQUIRED]
  
  The name of the Amazon S3 bucket to which errors associated with the bulk import job are sent.
- prefix (string) – [REQUIRED]
  
  Amazon S3 uses the prefix as a folder name to organize data in the bucket. Each Amazon S3 object has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/). For more information, see Organizing objects using prefixes in the Amazon Simple Storage Service User Guide.
jobConfiguration (dict) –
Contains the configuration information of a job, such as the file format used to save data in Amazon S3.
- fileFormat (dict) –
  
  The file format of the data in S3.
  - csv (dict) –
    
    The file is in .CSV format.
    - columnNames (list) – [REQUIRED]
      
      The column names specified in the .csv file.
      - (string) –
  - parquet (dict) –
    
    The file is in parquet format.
  - mp4 (dict) –
    
    The MP4 format configuration.
  - annotation (dict) –
    
    The annotation format configuration.
adaptiveIngestion (boolean) – If set to true, ingest new data into IoT SiteWise storage. Measurements with notifications, metrics and transforms are computed. If set to false, historical data is ingested into IoT SiteWise as is.
deleteFilesAfterImport (boolean) – If set to true, your data files is deleted from S3, after ingestion into IoT SiteWise storage.
datasetId (string) – The ID of the session dataset to ingest data into. Specify this field, together with workspaceName, to ingest data into a session dataset in a workspace.
workspaceName (string) – The name of the workspace that contains the session dataset. Specify this field together with datasetId.

Return type:

dict

Returns:

Response Syntax

{
    'jobId': 'string',
    'jobName': 'string',
    'jobStatus': 'PENDING'|'CANCELLED'|'RUNNING'|'COMPLETED'|'FAILED'|'COMPLETED_WITH_FAILURES'
}

Response Structure

(dict) –
- jobId (string) –
  
  The ID of the job.
- jobName (string) –
  
  The unique name that helps identify the job request.
- jobStatus (string) –
  
  The status of the bulk import job can be one of following values:
  - PENDING – IoT SiteWise is waiting for the current bulk import job to finish.
  - CANCELLED – The bulk import job has been canceled.
  - RUNNING – IoT SiteWise is processing your request to import your data from Amazon S3.
  - COMPLETED – IoT SiteWise successfully completed your request to import data from Amazon S3.
  - FAILED – IoT SiteWise couldn’t process your request to import data from Amazon S3. You can use logs saved in the specified error report location in Amazon S3 to troubleshoot issues.
  - COMPLETED_WITH_FAILURES – IoT SiteWise completed your request to import data from Amazon S3 with errors. You can use logs saved in the specified error report location in Amazon S3 to troubleshoot issues.