# Neptune Loader Get-Status API Gets the status of a `loader` job. To get load status, you must send an HTTP `GET` request to the `https://your-neptune-endpoint:port/loader` endpoint. To get the status for a particular load request, you must include the `loadId` as a URL parameter, or append the `loadId` to the URL path. Neptune only keeps track of the most recent 1,024 bulk load jobs, and only stores the last 10,000 error details per job. See [Neptune Loader Error and Feed Messages](loader-message.md) for a list of the error and feed messages returned by the loader in case of errors. **Contents** + [Neptune Loader Get-Status requests](load-api-reference-status-requests.md) + [Loader Get-Status request syntax](load-api-reference-status-requests.md#load-api-reference-status-request-syntax) + [Neptune Loader Get-Status request parameters](load-api-reference-status-requests.md#load-api-reference-status-parameters) + [Neptune Loader Get-Status Responses](load-api-reference-status-response.md) + [Neptune Loader Get-Status Response JSON layout](load-api-reference-status-response.md#load-api-reference-status-response-layout) + [Neptune Loader Get-Status `overallStatus` and `failedFeeds` response objects](load-api-reference-status-response.md#load-api-reference-status-response-objects) + [Neptune Loader Get-Status `errors` response object](load-api-reference-status-response.md#load-api-reference-status-errors) + [Neptune Loader Get-Status `errorLogs` response object](load-api-reference-status-response.md#load-api-reference-error-logs) + [Neptune Loader Get-Status Examples](load-api-reference-status-examples.md) + [Example request for load status](load-api-reference-status-examples.md#load-api-reference-status-examples-status-request) + [Example request for loadIds](load-api-reference-status-examples.md#load-api-reference-status-examples-loadId-request) + [Example request for detailed status](load-api-reference-status-examples.md#load-api-reference-status-examples-details-request) + [Neptune Loader Get-Status `errorLogs` examples](load-api-reference-error-logs-examples.md) + [Example detailed status response when errors occurred](load-api-reference-error-logs-examples.md#load-api-reference-status-examples-details-request-errors) + [Example of a `Data prefetch task interrupted` error](load-api-reference-error-logs-examples.md#load-api-reference-status-examples-task-interrupted) # Neptune Loader Get-Status requests ## Loader Get-Status request syntax ``` GET https://your-neptune-endpoint:port/loader?loadId=loadId ``` ``` GET https://your-neptune-endpoint:port/loader/loadId ``` ``` GET https://your-neptune-endpoint:port/loader ``` ## Neptune Loader Get-Status request parameters + **`loadId`**   –   The ID of the load job. If you do not specify a `loadId`, a list of load IDs is returned. + **`details`**   –   Include details beyond overall status. *Allowed values*: `TRUE`, `FALSE`. *Default value*: `FALSE`. + **`errors`**   –   Include the list of errors. *Allowed values*: `TRUE`, `FALSE`. *Default value*: `FALSE`. The list of errors is paged. The `page` and `errorsPerPage` parameters allow you to page through all the errors. + **`page`**   –   The error page number. Only valid with the `errors` parameter set to `TRUE`. *Allowed values*: Positive integers. *Default value*: 1. + **`errorsPerPage`**   –   The number of errors per each page. Only valid with the `errors` parameter set to `TRUE`. *Allowed values*: Positive integers. *Default value*: 10. + **`limit`**   –   The number of load ids to list. Only valid when requesting a list of load IDs by sending a `GET` request with no `loadId` specified. *Allowed values*: Positive integers from 1 through 100. *Default value*: 100. + **`includeQueuedLoads`**   –   An optional parameter that can be used to exclude the load IDs of queued load requests when a list of load IDs is requested. By default, the load IDs of all load jobs with status `LOAD_IN_QUEUE` are included in such a list. They appear before the load IDs of other jobs, sorted by the time they were added to the queue from most recent to earliest. *Allowed values*: `TRUE`, `FALSE`. *Default value*: `TRUE`. # Neptune Loader Get-Status Responses The following example response from the Neptune Get-Status API describes the overall structure of the response, explains the various fields and their data types, as well as the error handling and error log details. ## Neptune Loader Get-Status Response JSON layout The general layout of a loader status response is as follows: ``` { "status" : "200 OK", "payload" : { "feedCount" : [ { "LOAD_FAILED" : number } ], "overallStatus" : { "fullUri" : "s3://bucket/key", "runNumber" : number, "retryNumber" : number, "status" : "string", "totalTimeSpent" : number, "startTime" : number, "totalRecords" : number, "totalDuplicates" : number, "parsingErrors" : number, "datatypeMismatchErrors" : number, "insertErrors" : number, }, "failedFeeds" : [ { "fullUri" : "s3://bucket/key", "runNumber" : number, "retryNumber" : number, "status" : "string", "totalTimeSpent" : number, "startTime" : number, "totalRecords" : number, "totalDuplicates" : number, "parsingErrors" : number, "datatypeMismatchErrors" : number, "insertErrors" : number, } ], "errors" : { "startIndex" : number, "endIndex" : number, "loadId" : "string, "errorLogs" : [ ] } } } ``` ## Neptune Loader Get-Status `overallStatus` and `failedFeeds` response objects The possible responses returned for each failed feed, including the error descriptions, are the same as for the `overallStatus` object in a `Get-Status` response. The following fields appear in the `overallStatus` object for all loads, and the `failedFeeds` object for each failed feed: + **`fullUri` **  –   The URI of the file or files to be loaded. *Type:* *string* *Format*: `s3://bucket/key`. + **`runNumber`**   –   The run number of this load or feed. This is incremented when the load is restarted. *Type:* *unsigned long*. + **`retryNumber`**   –   The retry number of this load or feed. This is incremented when the loader automatically retries a feed or load. *Type:* *unsigned long*. + **`status`**   –   The returned status of the load or feed. `LOAD_COMPLETED` indicates a successful load with no problems. For a list of other load-status messages, see [Neptune Loader Error and Feed Messages](loader-message.md). *Type:* *string*. + **`totalTimeSpent`**   –   The time, in seconds, spent to parse and insert data for the load or feed. This does not include the time spent fetching the list of source files. *Type:* *unsigned long*. + **`totalRecords`**   –   Total records loaded or attempted to load. *Type:* *unsigned long*. Note that when loading from a CSV file, the record count does not refer to the number of lines loaded, but rather to the number of individual records in those lines. For example, take a tiny CSV file like this: ``` ~id,~label,name,team 'P-1','Player','Stokes','England' ``` Neptune would consider this file to contain 3 records, namely: ``` P-1 label Player P-1 name Stokes P-1 team England ``` + **`totalDuplicates`**   –   The number of duplicate records encountered. *Type:* *unsigned long*. As in the case of the `totalRecords` count, this value contains the number of individual duplicate records in a CSV file, not the number of duplicate lines. Take this small CSV file, for example: ``` ~id,~label,name,team P-2,Player,Kohli,India P-2,Player,Kohli,India ``` The status returned after loading it would look like this, reporting 6 total records, of which 3 are duplicates: ``` { "status": "200 OK", "payload": { "feedCount": [ { "LOAD_COMPLETED": 1 } ], "overallStatus": { "fullUri": "(the URI of the CSV file)", "runNumber": 1, "retryNumber": 0, "status": "LOAD_COMPLETED", "totalTimeSpent": 3, "startTime": 1662131463, "totalRecords": 6, "totalDuplicates": 3, "parsingErrors": 0, "datatypeMismatchErrors": 0, "insertErrors": 0 } } } ``` For openCypher loads, a duplicate is counted when: + The loader detects that a row in a node file has an ID without an ID space that is the same as another ID value without an ID space, either in another row or belonging to an existing node. + The loader detects that a row in a node file has an ID with an ID space that is the same as another ID value with ID space, either in another row or belonging to an existing node. See [Special considerations for loading openCypher data](load-api-reference-load.md#load-api-reference-load-parameters-opencypher). + **`parsingErrors`**   –   The number of parsing errors encountered. *Type:* *unsigned long*. + **`datatypeMismatchErrors`**   –   The number of records with a data type that did not match the given data. *Type:* *unsigned long*. + **`insertErrors`**   –   The number of records that could not be inserted due to errors. *Type:* *unsigned long*. ## Neptune Loader Get-Status `errors` response object Errors fall into the following categories: + **`Error 400`**   –   An invalid `loadId` returns an HTTP `400` bad request error. The message describes the error. + **`Error 500`**   –   A valid request that cannot be processed returns an HTTP `500` internal server error. The message describes the error. See [Neptune Loader Error and Feed Messages](loader-message.md) for a list of the error and feed messages returned by the loader in case of errors. When an error occurs, a JSON `errors` object is returned in the `BODY` of the response, with the following fields: + **`startIndex`**   –   The index of the first included error. *Type:* *unsigned long*. + **`endIndex`**   –   The index of the last included error. *Type:* *unsigned long*. + **`loadId`**   –   The ID of the load. You can use this ID to print the errors for the load by setting the `errors` parameter to `TRUE`. *Type:* *string*. + **`errorLogs`**   –   A list of the errors. *Type:* *list*. ## Neptune Loader Get-Status `errorLogs` response object The `errorLogs` object under `errors` in the loader Get-Status response contains an object describing each error using the following fields: + **`errorCode`**   –   Identifies the nature of error. It can take one of the following values: + `PARSING_ERROR` + `S3_ACCESS_DENIED_ERROR` + `FROM_OR_TO_VERTEX_ARE_MISSING` + `ID_ASSIGNED_TO_MULTIPLE_EDGES` + `SINGLE_CARDINALITY_VIOLATION` + `FILE_MODIFICATION_OR_DELETION_ERROR` + `OUT_OF_MEMORY_ERROR` + `INTERNAL_ERROR` (returned when the bulk loader cannot determine the type of the error). + **`errorMessage`**   –   A message describing the error. This can be a generic message associated with the error code or a specific message containing details, for example about a missing from/to vertex or about a parsing error. + **`fileName`**   –   The name of the feed. + **`recordNum`**   –   In the case of a parsing error, this is the record number in the file of the record that could not be parsed. It is set to zero if the record number is not applicable to the error, or if it could not be determined. For example, the bulk loader would generate a parsing error if it encountered a faulty row such as the following in an RDF `nquads` file: ``` |http://base#predicate> . ``` As you can see, the second `http` in the row above should be preceded by  `<`  rather than  `|` . The resulting error object under `errorLogs` in a status response would look like this: ``` { "errorCode" : "PARSING_ERROR", "errorMessage" : "Expected '<', found: |", "fileName" : "s3://bucket/key", "recordNum" : 12345 }, ``` # Neptune Loader Get-Status Examples The following examples showcase the usage of the Neptune loader's GET-Status API, which allows you to retrieve information about the status of your data loads into the Amazon Neptune graph database. These examples cover three main scenarios: retrieving the status of a specific load, listing the available load IDs, and requesting detailed status information for a specific load. ## Example request for load status The following is a request sent via HTTP `GET` using the `curl` command. ------ #### [ AWS CLI ] ``` aws neptunedata get-loader-job-status \ --endpoint-url https://your-neptune-endpoint:port \ --load-id loadId (a UUID) ``` For more information, see [get-loader-job-status](https://docs.aws.amazon.com/cli/latest/reference/neptunedata/get-loader-job-status.html) in the AWS CLI Command Reference. ------ #### [ SDK ] ``` import boto3 from botocore.config import Config client = boto3.client( 'neptunedata', endpoint_url='https://your-neptune-endpoint:port', config=Config(read_timeout=None, retries={'total_max_attempts': 1}) ) response = client.get_loader_job_status( loadId='loadId (a UUID)' ) print(response) ``` ------ #### [ awscurl ] ``` awscurl 'https://your-neptune-endpoint:port/loader/loadId (a UUID)' \ --region us-east-1 \ --service neptune-db ``` **Note** This example assumes that your AWS credentials are configured in your environment. Replace *us-east-1* with the Region of your Neptune cluster. ------ #### [ curl ] ``` curl -X GET 'https://your-neptune-endpoint:port/loader/loadId (a UUID)' ``` ------ **Example Response** ``` { "status" : "200 OK", "payload" : { "feedCount" : [ { "LOAD_FAILED" : 1 } ], "overallStatus" : { "datatypeMismatchErrors" : 0, "fullUri" : "s3://bucket/key", "insertErrors" : 0, "parsingErrors" : 5, "retryNumber" : 0, "runNumber" : 1, "status" : "LOAD_FAILED", "totalDuplicates" : 0, "totalRecords" : 5, "totalTimeSpent" : 3.0 } } } ``` ## Example request for loadIds The following is a request sent via HTTP `GET` using the `curl` command. ------ #### [ AWS CLI ] ``` aws neptunedata list-loader-jobs \ --endpoint-url https://your-neptune-endpoint:port \ --limit 3 ``` For more information, see [list-loader-jobs](https://docs.aws.amazon.com/cli/latest/reference/neptunedata/list-loader-jobs.html) in the AWS CLI Command Reference. ------ #### [ SDK ] ``` import boto3 from botocore.config import Config client = boto3.client( 'neptunedata', endpoint_url='https://your-neptune-endpoint:port', config=Config(read_timeout=None, retries={'total_max_attempts': 1}) ) response = client.list_loader_jobs( limit=3 ) print(response) ``` ------ #### [ awscurl ] ``` awscurl 'https://your-neptune-endpoint:port/loader?limit=3' \ --region us-east-1 \ --service neptune-db ``` **Note** This example assumes that your AWS credentials are configured in your environment. Replace *us-east-1* with the Region of your Neptune cluster. ------ #### [ curl ] ``` curl -X GET 'https://your-neptune-endpoint:port/loader?limit=3' ``` ------ **Example Response** ``` { "status" : "200 OK", "payload" : { "loadIds" : [ "a2c0ce44-a44b-4517-8cd4-1dc144a8e5b5", "09683a01-6f37-4774-bb1b-5620d87f1931", "58085eb8-ceb4-4029-a3dc-3840969826b9" ] } } ``` ## Example request for detailed status The following is a request sent via HTTP `GET` using the `curl` command. ------ #### [ AWS CLI ] ``` aws neptunedata get-loader-job-status \ --endpoint-url https://your-neptune-endpoint:port \ --load-id loadId (a UUID) \ --details ``` For more information, see [get-loader-job-status](https://docs.aws.amazon.com/cli/latest/reference/neptunedata/get-loader-job-status.html) in the AWS CLI Command Reference. ------ #### [ SDK ] ``` import boto3 from botocore.config import Config client = boto3.client( 'neptunedata', endpoint_url='https://your-neptune-endpoint:port', config=Config(read_timeout=None, retries={'total_max_attempts': 1}) ) response = client.get_loader_job_status( loadId='loadId (a UUID)', details=True ) print(response) ``` ------ #### [ awscurl ] ``` awscurl 'https://your-neptune-endpoint:port/loader/loadId (a UUID)?details=true' \ --region us-east-1 \ --service neptune-db ``` **Note** This example assumes that your AWS credentials are configured in your environment. Replace *us-east-1* with the Region of your Neptune cluster. ------ #### [ curl ] ``` curl -X GET 'https://your-neptune-endpoint:port/loader/loadId (a UUID)?details=true' ``` ------ **Example Response** ``` { "status" : "200 OK", "payload" : { "failedFeeds" : [ { "datatypeMismatchErrors" : 0, "fullUri" : "s3://bucket/key", "insertErrors" : 0, "parsingErrors" : 5, "retryNumber" : 0, "runNumber" : 1, "status" : "LOAD_FAILED", "totalDuplicates" : 0, "totalRecords" : 5, "totalTimeSpent" : 3.0 } ], "feedCount" : [ { "LOAD_FAILED" : 1 } ], "overallStatus" : { "datatypeMismatchErrors" : 0, "fullUri" : "s3://bucket/key", "insertErrors" : 0, "parsingErrors" : 5, "retryNumber" : 0, "runNumber" : 1, "status" : "LOAD_FAILED", "totalDuplicates" : 0, "totalRecords" : 5, "totalTimeSpent" : 3.0 } } } ``` # Neptune Loader Get-Status `errorLogs` examples The following examples showcase the detailed status response from the Neptune loader when errors have occurred during the data loading process. The examples illustrate the structure of the response, including information about failed feeds, overall status, and detailed error logs. ## Example detailed status response when errors occurred This a request sent via HTTP `GET` using `curl`: ------ #### [ AWS CLI ] ``` aws neptunedata get-loader-job-status \ --endpoint-url https://your-neptune-endpoint:port \ --load-id 0a237328-afd5-4574-a0bc-c29ce5f54802 \ --details \ --errors \ --errors-per-page 3 \ --page 1 ``` For more information, see [get-loader-job-status](https://docs.aws.amazon.com/cli/latest/reference/neptunedata/get-loader-job-status.html) in the AWS CLI Command Reference. ------ #### [ SDK ] ``` import boto3 from botocore.config import Config client = boto3.client( 'neptunedata', endpoint_url='https://your-neptune-endpoint:port', config=Config(read_timeout=None, retries={'total_max_attempts': 1}) ) response = client.get_loader_job_status( loadId='0a237328-afd5-4574-a0bc-c29ce5f54802', details=True, errors=True, errorsPerPage=3, page=1 ) print(response) ``` ------ #### [ awscurl ] ``` awscurl 'https://your-neptune-endpoint:port/loader/0a237328-afd5-4574-a0bc-c29ce5f54802?details=true&errors=true&page=1&errorsPerPage=3' \ --region us-east-1 \ --service neptune-db ``` **Note** This example assumes that your AWS credentials are configured in your environment. Replace *us-east-1* with the Region of your Neptune cluster. ------ #### [ curl ] ``` curl -X GET 'https://your-neptune-endpoint:port/loader/0a237328-afd5-4574-a0bc-c29ce5f54802?details=true&errors=true&page=1&errorsPerPage=3' ``` ------ **Example of a detailed response when errors have occurred** This is an example of the response that you might get from the query above, with an `errorLogs` object listing the load errors encountered: ``` { "status" : "200 OK", "payload" : { "failedFeeds" : [ { "datatypeMismatchErrors" : 0, "fullUri" : "s3://bucket/key", "insertErrors" : 0, "parsingErrors" : 5, "retryNumber" : 0, "runNumber" : 1, "status" : "LOAD_FAILED", "totalDuplicates" : 0, "totalRecords" : 5, "totalTimeSpent" : 3.0 } ], "feedCount" : [ { "LOAD_FAILED" : 1 } ], "overallStatus" : { "datatypeMismatchErrors" : 0, "fullUri" : "s3://bucket/key", "insertErrors" : 0, "parsingErrors" : 5, "retryNumber" : 0, "runNumber" : 1, "status" : "LOAD_FAILED", "totalDuplicates" : 0, "totalRecords" : 5, "totalTimeSpent" : 3.0 }, "errors" : { "endIndex" : 3, "errorLogs" : [ { "errorCode" : "PARSING_ERROR", "errorMessage" : "Expected '<', found: |", "fileName" : "s3://bucket/key", "recordNum" : 1 }, { "errorCode" : "PARSING_ERROR", "errorMessage" : "Expected '<', found: |", "fileName" : "s3://bucket/key", "recordNum" : 2 }, { "errorCode" : "PARSING_ERROR", "errorMessage" : "Expected '<', found: |", "fileName" : "s3://bucket/key", "recordNum" : 3 } ], "loadId" : "0a237328-afd5-4574-a0bc-c29ce5f54802", "startIndex" : 1 } } } ``` ## Example of a `Data prefetch task interrupted` error Occasionally when you get a `LOAD_FAILED` status and then request more detailed information, the error returned may be a `PARSING_ERROR` with a `Data prefetch task interrupted` message, like this: ``` "errorLogs" : [ { "errorCode" : "PARSING_ERROR", "errorMessage" : "Data prefetch task interrupted: Data prefetch task for 11467 failed", "fileName" : "s3://amzn-s3-demo-bucket/some-source-file", "recordNum" : 0 } ] ``` This error occurs when there was a temporary interruption in the data load process that was typically not caused by your request or your data. It can usually be resolved simply by running the bulk upload request again. If you are using default settings, namely `"mode":"AUTO"`, and `"failOnError":"TRUE"`, the loader skips the files that it already successfully loaded and resumes loading files it had not yet loaded when the interruption occurred.