Using data repository tasks to import changes - FSx for Lustre

Using data repository tasks to import changes

The import data repository task imports metadata of objects that are new or changed in your S3 data repository, creating a new file or directory listing for any new object in the S3 data repository. For any object that has been changed in the data repository, the corresponding file or directory listing is updated with the new metadata. No action is taken for objects that have been deleted from the data repository.

Use the following procedures to import metadata changes by using the Amazon FSx console and CLI. Note that you can use one data repository task for multiple DRAs.

  1. Open the Amazon FSx console at https://console.aws.amazon.com/fsx/.

  2. On the navigation pane, choose File systems, then choose your Lustre file system.

  3. Choose the Data repository tab.

  4. In the Data repository associations pane, choose the data repository associations you want to create the import task for.

  5. From the Actions menu, choose Import task. This choice isn't available if the file system isn't linked to a data repository. The Create import data repository task page appears.

  6. (Optional) Specify up to 32 directories or files to import from your linked S3 buckets by providing the paths to those directories or files in Data repository paths to import.

    Note

    If a path that you provide isn't valid, the task fails.

  7. (Optional) Choose Enable under Completion report to generate a task completion report after the task completes. A task completion report provides details about the files processed by the task that meet the scope provided in Report scope. To specify the location for Amazon FSx to deliver the report, enter a relative path on a linked S3 data repository for Report path.

  8. Choose Create.

    A notification at the top of the File systems page shows the task that you just created in progress.

To view the task status and details, scroll down to the Data Repository Tasks pane in the Data Repository tab for the file system. The default sort order shows the most recent task at the top of the list.

To view a task summary from this page, choose Task ID for the task you just created. The Summary page for the task appears.

  • Use the create-data-repository-task CLI command to import metadata changes on your FSx for Lustre file system. The corresponding API operation is CreateDataRepositoryTask.

    $ aws fsx create-data-repository-task \
    --file-system-id fs-0123456789abcdef0 \
    --type IMPORT_METADATA_FROM_REPOSITORY \
    --paths s3://bucketname1/dir1/path1 \
    --report Enabled=true,Path=s3://bucketname1/dir1/path1,Format=REPORT_CSV_20191124,Scope=FAILED_FILES_ONLY

    After successfully creating the data repository task, Amazon FSx returns the task description as JSON.

After creating the task to import metadata from the linked data repository, you can check the status of the import data repository task. For more information about viewing data repository tasks, see Accessing data repository tasks.