Specifying configuration options for a Delta Lake data store
When you configure a crawler for a Delta Lake data store, you specify these configuration parameters:
- Connection
-
Optionally select or add a Network connection to use with this Amazon S3 target. For information about connections, see Connecting to data.
- Create tables for querying
-
Select how you want to create the Delta Lake tables:
Create Native tables: Allow integration with query engines that support querying of the Delta transaction log directly.
Create Symlink tables: Create a symlink manifest folder with manifest files partitioned by the partition keys, based on the specified configuration parameters.
- Enable write manifest (configurable only you've selected to Create Symlink tables for a Delta Lake source
-
Select whether to detect table metadata or schema changes in the Delta Lake transaction log; it regenerates the manifest file. You should not choose this option if you configured an automatic manifest update with Delta Lake
SET TBLPROPERTIES
. - Include delta lake table path(s)
-
Specify one or more Amazon S3 paths to Delta tables as s3://
bucket
/prefix
/object
.