Use Spark properties to specify custom configuration - Amazon Athena
Using the Athena consoleUsing the AWS CLI or Athena API

Use Spark properties to specify custom configuration

When you create or edit a session in Amazon Athena for Apache Spark, you can use Spark properties to specify .jar files, packages, or another custom configuration for the session. To specify your Spark properties, you can use the Athena console, the AWS CLI, or the Athena API.

Use the Athena console to specify Spark properties

In the Athena console, you can specify your Spark properties when you create a notebook or edit a current session.

To add properties in the Create notebook or Edit session details dialog box

  1. Expand Spark properties.

  2. To add your properties, use the Edit in table or Edit in JSON option.

    • For the Edit in table option, choose Add property to add a property, or choose Remove to remove a property. Use the Key and Value boxes to enter property names and their values.

      • To add a custom .jar file, use the spark.jars property.

      • To specify a package file, use the spark.jars.packages property.

    • To enter and edit your configuration directly, choose the Edit in JSON option. In the JSON text editor, you can perform the following tasks:

      • Choose Copy to copy the JSON text to the clipboard.

      • Choose Clear to remove all text from the JSON editor.

      • Choose the settings (gear) icon to configure line wrapping or choose a color theme for the JSON editor.

Notes

  • You can set properties in Athena for Spark, which is the same as setting Spark properties directly on a SparkConf object.

  • Start all Spark properties with the spark. prefix. Properties with other prefixes are ignored.

  • Not all Spark properties are available for custom configuration on Athena. If you submit a StartSession request that has a restricted configuration, the session fails to start.

    • You cannot use the spark.athena. prefix because it is reserved.

Use the AWS CLI or Athena API to provide custom configuration

To use the AWS CLI or Athena API to provide your session configuration, use the StartSession API action or the start-session CLI command. In your StartSession request, use the SparkProperties field of EngineConfiguration object to pass your configuration information in JSON format. This starts a session with your specified configuration. For request syntax, see StartSession in the Amazon Athena API Reference.