Streaming job log management
Streaming jobs support log rotation for Spark application logs and event logs, and log compaction for Spark event logs. This helps you manage your resources effectively.
Log rotation
Streaming jobs support log rotation for Spark application logs and event logs. Log rotation prevents long streaming jobs from generating large log files that might take up all of your available disk space. Log rotation helps you save disk storage and prevents job failures because of low disk space. For more information, see Rotating logs.
Log compaction
Streaming jobs also support log compaction for Spark event logs whenever managed logging is available. For more details about managed logging, see Logging with managed storage. Streaming jobs can run for a long time, and the amount of event data can build up over time and significantly increase log file sizes. The Spark History Server reads and loads these events into memory for the Spark application UI. This process can cause high latencies and costs, especially if event logs stored in Amazon S3 are very large.
Log compaction reduces the event log size, so the Spark History Server doesn't have to load more than 1 GB of event logs at any time. For more information, see
Monitoring and Instrumentation
