在 EMR Serverless 中使用不同的 Python 版本

从 Python 虚拟环境提交作业

1. 使用以下示例中的命令构建虚拟环境。此示例将 Python 3.9.9 安装到虚拟环境包中，并将存档复制到 Amazon S3 位置。

   重要

   如果使用 Amazon EMR 7.0.0 及更高版本，则必须在类似于 EMR Serverless 应用程序所用环境的 Amazon Linux 2023 环境中运行命令。

   如果使用 6.15.0 或更低版本，则必须在类似的 Amazon Linux 2 环境中运行以下命令。

   # install Python 3.9.9 and activate the venv
   yum install -y gcc openssl-devel bzip2-devel libffi-devel tar gzip wget make
   wget https://www.python.org/ftp/python/3.9.9/Python-3.9.9.tgz && \
   tar xzf Python-3.9.9.tgz && cd Python-3.9.9 && \
   ./configure --enable-optimizations && \
   make altinstall
   
   # create python venv with Python 3.9.9
   python3.9 -m venv pyspark_venv_python_3.9.9 --copies 
   source pyspark_venv_python_3.9.9/bin/activate
   
   # copy system python3 libraries to venv
   cp -r /usr/local/lib/python3.9/* ./pyspark_venv_python_3.9.9/lib/python3.9/
   
   # package venv to archive. 
   # **Note** that you have to supply --python-prefix option 
   # to make sure python starts with the path where your 
   # copied libraries are present.
   # Copying the python binary to the "environment" directory.
   pip3 install venv-pack
   venv-pack -f -o pyspark_venv_python_3.9.9.tar.gz --python-prefix /home/hadoop/environment
   
   # stage the archive in S3 
   aws s3 cp pyspark_venv_python_3.9.9.tar.gz s3://<path>
   
   # optionally, remove the virtual environment directory
   rm -fr pyspark_venv_python_3.9.9

2. 设置属性以使用 Python 虚拟环境，然后提交 Spark 作业。

   # note that the archive suffix "environment" is the same as the directory where you copied the Python binary.
   --conf spark.archives=s3://amzn-s3-demo-bucket/EXAMPLE-PREFIX/pyspark_venv_python_3.9.9.tar.gz#environment 
   --conf spark.emr-serverless.driverEnv.PYSPARK_DRIVER_PYTHON=./environment/bin/python
   --conf spark.emr-serverless.driverEnv.PYSPARK_PYTHON=./environment/bin/python 
   --conf spark.executorEnv.PYSPARK_PYTHON=./environment/bin/python