Exemples de scripts visuels personnalisés

Les exemples suivants effectuent des transformations équivalentes. Cela dit, le deuxième exemple (SparkSQL) est le plus propre et le plus efficace, suivi de l'exemple d'UDF Pandas et enfin du mappage de bas niveau du premier exemple. L'exemple suivant est une illustration complète de transformation simple permettant d'ajouter deux colonnes :

from awsglue import DynamicFrame
 
# You can have other auxiliary variables, functions or classes on this file, it won't affect the runtime
def record_sum(rec, col1, col2, resultCol):
    rec[resultCol] = rec[col1] + rec[col2]
    return rec
 
 
# The number and name of arguments must match the definition on json config file
# (expect self which is the current DynamicFrame to transform
# If an argument is optional, you need to define a default value here
#  (resultCol in this example is an optional argument)
def custom_add_columns(self, col1, col2, resultCol="result"):
    # The mapping will alter the columns order, which could be important
    fields = [field.name for field in self.schema()]
    if resultCol not in fields:
        # If it's a new column put it at the end
        fields.append(resultCol)
    return self.map(lambda record: record_sum(record, col1, col2, resultCol)).select_fields(paths=fields)
 
 
# The name we assign on DynamicFrame must match the configured "functionName"
DynamicFrame.custom_add_columns = custom_add_columns
        

L'exemple suivant illustre une transformation équivalente utilisant l'API SparkSQL.

from awsglue import DynamicFrame
 
# The number and name of arguments must match the definition on json config file
# (expect self which is the current DynamicFrame to transform
# If an argument is optional, you need to define a default value here
#  (resultCol in this example is an optional argument)
def custom_add_columns(self, col1, col2, resultCol="result"):
    df = self.toDF()
    return DynamicFrame.fromDF(
        df.withColumn(resultCol, df[col1] + df[col2]) # This is the conversion logic
        , self.glue_ctx, self.name) 
 
 
# The name we assign on DynamicFrame must match the configured "functionName"
DynamicFrame.custom_add_columns = custom_add_columns           
        

L'exemple suivant utilise les mêmes transformations mais en utilisant une UDF Pandas, plus efficace qu'une UDF simple. Pour plus d'informations sur l'écriture d'UDF Pandas, consultez la documentation Apache Spark SQL.

from awsglue import DynamicFrame
import pandas as pd
from pyspark.sql.functions import pandas_udf
 
# The number and name of arguments must match the definition on json config file
# (expect self which is the current DynamicFrame to transform
# If an argument is optional, you need to define a default value here
#  (resultCol in this example is an optional argument)
def custom_add_columns(self, col1, col2, resultCol="result"):
    @pandas_udf("integer")  # We need to declare the type of the result column
    def add_columns(value1: pd.Series, value2: pd.Series) → pd.Series:
        return value1 + value2
 
    df = self.toDF()
    return DynamicFrame.fromDF(
        df.withColumn(resultCol, add_columns(col1, col2)) # This is the conversion logic
        , self.glue_ctx, self.name) 
 
# The name we assign on DynamicFrame must match the configured "functionName"
DynamicFrame.custom_add_columns = custom_add_columns