0Column expressions and functions help you work with data in tables easily. They let you change, calculate, or filter data in columns.
from pyspark.sql.functions import col, expr, sum as _sum, avg # Using col to refer to a column col('column_name') # Using expr for expressions expr('column_name + 1') # Using functions like sum or avg _sum('column_name') avg('column_name')
col() helps you refer to a column by name.
expr() lets you write SQL-like expressions as strings.
from pyspark.sql.functions import col df.select(col('age') + 1)
from pyspark.sql.functions import expr df.select(expr('salary * 0.1'))
from pyspark.sql.functions import sum df.groupBy('department').agg(sum('salary'))
This program creates a small table of people with their ages and salaries. It adds two new columns: one with age plus 5, and one with 10% bonus of salary. Then it calculates the average salary and shows the table.
from pyspark.sql import SparkSession from pyspark.sql.functions import col, expr, sum as _sum, avg spark = SparkSession.builder.appName('ColumnExpressions').getOrCreate() # Create sample data data = [ (1, 'Alice', 30, 1000), (2, 'Bob', 35, 1500), (3, 'Charlie', 40, 2000), (4, 'Diana', 25, 1200) ] columns = ['id', 'name', 'age', 'salary'] df = spark.createDataFrame(data, columns) # Add a new column with age plus 5 new_df = df.withColumn('age_plus_5', col('age') + 5) # Calculate 10% bonus on salary new_df = new_df.withColumn('bonus', expr('salary * 0.1')) # Calculate average salary avg_salary = new_df.select(avg('salary')).collect()[0][0] # Show the new dataframe new_df.show() print(f'Average salary: {avg_salary}') spark.stop()
Use col() when you want to refer to columns in expressions clearly.
expr() is useful for complex expressions written as strings.
Functions like sum() and avg() help with quick calculations on columns.
Column expressions let you work with data inside columns easily.
You can add, change, or calculate new columns using functions.
Using these helps you analyze and prepare data for insights.
withColumn function do in Apache Spark DataFrame?withColumnwithColumn function is used to add a new column or replace an existing one in a DataFrame.drop, filtering by filter, and grouping by groupBy. So these are not correct.withColumn = add/replace column [OK]age from a DataFrame df using Spark's column expression?select method expects column names as strings or Column objects. Using double quotes around column names is the standard correct syntax.df.select("age") (A) uses a valid string literal.df.select(age) (B) references undefined variable age - error.df.select(df.age()) (C) tries to call df.age as a function - Column not callable.df.select(`age`) (D) uses invalid backtick syntax.df.select("age") executes without error and selects the column correctly.df.show()?
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.getOrCreate()
data = [(1, 10), (2, 20), (3, 30)]
df = spark.createDataFrame(data, ['id', 'value'])
df = df.withColumn('double_value', col('value') * 2)
df.show()withColumn adds 'double_value' by multiplying 'value' by 2.from pyspark.sql.functions import col
df = spark.createDataFrame([(1, 'Alice'), (2, 'Bob')], ['id', 'name'])
df = df.withColumn('name_length', length(col('name')))
df.show()length function but only imports col. The length function must be imported from pyspark.sql.functions.withColumn is correct, and DataFrame creation is valid.df with columns id, score1, and score2. You want to create a new column max_score that contains the maximum value between score1 and score2 for each row. Which code snippet correctly achieves this?greatest function to get the maximum value among columns for each row.greatest correctly with imported function. df = df.withColumn('max_score', max(col('score1'), col('score2'))) uses max which is an aggregate function, not for row-wise max. df = df.withColumn('max_score', col('score1') if col('score1') > col('score2') else col('score2')) tries Python if-else which does not work on columns. df = df.withColumn('max_score', col('score1').max(col('score2'))) uses invalid syntax.Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.