0from pyspark.sql import SparkSession, functions as F spark = SparkSession.builder.getOrCreate() df = spark.createDataFrame([(1, 10), (2, 20)], ['id', 'value']) df2 = df.withColumn('value_plus_5', F.col('value') + 5) df2.show()
| Step | Action | Expression Evaluated | Resulting Column Values |
|---|---|---|---|
| 1 | Create DataFrame | N/A | [{'id':1, 'value':10}, {'id':2, 'value':20}] |
| 2 | Select column 'value' | F.col('value') | [10, 20] |
| 3 | Add 5 to 'value' | F.col('value') + 5 | [15, 25] |
| 4 | Add new column 'value_plus_5' | withColumn('value_plus_5', F.col('value') + 5) | [{'id':1, 'value':10, 'value_plus_5':15}, {'id':2, 'value':20, 'value_plus_5':25}] |
| 5 | Show DataFrame | df2.show() | Displays table with columns: id, value, value_plus_5 |
| Variable | Start | After Step 1 | After Step 4 | Final |
|---|---|---|---|---|
| df | None | [{'id':1, 'value':10}, {'id':2, 'value':20}] | [{'id':1, 'value':10}, {'id':2, 'value':20}] | [{'id':1, 'value':10}, {'id':2, 'value':20}] |
| df2 | None | None | [{'id':1, 'value':10, 'value_plus_5':15}, {'id':2, 'value':20, 'value_plus_5':25}] | [{'id':1, 'value':10, 'value_plus_5':15}, {'id':2, 'value':20, 'value_plus_5':25}] |
Column expressions in Spark allow you to create or transform columns using functions.
Use F.col('colname') to refer to columns.
Apply expressions like addition, multiplication, or built-in functions.
Use withColumn() to add or replace columns.
DataFrames are immutable; transformations return new DataFrames.withColumn function do in Apache Spark DataFrame?withColumnwithColumn function is used to add a new column or replace an existing one in a DataFrame.drop, filtering by filter, and grouping by groupBy. So these are not correct.withColumn = add/replace column [OK]age from a DataFrame df using Spark's column expression?select method expects column names as strings or Column objects. Using double quotes around column names is the standard correct syntax.df.select("age") (A) uses a valid string literal.df.select(age) (B) references undefined variable age - error.df.select(df.age()) (C) tries to call df.age as a function - Column not callable.df.select(`age`) (D) uses invalid backtick syntax.df.select("age") executes without error and selects the column correctly.df.show()?
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.getOrCreate()
data = [(1, 10), (2, 20), (3, 30)]
df = spark.createDataFrame(data, ['id', 'value'])
df = df.withColumn('double_value', col('value') * 2)
df.show()withColumn adds 'double_value' by multiplying 'value' by 2.from pyspark.sql.functions import col
df = spark.createDataFrame([(1, 'Alice'), (2, 'Bob')], ['id', 'name'])
df = df.withColumn('name_length', length(col('name')))
df.show()length function but only imports col. The length function must be imported from pyspark.sql.functions.withColumn is correct, and DataFrame creation is valid.df with columns id, score1, and score2. You want to create a new column max_score that contains the maximum value between score1 and score2 for each row. Which code snippet correctly achieves this?greatest function to get the maximum value among columns for each row.greatest correctly with imported function. df = df.withColumn('max_score', max(col('score1'), col('score2'))) uses max which is an aggregate function, not for row-wise max. df = df.withColumn('max_score', col('score1') if col('score1') > col('score2') else col('score2')) tries Python if-else which does not work on columns. df = df.withColumn('max_score', col('score1').max(col('score2'))) uses invalid syntax.Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.