面试题与答案

PySpark is the Python API for Apache Spark, a fast and general-purpose cluster computing system.

Example:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('example').getOrCreate()

'groupBy' is used to group the data based on one or more columns. It is often followed by aggregation functions to perform operations on each group.

Example:

grouped_data = df.groupBy('Category').agg({'Price': 'mean'})

SparkSession is the entry point to any PySpark functionality. It is used to create DataFrames, register DataFrames as tables, and execute SQL queries.

Example:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('example').getOrCreate()

The 'collect' action retrieves all elements of a distributed dataset (RDD or DataFrame) and brings them to the driver program.

Example:

data = df.collect()

You can use the 'union' method to combine two DataFrames with the same schema.

Example:

result = df1.union(df2)

'groupBy' is used to group the data based on one or more columns. It is often followed by aggregation functions to perform operations on each group.

Example:

grouped_data = df.groupBy('Category').agg({'Price': 'mean'})

You can use the 'createOrReplaceTempView' method to create a temporary view from a PySpark DataFrame.

Example:

df.createOrReplaceTempView('temp_view')

'OrderBy' is used to sort the rows of a DataFrame based on one or more columns.

Example:

result = df.orderBy('column')