PySpark Show
09.28.2021
Intro
The show function allows us to preview a data frame. The show method provides us with a few options to edit the output. In this article, we will learn how to use the PySpark show function.
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()from datetime import datetime, date
import pandas as pd
from pyspark.sql import Row
df = spark.createDataFrame([
Row(amount = 20000, month = 'jan', date = datetime(2000, 1, 1, 12, 0), desc = "a very very very long description"),
Row(amount = 40000, month = 'feb', date = datetime(2000, 2, 1, 12, 0), desc = "a very very very long description"),
Row(amount = 50000, month = 'mar', date = datetime(2000, 3, 1, 12, 0), desc = "a very very very long description")
])
df.show()+------+-----+-------------------+--------------------+
|amount|month| date| desc|
+------+-----+-------------------+--------------------+
| 20000| jan|2000-01-01 12:00:00|a very very very ...|
| 40000| feb|2000-02-01 12:00:00|a very very very ...|
| 50000| mar|2000-03-01 12:00:00|a very very very ...|
+------+-----+-------------------+--------------------+We can specify the number of row to display using the n named parameter.
df.show(n=2)+------+-----+-------------------+--------------------+
|amount|month| date| desc|
+------+-----+-------------------+--------------------+
| 20000| jan|2000-01-01 12:00:00|a very very very ...|
| 40000| feb|2000-02-01 12:00:00|a very very very ...|
+------+-----+-------------------+--------------------+
only showing top 2 rowsNotice above that our description column has been cut off. But default, pyspark will truncate this data. We can change that using the truncate named parameter.
df.show(truncate = False)+------+-----+-------------------+---------------------------------+
|amount|month|date |desc |
+------+-----+-------------------+---------------------------------+
|20000 |jan |2000-01-01 12:00:00|a very very very long description|
|40000 |feb |2000-02-01 12:00:00|a very very very long description|
|50000 |mar |2000-03-01 12:00:00|a very very very long description|
+------+-----+-------------------+---------------------------------+We can also specificy the length to truncate by passing a number to the truncate named parameter.
df.show(truncate = 20)+------+-----+-------------------+--------------------+
|amount|month| date| desc|
+------+-----+-------------------+--------------------+
| 20000| jan|2000-01-01 12:00:00|a very very very ...|
| 40000| feb|2000-02-01 12:00:00|a very very very ...|
| 50000| mar|2000-03-01 12:00:00|a very very very ...|
+------+-----+-------------------+--------------------+The final option we can do is to display the dataframe vertically using the vertical parameter.
df.show(vertical = True)-RECORD 0----------------------
amount | 20000
month | jan
date | 2000-01-01 12:00:00
desc | a very very very ...
-RECORD 1----------------------
amount | 40000
month | feb
date | 2000-02-01 12:00:00
desc | a very very very ...
-RECORD 2----------------------
amount | 50000
month | mar
date | 2000-03-01 12:00:00
desc | a very very very ... You can also combine all of these parameters.
df.show(n = 2, truncate = 30, vertical = True)-RECORD 0--------------------------------
amount | 20000
month | jan
date | 2000-01-01 12:00:00
desc | a very very very long descr...
-RECORD 1--------------------------------
amount | 40000
month | feb
date | 2000-02-01 12:00:00
desc | a very very very long descr...
only showing top 2 rows