JSON to Spark Struct Converter

⚠️ You can trust that this tool runs everything locally in your browser. Your information is not stored or transmitted to any external server.

👉 Paste your JSON below and click the button to convert it to Spark StructType:

JSON to Spark Struct Converter is a web tool designed to streamline the process of converting JSON data into Spark structType code. With this tool, users can input a JSON object and receive the corresponding structType code as output, ready to be used in their Spark applications. This tool is beneficial for developers and data analysts working with Spark who need to convert JSON data to a format compatible with Spark’s distributed processing system.

Example: Using JSON to Spark Struct Converter for API Responses

If you need to call an API from Spark, receive a JSON response, and then convert that JSON to a StructType. This example shows how to accomplish this efficiently using the JSON to Spark Struct Converter tool.

👉 You can find the complete example in Databricks Notebook - Integrating API Calls with Spark .

Dummy DF

Let’s create a dummy DataFrame (DF) that simulates transaction data. This DataFrame contains user IDs, transaction IDs, product names, quantities, and total prices.

user_ids = list(range(1, 11)) 
transactions = [
    (user_id, f"TXN00{user_id}", f"Product {user_id}", user_id, 100 * user_id)
    for user_id in user_ids
]

schema = StructType([
    StructField("userId", IntegerType(), True),
    StructField("transaction_id", StringType(), True),
    StructField("product_name", StringType(), True),
    StructField("quantity", IntegerType(), True),
    StructField("total_price", IntegerType(), True)
])

transactions_df = spark.createDataFrame(transactions, schema)

display(transactions_df)

Define the Response Schema

To handle the API response, we need to define its schema. We will use the JSON to Spark Struct Converter to automatically generate the necessary schema. This tool helps convert JSON data into a Spark StructType, saving valuable time.

schemaResponse = StructType([
    StructField("id", IntegerType(), True),
    StructField("name", StringType(), True),
    StructField("username", StringType(), True),
    StructField("email", StringType(), True),
    StructField("address", StructType([
    StructField("street", StringType(), True),
    StructField("suite", StringType(), True),
    StructField("city", StringType(), True),
    StructField("zipcode", StringType(), True),
    StructField("geo", StructType([
        StructField("lat", FloatType(), True),
        StructField("lng", FloatType(), True)
        ]), True)
    ]), True),
    StructField("phone", StringType(), True),
    StructField("website", StringType(), True),
    StructField("company", StructType([
    StructField("name", StringType(), True),
    StructField("catchPhrase", StringType(), True),
    StructField("bs", StringType(), True)
    ]), True)
])

Define UDF

Next, define a User Defined Function (UDF) to call the API and parse the JSON response. This UDF will take a user ID as input, call the API, parse the JSON response, and convert it to the Spark StructType schema.

from pyspark.sql.functions import udf, col
import http
import json
def callApi(userId):
  conn = http.client.HTTPSConnection("jsonplaceholder.typicode.com")
  payload = ''
  headers = {
  'Content-Type': 'application/json'
  }
  conn.request("GET", f"/users/{userId}", payload, headers)
  res = conn.getresponse()
  data = res.read().decode("utf-8")
  jsondata = json.loads(data)
  
  return jsondata

udf_callApi = udf(callApi, schemaResponse)

Call UDF

Now, use the UDF to call the API for each row in the dummy DataFrame. The API response will be added as a new column in the DataFrame.

![Manipulate the Response](/images/tool-01.png)
result_df = transactions_df.withColumn("result_api", udf_callApi(col("userId")))

result_df.createOrReplaceTempView("results")

Manipulate the Response

With the API response available in the DataFrame, you can manipulate it as a Spark Struct. This allows you to perform various operations and queries on the nested JSON data.

%sql

select
    userId,
    result_api,
    result_api.email,
    result_api.company.name as companyName,
    * except (userId, result_api)
from results

Manipulate the Response

Example: Using JSON to Spark Struct Converter for API Responses#

Example: Using JSON to Spark Struct Converter for API Responses