Read & Write JSON in Python

The JavaScript Object Notation (JSON) format is ubiquitous for data interchange in modern systems, from RESTful APIs to configuration files. Python's…

The JavaScript Object Notation (JSON) format is ubiquitous for data interchange in modern systems, from RESTful APIs to configuration files. Python's standard library provides robust and efficient tools for working with JSON data, enabling seamless conversion between Python objects and JSON strings or files. Understanding the core functionalities of the json module is essential for any developer interacting with external services or managing structured data.

This article details how to effectively read from and write to JSON files and strings using Python's built-in json module. We will cover basic serialization and deserialization, handle common data types, explore advanced formatting options, and address potential pitfalls.

Basic Serialization and Deserialization

The json module offers two primary functions for serialization (Python object to JSON) and two for deserialization (JSON to Python object).

json.dump() and json.load() for Files

For directly working with file-like objects, json.dump() and json.load() are the go-to functions. They handle file I/O operations internally, making them convenient for reading from or writing to JSON files.

import json

# Sample Python dictionary
python_data = {
    "name": "Alice",
    "age": 30,
    "isStudent": False,
    "courses": ["Math", "Physics"],
    "address": {
        "street": "123 Main St",
        "city": "Anytown"
    },
    "grades": None
}

# --- Writing to a JSON file ---
output_file_path = "output.json"
try:
    with open(output_file_path, "w", encoding="utf-8") as json_file:
        json.dump(python_data, json_file, indent=4) # 'indent=4' for pretty-printing
    print(f"Data successfully written to {output_file_path}")
except IOError as e:
    print(f"Error writing to file {output_file_path}: {e}")

# --- Reading from a JSON file ---
input_file_path = "output.json" # Using the file we just wrote
try:
    with open(input_file_path, "r", encoding="utf-8") as json_file:
        loaded_data = json.load(json_file)
    print(f"\nData successfully loaded from {input_file_path}:")
    print(loaded_data)
    print(f"Type of loaded_data: {type(loaded_data)}")
except FileNotFoundError:
    print(f"Error: File not found at {input_file_path}")
except json.JSONDecodeError as e:
    print(f"Error decoding JSON from {input_file_path}: {e}")

Key parameters:

• obj (for dump): The Python object to be serialized.
• fp (for dump/load): A file-like object with .write() (for dump) or .read() (for load) method.
• indent (for dump): If provided, JSON output will be pretty-printed with the specified indentation level. Common values are 2 or 4 spaces. Without indent, the JSON will be written on a single line, minimizing file size.
• encoding="utf-8": It's good practice to explicitly specify UTF-8 for robust handling of various characters.

json.dumps() and json.loads() for Strings

When you need to work with JSON data as a string (e.g., from an API response, for logging, or passing between functions), json.dumps() and json.loads() are appropriate.

import json

# Sample Python dictionary
python_dict = {"id": 101, "status": "active", "tags": ["premium", "new"]}

# --- Serializing to a JSON string ---
json_string = json.dumps(python_dict, sort_keys=True, indent=2)
print(f"\nSerialized JSON string:\n{json_string}")
print(f"Type of json_string: {type(json_string)}")

# --- Deserializing from a JSON string ---
another_json_string = '{"product_name": "Laptop", "price": 1200.50, "in_stock": true}'
loaded_dict = json.loads(another_json_string)
print(f"\nDeserialized Python dictionary:\n{loaded_dict}")
print(f"Type of loaded_dict: {type(loaded_dict)}")

# Example with a list of dictionaries
json_list_string = '[{"item": "apple", "quantity": 10}, {"item": "banana", "quantity": 5}]'
loaded_list = json.loads(json_list_string)
print(f"\nDeserialized list of dictionaries:\n{loaded_list}")

Key parameters:

• obj (for dumps): The Python object to serialize.
• s (for loads): The JSON string to deserialize.
• sort_keys (for dumps): If True, output of dictionaries will be sorted by key. This can be useful for diffing JSON outputs or ensuring consistent order.

Python-JSON Type Conversion Chart

The json module performs a straightforward mapping between Python types and JSON types:

It's important to note that Python tuples are serialized as JSON arrays, and deserialized back into Python lists. If you need strict tuple preservation, you would need custom serialization logic.

Handling Custom Objects (Serialization)

By default, json.dumps() and json.dump() cannot serialize custom Python objects directly, as they don't know how to convert instance attributes into JSON. Attempting to do so will raise a TypeError.

import json

class User:
    def __init__(self, user_id, username):
        self.user_id = user_id
        self.username = username

# user_obj = User(1, "john_doe")
# json.dumps(user_obj) # This would raise TypeError: Object of type User is not JSON serializable

To serialize custom objects, you can provide a custom encoder or implement a default method.

Method 1: Using a default function

The default parameter of json.dumps()/json.dump() accepts a function that will be called for objects that aren't serializable by the default encoder. This function should return a JSON-serializable representation of the object or raise a TypeError.

import json

class User:
    def __init__(self, user_id, username):
        self.user_id = user_id
        self.username = username

    def __repr__(self):
        return f"User(id={self.user_id}, username='{self.username}')"

def custom_encoder(obj):
    if isinstance(obj, User):
        return {"_type": "User", "user_id": obj.user_id, "username": obj.username}
    # For other types that default encoder can't handle, raise TypeError
    raise TypeError(f"Object of type {obj.__class__.__name__} is not JSON serializable")

user_obj = User(10, "jane_doe")
json_output = json.dumps(user_obj, default=custom_encoder, indent=2)
print(f"\nSerialized custom object:\n{json_output}")

# Output:
# {
#   "_type": "User",
#   "user_id": 10,
#   "username": "jane_doe"
# }

Method 2: Subclassing json.JSONEncoder

For more complex or reusable custom serialization logic, subclassing json.JSONEncoder is a cleaner approach.

import json

class User:
    def __init__(self, user_id, username):
        self.user_id = user_id
        self.username = username

class CustomUserEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, User):
            return {"_type": "User", "user_id": obj.user_id, "username": obj.username}
        # Let the base class default method raise the TypeError for other types
        return json.JSONEncoder.default(self, obj)

user_obj = User(20, "peter_pan")
json_output_encoder = json.dumps(user_obj, cls=CustomUserEncoder, indent=2)
print(f"\nSerialized custom object with custom encoder:\n{json_output_encoder}")

Handling Custom Objects (Deserialization)

Deserializing JSON back into custom Python objects requires a custom object hook. The object_hook parameter of json.loads()/json.load() accepts a function that will be called with the result of every JSON object decoded (a Python dict).

import json

class User:
    def __init__(self, user_id, username):
        self.user_id = user_id
        self.username = username

    def __eq__(self, other):
        if not isinstance(other, User):
            return NotImplemented
        return self.user_id == other.user_id and self.username == other.username

    def __repr__(self):
        return f"User(user_id={self.user_id}, username='{self.username}')"

def custom_decoder_hook(dct):
    if "_type" in dct and dct["_type"] == "User":
        return User(dct["user_id"], dct["username"])
    return dct # Return the dict as is if it's not a custom type we handle

json_string_with_user = """
{
  "event_id": "xyz123",
  "actor": {
    "_type": "User",
    "user_id": 30,
    "username": "sara_smith"
  },
  "action": "login"
}
"""

loaded_data = json.loads(json_string_with_user, object_hook=custom_decoder_hook)
print(f"\nLoaded data with custom object:\n{loaded_data}")
print(f"Type of loaded_data['actor']: {type(loaded_data['actor'])}")
print(f"Is loaded_data['actor'] an instance of User? {isinstance(loaded_data['actor'], User)}")

# Verify the object attributes
if isinstance(loaded_data['actor'], User):
    assert loaded_data['actor'].user_id == 30
    assert loaded_data['actor'].username == "sara_smith"
    print("User object attributes verified successfully.")

The object_hook function receives a dictionary. It should inspect the dictionary (e.g., check for a special _type key) and if it recognizes a custom object's representation, convert it back into an instance of that object. Otherwise, it should return the dictionary unchanged.

Common Pitfalls and Troubleshooting

• json.JSONDecodeError: This error indicates malformed JSON syntax. Common culprits include:
  • Using single quotes instead of double quotes for keys or string values.
  • Trailing commas in lists or objects (not allowed in strict JSON, though some parsers are lenient).
  • Missing commas between key-value pairs or list elements.
  • Unescaped special characters within strings.
  Validate your JSON string using an online validator or a linter if you suspect syntax issues.
• TypeError: Object of type X is not JSON serializable: As discussed, this occurs when you try to serialize Python objects that the json module doesn't know how to convert (e.g., datetime objects, custom class instances). Use default or a custom JSONEncoder.
• Incorrect file modes: Always open files for writing with "w" (or "a" for appending) and for reading with "r". Using the wrong mode can lead to IOError or unexpected behavior.
• Encoding issues: While UTF-8 is the default and usually sufficient, explicitly specifying encoding="utf-8" for open() calls is a good habit, especially when dealing with international characters. Without it, the system's default encoding might be used, leading to unexpected errors or corrupted data.
• Large JSON files: For extremely large JSON files that don't fit into memory, json.load() can consume significant RAM. Consider using streaming JSON parsers (e.g., ijson, json-stream libraries) for such scenarios.
• None vs. Empty String: Remember that Python's None maps to JSON's null, not an empty string (""). Ensure your data types align with expectations when exchanging data.