Here's something I was very surprised to see:
import polars as pl
data = [
{"id": 1, "metadata": '{"role": "admin", "status": "active"}'},
{"id": 2, "metadata": '{"role": "user", "status": "inactive"}'},
{"id": 3, "metadata": '{"role": "guest", "status": "active"'}, # Malformed JSON (missing })
{"id": 4, "metadata": '{"role": "user", "status": "active"}'}
]
df = pl.DataFrame(data)
parsed_df = df.with_columns(
pl.col("metadata").str.json_decode(pl.Struct).alias("parsed_meta")
).show()This raises an exception:
ComputeError: error deserializing JSON: json parsing error: 'ExpectedObjectKey at character 115 ('{')'
This error occurred in the following expression:
col("metadata").str.json_decode()This even happens when the JSON is well formed, but the schema changes between rows:
import polars as pl
data = [
{"id": 1, "metadata": '{"role": "admin", "status": "active"}'},
{
"id": 2,
"metadata": '{"role": "user", "status": [42]}',
}, # Malformed data (status should be a string)
{"id": 3, "metadata": '{"role": "guest", "status": "active"}'},
{"id": 4, "metadata": '{"role": "user", "status": "active"}'},
]
df = pl.DataFrame(data)
parsed_df = df.with_columns(
pl.col("metadata")
.str.json_decode(
# We even specify a schema here
pl.Struct([pl.Field("role", pl.String), pl.Field("status", pl.String)])
)
.alias("parsed_meta")
).show()Also blows up:
ComputeError: error deserializing JSON: error deserializing value "Array([Static(U64(42))])" as string.
Try increasing `infer_schema_length` or specifying a schema.
This error occurred in the following expression:
col("metadata").str.json_decode()So the problem statement is - we have many rows and we want to decode JSON efficiently and also in such a way that we can be lenient about parse errors (and understand why they happened).
Alternatively, we could use
json_path_match (for example, pl.col("raw_json").str.json_path_match("$.user.name")) but this is highly inefficient, since you need to reparse the same JSON row if you want to extract more than 1 field.Here is where I scoured the internet to see what I was missing, since this doesn't seem like that much of a niche problem. But it looks like there's no popular solution here (happy to hear if I'm wrong, please DM me!).
So I (+ some AI agents) wrote polars-fastjson.
It takes a different approach: provide the target schema once, then project each JSON string into a typed
Struct and allow for leniency (don't simply raise an exception if something goes wrong).import polars as pl
from polars_fastjson import fastjson_decode
schema = {
"id": pl.String,
"score": pl.Float64,
"tags": pl.List(pl.String),
}
parsed = df.with_columns(
fastjson_decode(pl.col("payload"), schema=schema).alias("parsed")
)By default:
- Malformed rows become null structs
- A bad "leaf" field becomes null and valid sibling fields are retained
- Compatible values can be coerced (my personal choice here, can be configured)
In addition:
- Strict modes are available when a pipeline should fail instead
- Nested structs and lists are supported
- You can supply a schema by a Polars dtype, a dictionary, a dataclass, a
TypedDict, or a Pydantic model. - You can ask to emit diagnostics which allows for a summary to understand why rows were nulled without logging each one (huge I/O strain, and also noisy)
The performance has been promising in local benchmarks, including at million-row scale, and appears to scale roughly linearly with the number of rows. The benchmark is included in the repository if you want to try it yourself.