PHP Classes

How to Implement a PHP json_decode Equivalent Function that Can Parse Large JSON Files Without Using Too Much Memory Using the Package Lazy JSON: Load and parse JSON files progressively

Recommend this page to a friend!
  Info   Documentation   Demos   View files Files   Install with Composer Install with Composer   Download Download   Reputation   Support forum   Blog    
Last Updated Ratings Unique User Downloads Download Rankings
2025-02-07 (Yesterday) RSS 2.0 feedNot yet rated by the usersTotal: Not yet counted Not yet ranked
Version License PHP version Categories
lazy-json 1.0MIT/X Consortium ...8Data types, Parsers, PHP 7
Description 

Author

This package can load and parse JSON files progressively.

It can take the name of a JSON file and start parsing it without loading the whole file at once.

The package provides classes for each type of element contained in a JSON document.

The parser can load the remaining parts of the JSON file and return objects of the classes with the details of each value element found in the file.

Picture of Rubens Takiguti Ribeiro
  Performance   Level  
Name: Rubens Takiguti Ribeiro <contact>
Classes: 8 packages by
Country: Brazil Brazil
Innovation award
Innovation award
Nominee: 4x

Documentation

LazyJson Usage

This document explains how to use the LazyJson library. If you are interested in the API specs of the classes, check API

Loading a JSON file

To construct a lazy object from a JSON file, open the file using SplFileObject with mode r (read) and use it in the static method LazyJson\JsonElement::load(). Example:

$file = new \SplFileObject('/path/to/my/file.json', 'r');

$lazyObj = \LazyJson\JsonElement::load($file);

The method load will detect the type of the root element of the JSON file (using an heuristic) and return an appropriate instance of a child class derived from LazyJson\JsonElement. The object will be an instance of one of these wrapper classes:

  • `LazyJson::NullElement`
  • `LazyJson::BooleanElement`
  • `LazyJson::NumberElement`
  • `LazyJson::StringElement`
  • `LazyJson::ArrayElement`
  • `LazyJson::ObjectElement`

Note: the method load will not parse the entire file imediatelly. It will just read the minimum number of bytes to detect the type of the root element of the JSON file. For example, if the file starts with the byte (char) t, it assumes the file content has a boolean value with the value true, since it is the unique valid JSON value that starts with the letter t. If it starts with [, it assumes the file has an array. And so on.

Note: the mentioned classes are used as wrapper classes to traverse the JSON hierarchically. This way, if the root element is an array or an object, it will work like a tree, where the root element can traverse the child elements and these child elements will always be an instance of LazyJson\JsonElement and can be traversed too (if they are arrays or objects).

Detecting the type of the element

To detect the type of a LazyJSON element, just use PHP's instanceof. Example:

$lazyObj = \LazyJson\JsonElement::load($file);

if ($lazyObj instanceof \LazyJson\ObjectElement) {
    ...
}

Traversing the JSON tree

Usually the root element of a JSON file will be an array or an object. If the method load detects the root element is not an array nor an object, you will just be able to manipulate a simple element. However, if the root element is an array or an object, these elements can be traversed to fetch their child elements.

There are 2 ways to traverse an array or object: (1) iterating over the child elements; or (2) accessing a specific random child.

Iterating over child elements

The classes LazyJson\ArrayElement and LazyJson\ObjectElement implements the PHP interface IteratorAggregate. This way, you can iterate over the elements:

$lazyObj = \LazyJson\JsonElement::load($file);

foreach ($lazyObj as $key => $value) {
    ...
}

If you are traversing an array, the $key will be an integer (starting from 0), representing the element position in the array. The $value will be an object of a wrapper class.

If you are traversing an object, the $key will be a native PHP string (a decoded value), representing a property name. The $value will be an object of a wrapper class.

Note: if you store the iterator in a variable, you cannot iterate over the same iterator instance twice. That means the code bellow will not work.

$iterator = $lazyObj->getIterator();

foreach ($iterator as $key => $value) {
    ...
}

// This second loop will fail, since the iterator has already been traversed
foreach ($iterator as $key => $value) {
    ...
}

Instead, you can just call the method twice:

foreach ($lazyObj->getIterator() as $key => $value) {
    ...
}

foreach ($lazyObj->getIterator() as $key => $value) {
    ...
}

Accessing random child elements

The class LazyJson\ArrayElement implements the PHP interfaces ArrayAccess and Countable. That means the objects of this class behaves like a regular PHP array, with few differences.

To access a random child element of an array, you just need to use [] to access the desired position.

The class LazyJson\ObjectElement implements the PHP interfaces ArrayAccess and Countable. Additionally, it implements the magic methods __get and __isset. That means the objects of this class behaves like a regular PHP object, with few differences.

To access a random child element of an object, you just need to use -> or [] to access the child property.

/* Assuming the file content is:
 * {
 *   "x": [
 *     1,
 *     2,
 *     3
 *   ]
 * }
 */
$lazyObj = \LazyJson\JsonElement::load($file);

$childElement = $lazyObj->x[0];

In the example above, the $childElement will be an object of type LazyJson\NumberElement, that wraps the value 1.

As mentioned, you can also access object properties using []. Example:

$childElement = $lazyObj['x'][0];

Counting the number of elements:

$totalPropertiesOfObject = count($lazyObj);
$totalElementsOfArray = count($lazyObj->x);

Checking if a position of the array or a property of the object exists:

$propertyExists = isset($lazyObj->x);
$positionExists = isset($lazyObj->x[0]);

Note: you cannot set or unset values from the wrapped classes. They were planned to be read-only.

Decoding a value

As you could see, when you traverse a JSON tree, accessing objects and arrays, you will always reach to a desired target element that will be represented by a wrapper class. To decode the value of an element wrapped by a wrapper class, you just need to call the method getDecodedValue. It will return the raw PHP value that corresponds to the read JSON value.

This method receives a boolean parameter to indicate whether the result must use associative arrays or objects (of class stdClass), just like PHP's function json_decode.

$decodedValue = $lazyObj->x[0]->getDecodedValue();

Note: if you decode a large object, array or string, it will consume a lot of memory, just like json_decode would do. Considering you are using this class to avoid consuming a lot o memory, it might be useful to not decode large elements. If you know an element is large, you can traverse it using the iterator.

Iterating over a string

Since a JSON can contain large strings too, it is useful to have a way to iterate over these strings to avoid consuming a lot of memory. For this reason, the class LazyJson\StringElement also implements the IteratorAggregate.

The difference between the iterator returned by StringElement and the iterator returned by ArrayElement and ObjectElement is that StringElement's iterator returns a PHP raw string on each iteration (a single unicode symbol per iteration). This way, you can decide what to do with the string content while you traverse it. For exemple, you can store the decoded value in a temporary file:

// $considering $lazyObj is a LazyJson\StringElement

$tempFilename = tempnam(sys_get_temp_dir(), 'temp-file-');
$tempFile = fopen($tempFilename, 'w');
foreach ($lazyObj as $unicodeSymbol) {
    fwrite($tempFile, $unicodeSymbol);
}
fclose($tempFile);

Maybe your string is encoded with an algorithm like Bas64, so you can decide to decode it on-demand and send the decoded value to a destiny (incrementally), instead of reading the entire string, put it in the memory and use base64_decode over it.


Details

LazyJson

Description

LazyJson is a PHP library that allows you to navigate a JSON file through a lazy-loaded, object-oriented interface.

Unlike json_decode, which loads the entire JSON file into memory at once, this library parses only the necessary elements on demand, based on the operations you perform during navigation.

Its primary goal is to provide a memory-efficient way to access JSON data rather than modifying it. While it can be used to validate a JSON structure, it assumes the JSON is well-formed and only parses elements as needed to return the requested data.

Advantages:

  • Memory-efficient: Optimized for large JSON files, supporting arrays, objects, and strings, with minimal memory usage during navigation.
  • Intuitive: Navigate the JSON as if it were parsed by `json_decode`, without needing to load the entire structure into memory.
  • Zero dependencies: No third-party libraries required, only standard PHP extensions.
  • PHP 8.1+ compatible.
  • High test coverage: Rigorous unit tests ensure reliability.
  • MIT License: You have the freedom to use, modify, and distribute it in both open-source and proprietary projects, with no warranty.

Use cases

For a JSON file with:

... a large array, you can:

  1. Access a specific element without loading the entire array into memory.
  2. Iterate over the array, keeping only one element in memory at a time.
  3. Count the elements without loading the whole array.

... a large object, you can:

  1. Access specific properties without parsing the entire object.
  2. Iterate over the object, keeping only one key-value pair in memory at a time.

... a large string, you can:

  1. Iterate over the string, processing one UTF-8 character at a time (e.g., stream it to an HTTP response, decode Base64, or save to a file).

Requirements

  • PHP 8.1 or higher
  • ext-ctype
  • ext-json
  • ext-spl

Installation

To use this library, you can install it via Composer:

$ composer require lazy-json/lazy-json

If you use PHP 7.4, you can try:

$ composer require 'lazy-json/lazy-json:dev-php7'

Documentation / API

To know how to use it, check the page Usage.

To have details about public methods, access the API spec.

You might also want to see the examples.


  sample.jsonExternal page  

Open in a separate window

  Files folder image Files (34)  
File Role Description
Files folder image.github (1 file, 1 directory)
Files folder imagedocs (4 files)
Files folder imageexamples (3 files)
Files folder imagesrc (7 files)
Files folder imagetests (1 directory)
Accessible without login Plain text file .dockerignore Data Auxiliary data
Accessible without login Plain text file .env.example Data Auxiliary data
Accessible without login Plain text file composer.json Data Auxiliary data
Accessible without login Plain text file composer.lock Data Auxiliary data
Accessible without login Plain text file docker-compose.yml Data Auxiliary data
Accessible without login Plain text file Dockerfile Data Auxiliary data
Accessible without login Plain text file LICENSE Lic. License text
Accessible without login Plain text file phpstan.neon Data Auxiliary data
Accessible without login Plain text file phpunit.xml Data Auxiliary data
Accessible without login Plain text file README.md Doc. Documentation

  Files folder image Files (34)  /  .github  
File Role Description
Files folder imageworkflows (1 file)
  Accessible without login Plain text file FUNDING.yml Data Auxiliary data

  Files folder image Files (34)  /  .github  /  workflows  
File Role Description
  Accessible without login Plain text file ci.yml Data Auxiliary data

  Files folder image Files (34)  /  docs  
File Role Description
  Accessible without login Plain text file API.md Doc. Documentation
  Accessible without login Plain text file CONTRIBUTING.md Doc. Documentation
  Accessible without login Plain text file index.md Doc. Documentation
  Accessible without login Plain text file Usage.md Doc. Documentation

  Files folder image Files (34)  /  examples  
File Role Description
  Accessible without login Plain text file basic.php Example Example script
  Accessible without login Plain text file full.php Example Example script
  Accessible without login Plain text file sample.json Data Auxiliary data

  Files folder image Files (34)  /  src  
File Role Description
  Plain text file ArrayElement.php Class Class source
  Plain text file BooleanElement.php Class Class source
  Plain text file JsonElement.php Class Class source
  Plain text file NullElement.php Class Class source
  Plain text file NumberElement.php Class Class source
  Plain text file ObjectElement.php Class Class source
  Plain text file StringElement.php Class Class source

  Files folder image Files (34)  /  tests  
File Role Description
Files folder imageunit (7 files, 1 directory)

  Files folder image Files (34)  /  tests  /  unit  
File Role Description
Files folder imageFixtures (1 file)
  Plain text file ArrayElementTest.php Class Class source
  Plain text file BooleanElementTest.php Class Class source
  Plain text file JsonElementTest.php Class Class source
  Plain text file NullElementTest.php Class Class source
  Plain text file NumberElementTest.php Class Class source
  Plain text file ObjectElementTest.php Class Class source
  Plain text file StringElementTest.php Class Class source

  Files folder image Files (34)  /  tests  /  unit  /  Fixtures  
File Role Description
  Plain text file TempFileHelper.php Class Class source

The PHP Classes site has supported package installation using the Composer tool since 2013, as you may verify by reading this instructions page.
Install with Composer Install with Composer
 Version Control Unique User Downloads  
 100%
Total:0
This week:0