A quick guide to working with NDJSON feeds

Data source:

https://myfusion.eu/api/products/full-export

File format: NDJSON (newline-delimited JSON). Important!!!! Use this file only for first products import, after - use full integration https://myfusion.eu/api

This means:

  • 1 row = 1 product
  • each line is a separate JSON object
  • the file should be read as a stream, not loaded completely into memory

Authorization

A Bearer Token is used.

Request header:
Authorization: Bearer YOUR_API_TOKEN

Why is NDJSON better than regular JSON?

  • fast reading of large files
  • it is possible to process in units
  • minimal memory consumption
  • suitable for importing very large amounts of data

The most important principle

Wrong:

 file_get_contents(...) json_decode(...)

This will load the entire file into memory.

Correct: read the file line by line or through a stream.

PHP example

  [ 'method' => 'GET', 'header' => "Authorization: Bearer {$token}\r\n" ] ]); $handle = fopen($url, 'r', false, $context); while (($line = fgets($handle)) !== false) { $row = json_decode($line, true); // produkta apstrāde echo $row['productId'] . PHP_EOL; } fclose($handle); ?>

Recommended imports

For high speed, it is recommended:

  • read data as a stream
  • process records in batches
  • use batch insert or batch update
  • do not store each record in a separate query unless necessary

Data updates

Use descriptionUpdateDate field.

This shows the date the description was last modified and can only be used to update modified products.