<<

NAME

ProductOpener::GS1 - convert data from GS1 Global Data Synchronization Network (GDSN) to the Open Food Facts format.

SYNOPSIS

DESCRIPTION

This module converts GDSN data that has previously been converted to JSON (either from XML with xml2json (e.g. Equadis data), or from JSON provided by a GDSN partner (e.g. CodeOnline) to the Open Food Facts CSV format.

The conversion is configured through the %gs1_to_off structure to indicate which source field maps to which target field.

And the %gs1_maps translate the GS1 specific identifiers (e.g. for allergens or units) to OFF identifiers.

GS1 MAPS

GS1 uses many different codes for allergens, packaging etc. We need to map the different values to the values we use in the taxonomies.

The GS1 standards are evolving, so new values may be introduced over time.

%unknown_entries_in_gs1_maps

Used to keep track of values we encounter in GS1 data for which we do not have a mapping.

%gs1_maps

Maps from GS1 to OFF

%gs1_message_to_off

Defines the structure of the GS1 message data and how to extract the fields useful to create a message confirmation.

%gs1_product_to_off

Defines the structure of the GS1 product data and how it maps to the OFF data.

FUNCTIONS

init_csv_fields ()

%seen_fields and @fields are used to output the fields in the order of the GS1 to OFF mapping configuration

assign_field ( $results_ref $target_field $target_value)

Used to assign a value to a field, and keep track of the order of the fields we are matching, so that we can output the fields in the same order when we export a CSV.

gs1_to_off ($gs1_to_off_ref, $json_ref, $results_ref)

Recursive function to go through all first level keys of the $gs1_to_off_ref mapping. All values that can be assigned at that level are assigned, and if we need to go into a nested level, the function calls itself again.

Arguments

$gs1_to_off_ref - Mapping configuration from GS1 to off for the current level

$json_ref - JSON structure for the current level

$results_ref - Hash of key / value pairs to store the complete output of the mapping

The same hash reference is passed to recursive calls to the gs1_to_off function.

convert_single_text_property_to_direct_value ($json )

There are different ways to convert a XML document to a JSON data structure.

Historically, we used nodejs xml2json module to convert the GS1 XML to JSON.

Then we added support for CodeOnline JSON exports which used slightly different conversions.

In order to remove the dependency on nodejs, we are now supporting Perl's XML:XML2JSON module that results in different structures.

This function is a recursive function to make the output of Perl XML::XML2JSON similar to nodejs xml2json, as the GS1 module expects this format.

Difference:

XML2JSON creates a hash for simple text values. Text values of tags are converted to $t properties. e.g. <gtin>03449862093657</gtin>

becomes:

gtin: { $t: "03449865355608" },

This function converts those hashes with one single $t scalar values to a direct value.

gtin: "03449865355608"

Arguments

$json_ref Reference to a decoded JSON structure

convert_gs1_json_message_to_off_products_csv_fields ($json, $products_ref, $messages_ref)

Thus function converts the data for one or more products in the GS1 format converted to JSON. GS1 format is in XML, it needs to be transformed to JSON with xml2json first. In some cases, the conversion to JSON has already be done by a third party (e.g. the CodeOnline database from GS1 France).

Note: This function is recursive if there are child products.

One GS1 message can include 1 or more products, typically products that contain other products (e.g. a pallet of cartons of products).

Arguments

$json_ref Reference to a decoded JSON structure

$product_ref - Reference to an array of product data

Each product data will be added as one element (a hash ref) of the product data array.

For each product, the key of the hash is the name of the OFF csv field, and it is associated with the corresponding value for the product.

$messages_ref - Reference to an array of GS1 messages data

Each message will be added as one element (a hash ref) of the messages data array.

read_gs1_json_file ($json_file, $products_ref, $messages_ref)

Read a GS1 message file in json format, convert the included products in the OFF format, and store the resulting products in the $products_ref array

The encapsulating GS1 message is added to the $messages_ref array

Arguments

input json file path and name $json_file

reference to output products array $products_ref

reference to output messages array $messages_ref

generate_gs1_confirmation_message ($notification_message_ref, $timestamp)

GS1 data pools (catalogs) send us GSDN Catalogue Item Notification (CIN) which are messages that contain the data for 1 product (and possibly sub-products).

The GS1 standard offers data recipient (such as Open Food Facts) to send back Catalogue Item Confirmation (CIC) messages to acknowledge the notification and give its status.

This function generates the CIC message corresponding to a CIN message.

See https://www.gs1.org/docs/gdsn/tiig/3_1/GDSN_Trade_Item_Implementation_Guide.pdf for more details.

Arguments

reference to the notification message (as parsed by convert_gs1_json_message_to_off_products_csv)

timestamp

The current time is passed as a parameter to the function. This is so that we can generate test confirmation messages which don't have a different content every time we run them.

write_off_csv_file ($csv_file, $products_ref)

Write all product data from the $products_ref array to a CSV file in OFF format.

Arguments

output CSV file path and name

reference to output products array $products_ref

print_unknown_entries_in_gs1_maps ()

Prints the entries for GS1 data types for which we do not have a corresponding OFF match, ordered by the number of occurrences in the GS1 data

convert_gs1_xml_file_to_json ($xml_file, $json_file)

Convert a GS1 XML file to a JSON file

<<