How to setup the Dev environment (quick start guide)

This guide will allow you to rapidly build a ready-to-use development environment for Product Opener running in Docker. As an alternative to setting up your environment locally, follow the Gitpod how-to guide to instantly provision a ready-to-code development environment in the cloud.

First setup time estimate is ~10min with the following specs:

• 8 GB of RAM dedicated to Docker client
• 6 cores dedicated to Docker client
• 12 MB/s internet speed

1. Prerequisites

Docker is the easiest way to install the Open Food Facts server, play with it, and even modify the code.

Docker provides an isolated environment, very close to a Virtual Machine. This environment contains everything required to launch the Open Food Facts server. There is no need to install Perl, Perl modules, Nginx, or Apache separately.

NOTE: New to Perl? Check how to learn perl!

Installation steps:

• GIT version >= 2.25.0
• Install Docker CE

If you run e.g. Debian, don't forget to add your user to the docker group!

MacOS Prerequisites: Installing GNU grep for Full Regex Support

On macOS, the default version of grep is BSD grep, which does not support certain GNU-specific options—most notably, the -P flag for Perl-compatible regular expressions. If you encounter errors like:

grep: invalid option -- P
usage: grep [-abcdDEFGHhIiJLlMmnOopqRSsUVvwXxZz] [-A num] [-B num] [-C[num]] …

You can resolve the issue by installing GNU grep and prioritizing it in your shell's PATH.

Steps to Install GNU grep

1. Install GNU grep via Homebrew:

   brew install grep

2. Update Your PATH:

   Add the following line to your ~/.zshrc (or your shell’s configuration file):

   export PATH="$(brew --prefix grep)/libexec/gnubin:$PATH"

   This ensures that the GNU version (often available as ggrep) is used by default instead of BSD grep.

3. Apply the Changes:

   Restart your terminal or run:

   source ~/.zshrc

4. Verify Installed Version:

Run the following command to ensure that GNU grep is installed and properly configured:

grep --version

You should see an output similar to the example below, indicating that GNU grep (with Homebrew packaging) is active and supports the -P option via PCRE2:

grep (GNU grep) 3.11
Packaged by Homebrew
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others; see
<https://git.savannah.gnu.org/cgit/grep.git/tree/AUTHORS>.

grep -P uses PCRE2 10.44 2024-06-07

After completing these steps, running grep commands (for example, as part of make log) will use GNU grep with full support for options like -P.

Windows Prerequisites

When running with Windows, install Docker Desktop which will cover all of the above.

The Make tasks use a number of Linux commands, such as rm and nproc, so it is recommended to run Make commands from the Git Bash shell. In addition, the following need to be installed and included in the PATH:

• Make for Windows
• wget for windows (In order to download the full product database). If you want to download wget with the executable, copy the wget.exe file to C:/Windows/System32 and you are done.

The process of cloning the repository will create a number of symbolic links which require specific permissions under Windows. In order to do this you can use any one of these alternatives:

• Use an Administrative command prompt for all Git commands
• Completely disable UAC
• Specifically grant the Create symbolic links permission to your user

Make sure you also activated the Developer mode on your device.

Windows Subsystem for Linux (WSL) Prerequisites

Ensure that you have WSL installed on your Windows machine. For instructions on how to do so, you can follow Microsoft's guide to install WSL.

After successfully installing WSL, you need to set up your Linux distribution and install Docker on it:

• Go to the Microsoft Store and install Ubuntu as your Linux distribution
• Open Ubuntu and execute the commands/instructions specified in Installing Docker Engine on Ubuntu (**NOTE: this is NOT the same thing as Docker for Desktop)
• Restart your computer to ensure all changes take effect and WSL can properly integrate with Docker

Docker Desktop Prerequisite

If you use Docker Desktop:

• ensure you allow enough memory for your VMs (at least 4G)
• ensure you Enabled host networking (in Resources / Network)

2. Fork and clone the repository from GitHub

You must have a GitHub account and fork the project if you want to contribute to Open Food Facts development, but it’s not required if you just want to see how it works.

Cloning Open Food Facts server with the default options downloads 2.23 GiB (as of 2024-03). See Shallow Clone if this might be a problem for you.

Fork the repository

If you want to collaborate you will need to fork the repository. (See Creating a pull request from a fork in Github documentation for an explanation.)

Go to https://github.com/openfoodfacts/openfoodfacts-server and click the "Fork" button.

This will create a new repository in your GitHub account. eg. (for an hypothetical my-user-id user): https://github.com/my-user-id/openfoodfacts-server.

This is the url you have to use in the following instructions.

Very important: always work in new branches, never in "main", otherwise it will be difficult to sync your project with the upstream repository (the openfoodfacts original repository).

From time to time (before starting a new branch), it's a good idea to sync your fork, either using the github interface and then pulling "main" locally, or from the command line.

Clone

Choose your preferred way to clone, either:

On Windows:

If you are running Docker on Windows, please use the following git clone command:

# replace my-user-id with your user id if you forked the repository, otherwise use openfoodfacts
git clone -c core.symlinks=true https://github.com/my-user-id/openfoodfacts-server.git

or (if you want to use ssh)

# replace my-user-id with your user id if you forked the repository, otherwise use openfoodfacts
git clone -c core.symlinks=true git@github.com:my-user-id/openfoodfacts-server.git

On other systems:

# replace my-user-id with your user id if you forked the repository, otherwise use openfoodfacts
git clone git@github.com:my-user-id/openfoodfacts-server.git

or

# replace my-user-id with your user id if you forked the repository, otherwise use openfoodfacts
git clone https://github.com/my-user-id/openfoodfacts-server.git

Go to the cloned directory:

cd openfoodfacts-server/

Shallow Clone

A full clone of the openfoodfacts-server repository can consume a significant amount of your PC's resources. It will download over 2 GiB of data and take up over 3 GiB of drive space.

git clone --single-branch --depth=1 https://github.com/openfoodfacts/openfoodfacts-server.git

To save your PC's resources, consider using a shallow clone by only cloning the main branch combined with a shallow fetch. This reduces the downloaded data to approximately 333 MiB and the repo will only use around 350 MiB of drive space.

You will still be able to contribute PRs based on the main branch. However, you will not be able to view the full Git version history of any files, and you need to use an additional branch from GitHub, e.g., gh-pages, you can do it like this:

git remote set-branches --add origin gh-pages
git fetch --depth=1 origin gh-pages:gh-pages
git checkout gh-pages

Note: you can skip this step for the first setup since the default .env in the repo contains all the default values required to get started.

Before running the docker compose deployment, you can review and configure Product Opener's environment (.env file).

The .env file contains ProductOpener default settings:

The .env file also contains some useful Docker Compose variables:

• COMPOSE_PROJECT_NAME is the compose project name that sets the prefix to every container name. Do not update this unless you know what you're doing.
• COMPOSE_FILE is the ;-separated list of Docker compose files that are included in the deployment:
  • For a development-like environment, set it to docker-compose.yml;docker/dev.yml (default)
  • For a production-like environment, set it to docker-compose.yml;docker/prod.yml;docker/mongodb.yml
  • For more features, you can add:
    • docker/admin-uis.yml: add the Admin UIS container
    • docker/geolite2.yml: add the Geolite2 container
    • docker/perldb.yml: add the Perl debugger container
• COMPOSE_SEPARATOR is the separator used for COMPOSE_FILE.

Note: Instead of modifying .env (and risk committing it inadvertently), you can also set needed variables in your shell; they will override .env values. Consider creating a .envrc file that you source each time you need to work on the project. On linux and macOS, you can automatically do it if you use direnv.

3. Build your dev environment

From the repository root, run:

make dev

Parallel execution with make

By default, GNU make runs one recipe at a time, sequentially. You may choose to run make dev with multiple jobs in parallel to speed up the build process, particularly operations that involve many network calls (such as building dependencies and downloading assets). Use either:

make dev --jobs=32

or set the MAKEFLAGS environment variable before running make dev:

export MAKEFLAGS=--jobs=32
make dev

Choosing a higher job count than the number of CPU cores is safe for Product Opener because most of the time the tasks are network-bound rather than CPU-bound. A good starting point is 2-4 times your CPU core count, but monitor system performance and adjust accordingly to match your available bandwidth and system responsiveness.

Note:

If you are using Windows, you may encounter issues regarding this command. Take a look at the Troubleshooting section further in this tutorial.

Note:

If docker complains about

ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

It can be solved by adding {"base": "172.80.0.0/16","size": 24}, {"base": "172.90.0.0/16","size": 24} to default-address-pools in /etc/docker/daemon.json and then restarting the docker daemon. Credits to https://theorangeone.net/posts/increase-docker-ip-space/ for this solution.

The command will run 2 subcommands:

• make up: Build and run containers from the local directory and bind local code files, so that you do not have to rebuild everytime.
• make import_sample_data: Load sample data into mongodb container (~100 products).

Notes:

• The first build can take between 10 and 30 minutes depending on your machine and internet connection (broadband connection heavily recommended, as this will download Docker base images, install Debian and Perl modules in preparation of the final container image).

  • Note: see the "Parallel execution with make" section above for guidance on running make dev with multiple jobs in parallel.

• You might not immediately see the test products: create an account, login, and they should appear.

• For a full description of available make targets, see Docker / Makefile commands

Hosts file:

Since the default PRODUCT_OPENER_DOMAIN in the .env file is set to openfoodfacts.localhost, add the following to your hosts file (Windows: C:\Windows\System32\drivers\etc\hosts; Linux/MacOSX: /etc/hosts):

127.0.0.1 world.openfoodfacts.localhost fr.openfoodfacts.localhost static.openfoodfacts.localhost ssl-api.openfoodfacts.localhost fr-en.openfoodfacts.localhost

You're done! Check http://openfoodfacts.localhost/.

Going further

To learn more about developing with Docker, see the Docker developer's guide.

To have all site pages on your dev instance, see Using pages from openfoodfacts-web

Using Repl offers you a way to play with perl.

Specific notes are provided on applying AGRIBALYSE updates to support the Ecoscore calculation.

Visual Studio Code

WARNING: Devcontainer support is currently experimental. It's recommended to run the normal docker commands before, and stop the containers: make dev down. Note that make dev, make test, and so on may currently conflict with the devcontainer.

This repository comes with a configuration for Visual Studio Code (VS Code) development containers (devcontainer). This enables some Perl support in VS Code without the need to install the correct Perl version and modules on your local machine.

To use the devcontainer, install prerequisites, clone the repository from GitHub, and (optionally) review Product Opener's environment. Additionally, install Visual Studio Code. VS Code will automatically recommend some extensions, but if you don't want to install all of them, please do install Remote - Containers manually. You can then use the extension command Remote-Containers: Reopen Folder in Container, which will automatically build the container and start the services. No need to use make!

Troubleshooting

make dev error: make: command not found

When running make dev:

bash: make: command not found

Solution (if using Windows): Click the Windows button, then type “environment properties” into the search bar and hit Enter. Click Environment Variables, then under System variables choose Path and click Edit. Click New and insert C:\Program Files (x86)\GnuWin32\bin, then save the changes. Open a new terminal and test that the command works. (See Make Windows for more.)

Solution (if using a Linux distro): Check that your bash PATH setting is correct. If it is, your Linux distro doesn't include make by default. For Debian (which I had that problem with --Pau Amma), use

sudo apt-get install make

For other distros, consult your distro's documentation or support resources for the command to use.

make dev error: [build_lang] Error 2 - Could not load taxonomy: /mnt/podata/taxonomies/traces.result.json

When running make dev:

<h1>Software error:</h1>
<pre>Could not load taxonomy: /mnt/podata/taxonomies/traces.result.json at /opt/product-opener/lib/ProductOpener/Tags.pm line 1976.
Compilation failed in require at /opt/product-opener/scripts/build_lang.pl line 31, &lt;DATA&gt; line 2104.
BEGIN failed--compilation aborted at /opt/product-opener/scripts/build_lang.pl line 31, &lt;DATA&gt; line 2104.
</pre>
<p>
For help, please send mail to this site's webmaster, giving this error message
and the time and date of the error.
</p>
[Tue Apr  5 19:36:40 2022] build_lang.pl: Could not load taxonomy: /mnt/podata/taxonomies/traces.result.json at /opt/product-opener/lib/ProductOpener/Tags.pm line 1976.
[Tue Apr  5 19:36:40 2022] build_lang.pl: Compilation failed in require at /opt/product-opener/scripts/build_lang.pl line 31, <DATA> line 2104.
[Tue Apr  5 19:36:40 2022] build_lang.pl: BEGIN failed--compilation aborted at /opt/product-opener/scripts/build_lang.pl line 31, <DATA> line 2104.
make: *** [build_lang] Error 2

Solution: Project needs Symlinks to be enabled. traces.result.json is a symlink to allergens.result.json

You have to enable the 'Developer Mode' in order to use the symlinks. To enable Developer Mode:

• on windows 10: Settings > Update & Security > 'For developers' …
• on windows 11: Settings > Privacy & Security > 'For developers' …

and turn on the toggle for Developer Mode.

On Windows systems, the git repository needs to be cloned with symlinks enabled.

You need to remove the directory where you cloned the project, and clone the project again, using the right options:

# replace my-user-id with your user id if you forked the repository, otherwise use openfoodfacts
git clone -c core.symlinks=true git@github.com:my-user-id/openfoodfacts-server.git

make dev error: open /.docker/buildx/current: permission denied

On macOS When running make dev:

docker compose --env-file=.env  build   2>&1
open /.docker/buildx/current: permission denied
make: *** [build] Error 1
openfoodfacts-server %

Solution: Check for permissions by

ls -la /.docker/buildx

If there is a file that is not owned by user with root instead of USER:

-rw-------   1 root    staff   48 Apr 28 17:04 current

Then Run

sudo chown -R USER:USER /Users/USER/.docker/buildx

Replacing USER with current user

'rm' is not recognized as an internal or external command

When running make import_prod_data or some other commands.

Solution:

Use the Git Bash shell to run the make commands in windows so that programs like nproc and rm are found.

System cannot find wget

When running make import_prod_data.

process_begin: CreateProcess(NULL, wget --no-verbose https://static.openfoodfacts.org/data/openfoodfacts-mongodbdump.gz, ...) failed.
make (e=2): The system cannot find the file specified.

You need to install wget for windows. The referenced version is able to use the Windows Certificate Store, whereas the standard gnuwin32 version will give errors about not being able to verify the server certificate.

make: *** [Makefile:154: import_sample_data] Error 22

When running make import_sample_data

<hl>Software error:</h1>
<pre>MongoDB: :SelectionError: No writable server available. MongoDB server status:
Topology type: Single; Member status:
mongodb:27017 (type: Unknown, error: MongoDB::NetworkError: Could not connect to 'mongodb:27017': Temporary failure in name resolution )
</pre>
<p>
For help, please send mail to this site's webmaster, giving this error message
and the time and date of the error.
<p>
[Sat Dec 17 19:52:21 2022] update_all_products from_dir_in_mongodb.pl: MongoDB::SelectionError: No writable server available. MongoDB server status:

[Sat Dec 17 19:52:21 2022] update_all_products from_dir_in_mongodb.pl: Topology type: Single; Member status:

[Sat Dec 17 19:52:21 2022] update_all_products from_dir_in_mongodb.pl: mongodb:27017 (type: Unknown, error: MongoDB::NetworkError: Could not connect to 'mongodb:27017': Temporary failure in name resolution )

make: *** [Makefile:154: import_sample data] Error 22

Solution: The cause of this issue is that you already have the mongodb database server running on your local machine at port 27017.

For Linux users:

First stop the MongoDB server from your OS

sudo systemctl stop mongod

Then check that mongod is stopped with:

systemctl status mongod | grep Active

Note: The output of this command should be: Active: inactive (dead)

Then, execute this:

docker compose up

Note: To know more about docker compose commands do read this guide

make dev error: [build_lang] Error 13 - can't write into /mnt/podata/data/Lang.openfoodfacts.localhost.sto

When running make dev:

<h1>Software error:</h1>
<pre>can't write into /mnt/podata/data/Lang.openfoodfacts.localhost.sto: Permission denied at /opt/product-opener/lib/ProductOpener/Store.pm line 234.
</pre>
<p>
For help, please send mail to this site's webmaster, giving this error message
and the time and date of the error.

</p>
make: *** [Makefile:126: build_lang] Error 13

Solution:

Use the powershell/cmd to run the make dev commands in windows.

make dev error: [build] Error 1 - failed to solve: process "/bin/sh -c usermod --uid $USER_UID www-data && groupmod --gid $USER_GID www-data" did not complete successfully: exit code: 4

On WSL(for root user) When using make dev:

<p>
------
 > [backend modperl 5/5] RUN usermod --uid 0 www-data &&     groupmod --gid 1000 www-data:
0.492 usermod: UID '0' already exists
------
failed to solve: process "/bin/sh -c usermod --uid $USER_UID www-data &&     groupmod --gid $USER_GID www-data" did not complete successfully: exit code: 4
</p>
make: *** [Makefile:147: build] Error 1

Solution:

Reason for the error: "usermod: UID '0' already exists" means that the user is being assigned UID 0, which is already used by the root user.

To solve it create an environment variable(USER_UID=1000) inside .envrc using direnv. Steps to do that:

Install direnv

sudo apt install direnv

Enable direnv for Your Shell

echo 'eval "$(direnv hook bash)"' >> ~/.bashrc
source ~/.bashrc

Create a .envrc File in Your Project

echo 'USER_UID=1000' > .envrc

Finally, run this to allow .envrc:

direnv allow

make dev error: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied make: *** [Makefile:147 : build] Erreur 1

Solution:

Add the user to docker group :

sudo usermod -aG docker $USER