OpenFoodFacts Infrastructure#
Sysadmin repository for the various parts of the Open Food Facts infrastructure. We have a specific repository regarding monitoring
Current priorities#
As of 2023 our current priorities are:
- server migration with hardware upgrade and a clean containerized install - and zfs syncs
- better encrypted two-way communication between data centers (stunnel + https)
- backups checks
- testing backups through staging
- with automated deployment of new clones
- monitoring backups
- testing backups through staging
- better monitoring
- more/better dashboards
- more active checks (monitoring - alerts)
- less false positives in alerts
- GPU server for inference and possibly one for training (not hosted) - may
Incident logs#
We started logging incidents by server:
- off1 incident logs
- off2 incident logs
- off3 incident logs
- ovh1 incident logs
- ovh2 incident logs
- ovh3 incident logs
Documentation#
Link to Github Page
The infrastructure documentation is as follows:
- Free Datacenter - Data center with main production servers
- Linux Server - servers general setup
- Mail - servers mail setup
- An introduction to ZFS - ZFS is much used in our infrastructure
- Proxmox - about proxmox management
- CICD - continuous integration and deployment
- Observability - doc on monitoring / logs / etc.
- Docker Onboarding
- Docker Infrastructure
- Virtual Machines
The main services:
Some services:
- Discourse for forum
- NGINX reverse proxy the reverse proxy for OVH services
- Folksonomy user editable labels and values
- Matomo for web analytics
- Producers sftp to push product updates on producer platform
- Zammad for support
- Odoo the CRM
Also look at all install and post-mortem reports in docs/reports
Monthly meetings
* We e-meet monthly at [18:00 CET](https://dateful.com/convert/paris-france?t=6pm) the second Thuesday of the month
*  Video call link: meet.google.com/ybq-dwuk-pvi
* Join by phone: https://tel.meet/nnw-qswu-hza?pin=2111028061202
* Add the Event to your Calendar by [adding the Open Food Facts community calendar to your calendar](https://wiki.openfoodfacts.org/Events)
* [Monthly Agenda](https://docs.google.com/document/d/18BNNqxL6MSLAMrwsQ9F-CaVBUojnUEilQmQuC7XdVks/edit?usp=sharing): please add the Agenda items as early as you can.
* Make sure to check the Agenda items in advance of the meeting, so that we have the most informed discussions possible.
* The meeting will handle Agenda items first, and if time permits, collaborative bug triage.
Requests#
Virtual Machines#
| Title | State | OS | CPU # | RAM | SSD (Local) | HDD (Remote) | Services |
|---|---|---|---|---|---|---|---|
| Wordpress test CT [#200] | open | Debian last Stable. | 3 | [Explain if > 4 Gb.] | 6 GB. | 0 | Apache, PHP, Wordpress. |
| Monitoring - VM (QEMU host for docker) [#159] | open | Debian | * 4 CPUs | * 12G for we have influxdb and elastic-search that needs memory | * 30 Go disk (it is currently around 14G, but this will grow because we want to harvest more logs and more metrics) | * 50Go for ES backups | Docker, docker-compose |
| CT for new blog engine [#80] | open | Debian stable. | 3 CPU. | 2 GB. | 10 GB | -- | LAMP + wordpress. |
| CT for Folksonomy Engine API dev [#76] | open | Default to Debian last Stable. | 2 | 1 GB | 12 GB. | - | PostgreSQL, Python3. |
| Wild School Eco-Score project [#37] | open | Debian 10 | 4 | 16 Gb | 30 Gb | 0 | MongoDB |
| slack-org [#36] | open | Debian 10 | 1 | 1 Gb | 10 Gb | None | Node.js |
| adminer-org [#29] | open | Debian 10 | 2 | 512 Mb. | 4 Gb or even less. | 0 | Nginx, PHP, Adminer. |
| Containers (x2) to build a replica set for OFF database [#28] | open | Debian 10 | 4 | 32 GB | 50 GB (DB = 20 GB). | 0 | Mongodb. |
| feedme-org [#27] | open | Debian 10 | 3 | 3 Gb. | 15 Gb. | 0 | PostgreSQL, Node.js, Nginx. |
| off-wiki-org [#21] | open | Debian 10 | 2 | 3 Gb | 14 Gb. | 14 Gb | Apache, PHP, MySQL, Mediawiki. |
| VM for the Community Portal [#124] | closed | Debian last Stable. | [Explain if > 4.] | [Explain if > 4 Gb.] | [Explain if > 32 Gb.] | [Explain if > 1 Tb.] | Python/Django, probably PostgreSQL, probably Apache and all Dockerized |
| VM for the Taxonomy Editor [#123] | closed | Debian last Stable. | [Explain if > 4.] | [Explain if > 4 Gb.] | [Explain if > 32 Gb.] | [Explain if > 1 Tb.] | Python, probably PostgreSQL, probably Apache for lightweight API serving from Docker |
| New VM QEMU for prod docker containers [#71] | closed | Debian 11 (stable) | 8 | 24 GB | 256 GB. | - | Services deployed in production: |
| monitoring [#59] | closed | Debian 11 | 4 | 32GB | 64GB | 500GB (ovh3 mount) | Docker: ElasticSearch (Kibana?, Logstash?), Grafana, InfluxDB, Prometheus, Alertmanager |
| impactestimator-net [#55] | closed | Debian 11 | 1 | 1GB | 1Gb | 0 | https://github.com/openfoodfacts/impactestimator |
| robotoff-ml [#53] | closed | Debian 11 | 8 | 96GB (Tensorflow, ANN) | 192GB [ML models] | 100GB | Tensorflow + ElasticSearch |
| robotoff-net [#51] | closed | Debian 11 | 4 | 16GB (DB 4GB, Services 8GB) | 92GB | 0GB | Robotoff API + Schedulers + Workers, PostgreSQL DB |
| mongo-dev [#45] | closed | Debian 10 | 2 | 16GB | 40GB | MongoDB running in Docker | |
| off-net [#41] | closed | Debian 10 | 4 | 16GB (PO needs > 6GB) | 192GB | 0GB | ProductOpener frontend + backend, MongoDB, PostgreSQL, Memcached |
| robotoff-dev [#40] | closed | Debian 10 | 4 | 8 Gb | 32 Gb | 100 Gb | robotoff, elastic search, tensorflow, postgresql |
| Matomo [#24] | closed | Debian 10 | No idea. | No idea. | No idea. | No idea. | LAMP |
| robotoff-org [#20] | closed | Debian 10 | 4 | 8 Gb | 32 Gb | 100 Gb | robotoff, elastic search, tensorflow, postgresql |
