Skip to content

Nginx exporter on off2 proxy and off#

On off2 reverse proxy#

Expose nginx stats#

For nginx exporter we need to expose /stub_status on port 8080 (see doc) https://nginx.org/en/docs/http/ngx_http_stub_status_module.html#stub_status

I created the stub_status conf for that (linked from this repository). I limited to 127.0.0.1, it should be internal.

Test it: curl http://127.0.0.1:8080

Install Prometheus nginx exporter#

Now install the exporter: sudo apt install prometheus-nginx-exporter

As we are on a public interface, we don't want to expose the exporter widely, so we restrict it to 127.0.0.1

For that we do systemctl edit prometheus-nginx-exporter to add: 

[Service]
# ONLY listen on 127.0.0.1 for security reasons
Environment="LISTEN_ADDRESS=127.0.0.1:9113"
And moved it to versioned space: 
mv /etc/systemd/system/prometheus-nginx-exporter.service.d /opt/openfoodfacts-infrastructure/confs/off2-reverse-proxy/systemd/system/ && \
ln -s /opt/openfoodfacts-infrastructure/confs/off2-reverse-proxy/systemd/system/prometheus-nginx-exporter.service.d /etc/systemd/system/

Reload: systemctl daemon-reload && systemctl restart prometheus-nginx-exporter

Check exporter status: systemctl status prometheus-nginx-exporter

Test it locally: 

curl http://127.0.0.1:9113/metrics

Test it's not available from the public interface: 

nc -vz 213.36.253.214 9113
nc: connect to 213.36.253.214 port 9113 (tcp) failed: Connection refused

Tested it from ovh2, it does not work but from ovh1 it works !

Expose the metrics#

Edit the /etc/nginx/sites-enabled/free-exporters.openfoodfacts.org to add the right entry to the map directive:

# map from service to exporter
map $uri $exporter {
    ...
    # nginx on this proxy
    "/proxy/nginx/metrics" 127.0.0.1:9113;
    ...

Restart nginx and test it: 

curl -u prometheus:**password-here**  https://free-exporters.openfoodfacts.org/proxy/nginx/metrics

On off#

Expose nginx stats#

I created the stub_status conf for that in /srv/off/conf/nginx/sites-available/stub_status 

# This enables prometheus exporter to get data from nginx
server {
        listen 127.0.0.1:8080;
        stub_status on;
}
linked it: ln -s /srv/off/conf/nginx/sites-available/stub_status /etc/nginx/sites-enabled/

Restart nginx and test it: curl http://10.1.0.107:8080

Install Prometheus nginx exporter#

Install the exporter: sudo apt install prometheus-nginx-exporter

We are on a container without public interface so it's ok to expose the exporter.

Check exporter status: systemctl status prometheus-nginx-exporter

Test it locally: 

curl http://127.0.0.1:9113/metrics

And on the reverse proxy: 

curl http://10.1.0.113:9113/metrics

Apache exporter#

sudo apt install prometheus-apache-exporter
# get conf in git
sudo -u off mkdir conf/etc-default
sudo mv /etc/default/prometheus-apache-exporter /srv/off/conf/etc-default/
sudo ln -s /srv/off/conf/etc-default/prometheus-apache-exporter /etc/default/
sudo chown off:off /srv/off/conf/etc-default/prometheus-apache-exporter

Our apache is not exposing on a specific port 8004, so we need to edit the config file.

We edit /etc/default/prometheus-apache-exporter

# Set the command-line arguments to pass to the server.
ARGS='-scrape_uri http://127.0.0.1:8004/server-status/?auto'

restart: sudo systemctl restart prometheus-apache-exporter verify: sudo systemctl status prometheus-apache-exporter

Test: 

curl http://10.1.0.113:9117/metrics

Expose the metrics#

As before by editing the map directive in /etc/nginx/sites-enabled/free-exporters.openfoodfacts.org

test it: 

curl -u prometheus:**password-here**  https://free-exporters.openfoodfacts.org/off/nginx/metrics
curl -u prometheus:**password-here**  https://free-exporters.openfoodfacts.org/off/apache/metrics

Add the exporter to the monitoring stack#

Edit prometheus/config.yml to add the two services path to free-exporters section. (see commit 811aaa36c and commit e3ee4109e)

The nginx dashboards works immediately.

I had to edit the apache dashboard as it was expecting a host:port pattern, which is not the case here, I now only use instance name directly. (see commit 08972b0)