Skip to content

Logo-ANN

About 1600 products are added every day to the database. Each product having multiple logos on its packaging, thousands of new logos are added every day to the valuable sources of information. These logos are often useful to get important data on products (origin, brand, quality, label, etc...).

A logo automatic detection from images and a logo manual classification features are implemented to Robotoff. The first step is to extract logos from products images of the database. The second one is to vectorize each logo thanks to a computer vision model. The third and last one is to search for each logo its nearest neighbors in an index containing all the embedded logos.

Logos extraction

When a new image is added to the database, Robotoff applies an object detection model to extract logos from it.1 This model, named "universal-logo-detector" 2, is an ONNX model trained by Open Food Facts on numerous data from the database. For each input image, it returns bounding boxes that represent the detection zone of each logo of the image and the category of the logo, namely "brand" or "label". To know more about this model, see the robotoff-models release.

Logos embedding

After the detection of a logo, in the same function 1, Robotoff uses a computer vision model to vectorize it. The model we use is CLIP-vit-base-patch32, a model developed and trained by OpenAI. Only the vision part of the model is used here, as the objective is only to vectorize the logos. The choice of CLIP-vit-base-patch32 was made after this benchmark. The model is loaded with Triton and is used only for inference.

With the logo crop of the initial image as input, CLIP returns an embedding and Robotoff stores it in its postgresql database.3

Each generated embedding is stored in an ElasticSearch index for nearest neighbor search. ElasticSearch allows for approximate nearest neighbor (ANN) search with an HNSW (Hierarchical Navigable Small World) index, which leads to fast and accurate search (see ANN benchmark).

After storing the embedding in the index, a search for its nearest neighbors is performed and the IDs of these neighbors are stored in the Robotoff PostgreSQL database. The nearest neighbor search is available via an API 4 available (here)[https://robotoff.openfoodfacts.org/api/v1/ann/search/185171?count=50] and used by (Hunger Games)[https://hunger.openfoodfacts.org/], the annotation game connected to Robotoff.

  1. see robotoff.workers.tasks.import_image.run_logo_object_detection ↩↩

  2. see models.universal-logo-detector ↩

  3. see robotoff.workers.tasks.import_image.save_logo_embeddings ↩