Skip to content

How to update the index#

As you use search-a-licious, you will first import the data, but then, you might need to update the index to keep it up to date with the latest data.

There are two strategies to update the index:

  • either you push events to the redis stream, and search-a-licious update the index continuously
  • either you update the index from time to time using a new import of whole dataset

First import#

First import will populate Elasticsearch index with all the data.

See initial import section in tutorial and import reference documentation

It's important to note that if you don't use the continous updates strategy, you need to use --skip-updates option.

Continuous updates#

To have continuous updates, you need to push events to the redis stream.

Normally this will be done by your application.

On each update/removal/etc. it must push events with at least:

  • the document id
  • and eventually more info (if you need them to filter out items, for example).

Of course you can also imagine to push events from another service, if you have another way of getting changes, but this part is up to you.

Then you just have to run the updater container that comes in the docker-compose configuration.

docker-compose up -d updater

This will continuously fetch updates from the event stream, and update the index accordingly.

The updater only processes events pushed after it started. It does not catch up on events pushed while it was stopped.

To catch up after a downtime, run the import command without --skip-updates: once the import is done, it processes the events of the stream since the most recent last_modified_field_name found in the index.

Replay relies on the ids of the Redis stream being Unix timestamps in milliseconds (the default <ms>-<seq> ids): events with ids that are not compatible timestamps (eg. 0-1) would be skipped.

An event pushed after the replay ended, but before the new index is switched in as the alias, would be written to the old index and missed by the new one. To be safe, pause the production of events until the import is finished and the updater is listening to the stream, then resume it.

Updating the index from time to time with an export#

Another way to update the index is to periodically re-import the all the data, or changed data.

This is less compelling to your users, but this might be the best way if you are using an external database publishing changes on a regular bases.

For that you can use the import command, with the --skip-updates option, and with the --partial option if you are importing only changed data (otherwise it is advised to use the normal import process, which can be rolled-back (it create a new index)).

Document fetcher and pre-processing#

In the configuration, you can define a document_fetcher and a preprocessor to transform the data.

Those are fully qualified dotted names to python classes.

document_fetcher is only used on continuous updates, while preprocessor is used both on continuous updates and on initial import.