Configuration

Config Module

class app.config.BaseESIndexConfig(*, name: str, number_of_shards: int = 4, number_of_replicas: int = 1)[source]

Base class for configuring ElasticSearch indexes

model_config = {}[source]

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class app.config.Config(*, indices: dict[str, IndexConfig], default_index: str)[source]

Search-a-licious server configuration.

The configuration is loaded from a YAML file, that must satisfy this schema.

Validations will be performed while we load it.

defaut_index_must_exist()[source]

Validator that checks that default_index exists.

classmethod from_yaml(path: Path) → Config[source]

Create a Config from a yaml configuration file.

get_index_config(index_id: str | None) → tuple[str, IndexConfig][source]

Return a (index_id, IndexConfig) for the given index_id.

If no index_id is provided, the default index is used.

Raises:

ValueError – if the index_id is not found

model_config = {}[source]

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

redis_stream_name_should_be_unique()[source]

Validator that checks that every redis_stream_name is unique.

class app.config.ConfigGenerateJsonSchema(by_alias: bool | None = None, ref_template: str = '#/$defs/{model}', union_format: Literal['any_of', 'primitive_type_array'] = 'any_of')[source]

Config to add fields to generated JSON schema for Config.

generate(schema, mode='validation')[source]

Generates a JSON schema for a specified schema in a specified mode.

Args:

schema: A Pydantic model. mode: The mode in which to generate the schema. Defaults to ‘validation’.

Returns:

A JSON schema representing the specified schema.

Raises:

PydanticUserError: If the JSON schema generator has already been used to generate a JSON schema.

class app.config.ESIndexConfig(*, name: str, number_of_shards: int = 4, number_of_replicas: int = 1, id_field_name: str, last_modified_field_name: str)[source]

This is the configuration for the main index containing the data.

It’s used to create the index in ElasticSearch, and configure its mappings (along with the fields config)

model_config = {}[source]

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class app.config.FieldConfig(*, name: str = '', type: FieldType, required: bool = False, index: bool = True, input_field: str | None = None, split: bool = False, full_text_search: bool = False, bucket_agg: bool = False, taxonomy_name: str | None = None, fields: dict[str, FieldConfig] | None = None)[source]
classmethod add_field_name_to_each_field(fields: dict[str, FieldConfig])[source]

It’s handy to have the name of the field in the field definition

bucket_agg_should_be_used_for_keyword_and_numeric_types_only()[source]

Validator that checks that bucket_agg is only provided for fields with a numeric type, keyword, taxonomy or bool.

get_input_field()[source]

Return the name of the field to use in input data.

has_lang_subfield() → bool[source]

Return wether this field type is supposed to have different values per languages

model_config = {}[source]

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

no_index_false_for_object_and_nested()[source]

Validator that checks that index is not set to False for fields with types object or nested. (because ES does not support it)

subfields_only_if_object_or_nested()[source]

If we have an object or nested field, and only in these cases, we need a fields object

class app.config.FieldType(*values)[source]

Supported field types in Search-a-Licious are:

  • keyword: string values that won’t be interpreted (tokenized). Good for things like tags, serial, property values, etc.

  • date: Date fields

  • double, float, half_float, scaled_float: different ways of storing floats with different capacity

  • short, integer, long, unsigned_long : integers (with different capacity: 8 / 16 / 32 bits)

  • bool: boolean (true / false) values

  • text: a text which is tokenized to enable full text search

  • text_lang: like text, but with different values in different languages. Tokenization will use analyzers specific to each languages.

  • taxonomy: a field akin to keyword but with support for matching using taxonomy synonyms and translations (and in fact also a text mapping possibility)

  • disabled: a field that is not stored nor searchable (see [Elasticsearch help])

  • object: this field contains a dict with sub-fields.

  • nested: this field contains an array of objects.

[Elasticsearch help](https://www.elastic.co/guide/en/elasticsearch/reference/current/enabled.html)

is_numeric()[source]

Return wether this field type can be considered numeric

class app.config.IndexConfig(*, index: ~app.config.models.ESIndexConfig, fields: dict[str, ~app.config.models.FieldConfig], split_separator: str = ',', lang_separator: str = '_', primary_color: str = '#aaa', accent_color: str = '#222', synonyms_strategy: ~app.config.models.SynonymsStrategy = SynonymsStrategy.INDEX, taxonomy: ~app.config.models.TaxonomyConfig, supported_langs: list[str], document_fetcher: str, preprocessor: ~typing.Annotated[str, FieldInfo(annotation=NoneType, required=True, description='The full qualified reference to the preprocessor\nto use before data import.\n\nThis class must inherit `app.indexing.BaseDocumentPreprocessor`\nand specialize the `preprocess` method.\n\nThis is used to adapt the data schema\nor to add search-a-licious specific fields\nfor example.', examples=['app.openfoodfacts.DocumentPreprocessor'])] | None = None, result_processor: ~typing.Annotated[str, FieldInfo(annotation=NoneType, required=True, description='The full qualified reference to the elasticsearch result processor\n                    to use after search query to Elasticsearch.\n\n)                    This class must inherit `app.postprocessing.BaseResultProcessor`\n                    and specialize the `process_after`\n\n                    This is can be used to add custom fields computed from index content.\n                    ', examples=['app.openfoodfacts.ResultProcessor'])] | None = None, scripts: ~typing.Annotated[dict[str, ~app.config.models.ScriptConfig], FieldInfo(annotation=NoneType, required=True, description="You can add scripts that can be used for sorting results.\n\nEach key is a script name, with it's configuration.")] | None = None, match_phrase_boost: float = 2.0, match_phrase_boost_proximity: int | None = None, document_denylist: set[str] = <factory>, redis_stream_name: str | None = None)[source]

This object gives configuration for one index.

One index usually correspond to one dataset.

classmethod add_field_name_to_each_field(fields: dict[str, FieldConfig])[source]

It’s handy to have the name of the field in the field definition

classmethod ensure_no_fields_use_reserved_name(fields: dict[str, FieldConfig])[source]

Verify that no field name clashes with a reserved name

field_references_must_exist_and_be_valid()[source]

Validator that checks that every field reference in ESIndexConfig refers to an existing field and is valid.

property full_text_fields: dict[str, FieldConfig][source]

Fully qualified name of fields that are part of default full text search

get_synonym_set_name(taxonomy_name: str, lang: str) → str[source]

Return the prefix that the synonyms set for a taxonomy and language should have

property lang_fields: dict[str, FieldConfig][source]

Fully qualified name of fields that are translated

model_config = {}[source]

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

taxonomy_name_should_be_defined()[source]

Validator that checks that for if taxonomy_type is defined for a field, it refers to a taxonomy defined in taxonomy.sources.

property text_lang_fields: dict[str, FieldConfig][source]

List all text_lang fields in an efficient way

class app.config.LoggingLevel(*values)[source]

Accepted logging levels

  • NOTSET - means no los

  • DEBUG / INFO / WARNING / ERROR / CRITICAL - match standard Python logging levels

class app.config.ScriptConfig(*, lang: ScriptType = ScriptType.expression, source: str, params: Annotated[dict[str, Any], FieldInfo(annotation=NoneType, required=True, description='Params for the scripts. We need this to retrieve and validate parameters')] | None, static_params: Annotated[dict[str, Any], FieldInfo(annotation=NoneType, required=True, description="Additional params for the scripts that can't be supplied by the API (constants)")] | None)[source]

Scripts can be used to sort results of a search.

This use ElasticSearch internal capabilities

model_config = {}[source]

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class app.config.ScriptType(*values)[source]
class app.config.Settings(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: EnvPrefixTarget | None = None, _env_file: DotenvType | None = PosixPath('.'), _env_file_encoding: str | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_show_env_vars: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: Mapping[str, str | list[str]] | None = None, _secrets_dir: PathType | None = None, _build_sources: tuple[tuple[PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, *, config_path: Path | None = None, elasticsearch_url: str = 'http://localhost:9200', elasticsearch_request_timeout: Annotated[float, Gt(gt=0)] = 10.0, redis_host: str = 'localhost', redis_port: int = 6379, redis_reader_timeout: int = 5, sentry_dns: str | None = None, log_level: LoggingLevel = LoggingLevel.INFO, taxonomy_cache_dir: Path = PosixPath('data/taxonomies'), user_agent: str = 'search-a-licious')[source]

Settings for Search-a-licious

The most important settings is config_path.

Those settings can be overridden through environment by using the name in capital letters. If you use docker compose, a good way to do that is to modify those values in your .env file.

model_config = {'arbitrary_types_allowed': True, 'case_sensitive': False, 'cli_avoid_json': False, 'cli_enforce_required': False, 'cli_exit_on_error': True, 'cli_flag_prefix_char': '-', 'cli_hide_none_type': False, 'cli_ignore_unknown_args': False, 'cli_implicit_flags': False, 'cli_kebab_case': False, 'cli_parse_args': None, 'cli_parse_none_str': None, 'cli_prefix': '', 'cli_prog_name': None, 'cli_shortcuts': None, 'cli_show_env_vars': False, 'cli_use_class_docs_for_groups': False, 'enable_decoding': True, 'env_file': None, 'env_file_encoding': None, 'env_ignore_empty': False, 'env_nested_delimiter': None, 'env_nested_max_split': None, 'env_parse_enums': None, 'env_parse_none_str': None, 'env_prefix': '', 'env_prefix_target': 'variable', 'extra': 'forbid', 'json_file': None, 'json_file_encoding': None, 'nested_model_default_partial_update': False, 'protected_namespaces': ('model_validate', 'model_dump', 'settings_customise_sources'), 'secrets_dir': None, 'toml_file': None, 'validate_default': True, 'yaml_config_section': None, 'yaml_file': None, 'yaml_file_encoding': None}[source]

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class app.config.SettingsGenerateJsonSchema(by_alias: bool | None = None, ref_template: str = '#/$defs/{model}', union_format: Literal['any_of', 'primitive_type_array'] = 'any_of')[source]

Config to add fields to generated JSON schema for Settings.

generate(schema, mode='validation')[source]

Generates a JSON schema for a specified schema in a specified mode.

Args:

schema: A Pydantic model. mode: The mode in which to generate the schema. Defaults to ‘validation’.

Returns:

A JSON schema representing the specified schema.

Raises:

PydanticUserError: If the JSON schema generator has already been used to generate a JSON schema.

class app.config.SynonymsStrategy(*values)[source]

Strategy to use for synonyms.

  • index: synonyms are implemented using the ElasticSearch synonyms sets strategy. It can be efficient but it is not well supported if you have a lot of taxonomies and a lot of languages. (ElasticSearch start will be very very slow)

  • none: synonyms are not automatically accounted for during requests (you still get the taxonomies separate indexes). This means your queries have to use the taxonomy id, instead of a synonym or a translation.

A “query” strategy might be implemented in the future, acting before sending the query to ElasticSearch.

class app.config.TaxonomyConfig(*, sources: list[TaxonomySourceConfig], index: TaxonomyIndexConfig, preprocessor: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='The full qualified reference to the preprocessor\nto use before taxonomy entry import.\n\nThis class must inherit `app.indexing.BaseTaxonomyPreprocessor`\nand specialize the `preprocess` method.\n\nThis is used to adapt the taxonomy schema\nor to add specific fields for example.', examples=['app.openfoodfacts.TaxonomyPreprocessor'])] | None = None)[source]

Configuration of taxonomies, that is collections of entries with synonyms in multiple languages.

See [Explain taxonomies](../explain-taxonomies)

Field may be linked to taxonomies.

It enables enriching search with synonyms, as well as providing suggestions, or informative facets.

Note: if you define taxonomies, you must import them using [import-taxonomies command](../ref-python/cli.html#python3-m-app-import-taxonomies)

model_config = {}[source]

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class app.config.TaxonomyIndexConfig(*, name: str, number_of_shards: int = 4, number_of_replicas: int = 1)[source]

This is the configuration of the ElasticSearch index storing the taxonomies.

All taxonomies are stored within the same index.

It enables functions like auto-completion, or field suggestions as well as enrichment of requests with synonyms.

model_config = {}[source]

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class app.config.TaxonomySourceConfig(*, name: str, url: FileUrl | HttpUrl, max_synonyms_entries: int = 10000)[source]

Configuration on how to fetch a particular taxonomy.

model_config = {}[source]

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

app.config.get_config() → Config[source]

Return the object containing global configuration

It raises if configuration was not yet set