Configuration¶
Config Module¶
- class app.config.BaseESIndexConfig(*, name: str, number_of_shards: int = 4, number_of_replicas: int = 1)[source]¶
Base class for configuring ElasticSearch indexes
- class app.config.Config(*, indices: dict[str, IndexConfig], default_index: str)[source]¶
Search-a-licious server configuration.
The configuration is loaded from a YAML file, that must satisfy this schema.
Validations will be performed while we load it.
- get_index_config(index_id: str | None) tuple[str, IndexConfig][source]¶
Return a (index_id, IndexConfig) for the given index_id.
If no index_id is provided, the default index is used.
- Raises:
ValueError – if the index_id is not found
- class app.config.ConfigGenerateJsonSchema(by_alias: bool | None = None, ref_template: str = '#/$defs/{model}', union_format: Literal['any_of', 'primitive_type_array'] = 'any_of')[source]¶
Config to add fields to generated JSON schema for Config.
- generate(schema, mode='validation')[source]¶
Generates a JSON schema for a specified schema in a specified mode.
- Args:
schema: A Pydantic model. mode: The mode in which to generate the schema. Defaults to ‘validation’.
- Returns:
A JSON schema representing the specified schema.
- Raises:
PydanticUserError: If the JSON schema generator has already been used to generate a JSON schema.
- class app.config.ESIndexConfig(*, name: str, number_of_shards: int = 4, number_of_replicas: int = 1, id_field_name: str, last_modified_field_name: str)[source]¶
This is the configuration for the main index containing the data.
It’s used to create the index in ElasticSearch, and configure its mappings (along with the fields config)
- class app.config.FieldConfig(*, name: str = '', type: FieldType, required: bool = False, index: bool = True, input_field: str | None = None, split: bool = False, full_text_search: bool = False, bucket_agg: bool = False, taxonomy_name: str | None = None, fields: dict[str, FieldConfig] | None = None)[source]¶
- classmethod add_field_name_to_each_field(fields: dict[str, FieldConfig])[source]¶
It’s handy to have the name of the field in the field definition
- bucket_agg_should_be_used_for_keyword_and_numeric_types_only()[source]¶
Validator that checks that bucket_agg is only provided for fields with a numeric type, keyword, taxonomy or bool.
- has_lang_subfield() bool[source]¶
Return wether this field type is supposed to have different values per languages
- model_config = {}[source]¶
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class app.config.FieldType(*values)[source]¶
Supported field types in Search-a-Licious are:
keyword: string values that won’t be interpreted (tokenized). Good for things like tags, serial, property values, etc.
date: Date fields
double, float, half_float, scaled_float: different ways of storing floats with different capacity
short, integer, long, unsigned_long : integers (with different capacity: 8 / 16 / 32 bits)
bool: boolean (true / false) values
text: a text which is tokenized to enable full text search
text_lang: like text, but with different values in different languages. Tokenization will use analyzers specific to each languages.
taxonomy: a field akin to keyword but with support for matching using taxonomy synonyms and translations (and in fact also a text mapping possibility)
disabled: a field that is not stored nor searchable (see [Elasticsearch help])
object: this field contains a dict with sub-fields.
nested: this field contains an array of objects.
[Elasticsearch help](https://www.elastic.co/guide/en/elasticsearch/reference/current/enabled.html)
- class app.config.IndexConfig(*, index: ~app.config.models.ESIndexConfig, fields: dict[str, ~app.config.models.FieldConfig], split_separator: str = ',', lang_separator: str = '_', primary_color: str = '#aaa', accent_color: str = '#222', synonyms_strategy: ~app.config.models.SynonymsStrategy = SynonymsStrategy.INDEX, taxonomy: ~app.config.models.TaxonomyConfig, supported_langs: list[str], document_fetcher: str, preprocessor: ~typing.Annotated[str, FieldInfo(annotation=NoneType, required=True, description='The full qualified reference to the preprocessor\nto use before data import.\n\nThis class must inherit `app.indexing.BaseDocumentPreprocessor`\nand specialize the `preprocess` method.\n\nThis is used to adapt the data schema\nor to add search-a-licious specific fields\nfor example.', examples=['app.openfoodfacts.DocumentPreprocessor'])] | None = None, result_processor: ~typing.Annotated[str, FieldInfo(annotation=NoneType, required=True, description='The full qualified reference to the elasticsearch result processor\n to use after search query to Elasticsearch.\n\n) This class must inherit `app.postprocessing.BaseResultProcessor`\n and specialize the `process_after`\n\n This is can be used to add custom fields computed from index content.\n ', examples=['app.openfoodfacts.ResultProcessor'])] | None = None, scripts: ~typing.Annotated[dict[str, ~app.config.models.ScriptConfig], FieldInfo(annotation=NoneType, required=True, description="You can add scripts that can be used for sorting results.\n\nEach key is a script name, with it's configuration.")] | None = None, match_phrase_boost: float = 2.0, match_phrase_boost_proximity: int | None = None, document_denylist: set[str] = <factory>, redis_stream_name: str | None = None)[source]¶
This object gives configuration for one index.
One index usually correspond to one dataset.
- classmethod add_field_name_to_each_field(fields: dict[str, FieldConfig])[source]¶
It’s handy to have the name of the field in the field definition
- classmethod ensure_no_fields_use_reserved_name(fields: dict[str, FieldConfig])[source]¶
Verify that no field name clashes with a reserved name
- field_references_must_exist_and_be_valid()[source]¶
Validator that checks that every field reference in ESIndexConfig refers to an existing field and is valid.
- property full_text_fields: dict[str, FieldConfig][source]¶
Fully qualified name of fields that are part of default full text search
- get_synonym_set_name(taxonomy_name: str, lang: str) str[source]¶
Return the prefix that the synonyms set for a taxonomy and language should have
- property lang_fields: dict[str, FieldConfig][source]¶
Fully qualified name of fields that are translated
- model_config = {}[source]¶
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- taxonomy_name_should_be_defined()[source]¶
Validator that checks that for if taxonomy_type is defined for a field, it refers to a taxonomy defined in taxonomy.sources.
- property text_lang_fields: dict[str, FieldConfig][source]¶
List all text_lang fields in an efficient way
- class app.config.LoggingLevel(*values)[source]¶
Accepted logging levels
NOTSET - means no los
DEBUG / INFO / WARNING / ERROR / CRITICAL - match standard Python logging levels
- class app.config.ScriptConfig(*, lang: ScriptType = ScriptType.expression, source: str, params: Annotated[dict[str, Any], FieldInfo(annotation=NoneType, required=True, description='Params for the scripts. We need this to retrieve and validate parameters')] | None, static_params: Annotated[dict[str, Any], FieldInfo(annotation=NoneType, required=True, description="Additional params for the scripts that can't be supplied by the API (constants)")] | None)[source]¶
Scripts can be used to sort results of a search.
This use ElasticSearch internal capabilities
- class app.config.Settings(_case_sensitive: bool | None = None, _nested_model_default_partial_update: bool | None = None, _env_prefix: str | None = None, _env_prefix_target: EnvPrefixTarget | None = None, _env_file: DotenvType | None = PosixPath('.'), _env_file_encoding: str | None = None, _env_ignore_empty: bool | None = None, _env_nested_delimiter: str | None = None, _env_nested_max_split: int | None = None, _env_parse_none_str: str | None = None, _env_parse_enums: bool | None = None, _cli_prog_name: str | None = None, _cli_parse_args: bool | list[str] | tuple[str, ...] | None = None, _cli_settings_source: CliSettingsSource[Any] | None = None, _cli_parse_none_str: str | None = None, _cli_hide_none_type: bool | None = None, _cli_avoid_json: bool | None = None, _cli_enforce_required: bool | None = None, _cli_use_class_docs_for_groups: bool | None = None, _cli_show_env_vars: bool | None = None, _cli_exit_on_error: bool | None = None, _cli_prefix: str | None = None, _cli_flag_prefix_char: str | None = None, _cli_implicit_flags: bool | Literal['dual', 'toggle'] | None = None, _cli_ignore_unknown_args: bool | None = None, _cli_kebab_case: bool | Literal['all', 'no_enums'] | None = None, _cli_shortcuts: Mapping[str, str | list[str]] | None = None, _secrets_dir: PathType | None = None, _build_sources: tuple[tuple[PydanticBaseSettingsSource, ...], dict[str, Any]] | None = None, *, config_path: Path | None = None, elasticsearch_url: str = 'http://localhost:9200', elasticsearch_request_timeout: Annotated[float, Gt(gt=0)] = 10.0, redis_host: str = 'localhost', redis_port: int = 6379, redis_reader_timeout: int = 5, sentry_dns: str | None = None, log_level: LoggingLevel = LoggingLevel.INFO, taxonomy_cache_dir: Path = PosixPath('data/taxonomies'), user_agent: str = 'search-a-licious')[source]¶
Settings for Search-a-licious
The most important settings is config_path.
Those settings can be overridden through environment by using the name in capital letters. If you use docker compose, a good way to do that is to modify those values in your .env file.
- model_config = {'arbitrary_types_allowed': True, 'case_sensitive': False, 'cli_avoid_json': False, 'cli_enforce_required': False, 'cli_exit_on_error': True, 'cli_flag_prefix_char': '-', 'cli_hide_none_type': False, 'cli_ignore_unknown_args': False, 'cli_implicit_flags': False, 'cli_kebab_case': False, 'cli_parse_args': None, 'cli_parse_none_str': None, 'cli_prefix': '', 'cli_prog_name': None, 'cli_shortcuts': None, 'cli_show_env_vars': False, 'cli_use_class_docs_for_groups': False, 'enable_decoding': True, 'env_file': None, 'env_file_encoding': None, 'env_ignore_empty': False, 'env_nested_delimiter': None, 'env_nested_max_split': None, 'env_parse_enums': None, 'env_parse_none_str': None, 'env_prefix': '', 'env_prefix_target': 'variable', 'extra': 'forbid', 'json_file': None, 'json_file_encoding': None, 'nested_model_default_partial_update': False, 'protected_namespaces': ('model_validate', 'model_dump', 'settings_customise_sources'), 'secrets_dir': None, 'toml_file': None, 'validate_default': True, 'yaml_config_section': None, 'yaml_file': None, 'yaml_file_encoding': None}[source]¶
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class app.config.SettingsGenerateJsonSchema(by_alias: bool | None = None, ref_template: str = '#/$defs/{model}', union_format: Literal['any_of', 'primitive_type_array'] = 'any_of')[source]¶
Config to add fields to generated JSON schema for Settings.
- generate(schema, mode='validation')[source]¶
Generates a JSON schema for a specified schema in a specified mode.
- Args:
schema: A Pydantic model. mode: The mode in which to generate the schema. Defaults to ‘validation’.
- Returns:
A JSON schema representing the specified schema.
- Raises:
PydanticUserError: If the JSON schema generator has already been used to generate a JSON schema.
- class app.config.SynonymsStrategy(*values)[source]¶
Strategy to use for synonyms.
index: synonyms are implemented using the ElasticSearch synonyms sets strategy. It can be efficient but it is not well supported if you have a lot of taxonomies and a lot of languages. (ElasticSearch start will be very very slow)
none: synonyms are not automatically accounted for during requests (you still get the taxonomies separate indexes). This means your queries have to use the taxonomy id, instead of a synonym or a translation.
A “query” strategy might be implemented in the future, acting before sending the query to ElasticSearch.
- class app.config.TaxonomyConfig(*, sources: list[TaxonomySourceConfig], index: TaxonomyIndexConfig, preprocessor: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='The full qualified reference to the preprocessor\nto use before taxonomy entry import.\n\nThis class must inherit `app.indexing.BaseTaxonomyPreprocessor`\nand specialize the `preprocess` method.\n\nThis is used to adapt the taxonomy schema\nor to add specific fields for example.', examples=['app.openfoodfacts.TaxonomyPreprocessor'])] | None = None)[source]¶
Configuration of taxonomies, that is collections of entries with synonyms in multiple languages.
See [Explain taxonomies](../explain-taxonomies)
Field may be linked to taxonomies.
It enables enriching search with synonyms, as well as providing suggestions, or informative facets.
Note: if you define taxonomies, you must import them using [import-taxonomies command](../ref-python/cli.html#python3-m-app-import-taxonomies)
- class app.config.TaxonomyIndexConfig(*, name: str, number_of_shards: int = 4, number_of_replicas: int = 1)[source]¶
This is the configuration of the ElasticSearch index storing the taxonomies.
All taxonomies are stored within the same index.
It enables functions like auto-completion, or field suggestions as well as enrichment of requests with synonyms.