Managing the database

Updating the databases

For production, your copy of the database should stay updated. In order to do that, we provide a docker image for the updating tool. This image is provided as a tar file.

To import the image into your docker instance, use the following commands:

$ gunzip updater-sdk-vxxx.tar.gz
$ docker image load -i updater-sdk-vxxx.tar

You can then run the updater in your docker environment.

$ docker run --rm olfeo-oem-updater:v0.8.0 --version
nexus-updater version sdk/v0.8.0-beae5f24

The updater should have access to your Redis database and be provided with your login and password.

You can also provide these using command line arguments, but also using environnement variables. You can mix and match, for instance:

$ docker run --rm olfeo-oem-updater:v0.8.0 -e "OLFEO_OEM_SDK_USERNAME=client" -e "OLFEO_OEM_SDK_PASSWORD=$password" --redis-dsn rediss://my-redis-instance.local/0 --update-url https://api.oem.olfeo.eu

The process will start the update. At first, the process can take quite a while. The progress is saved so the process will start where it left off if stopped and restarted.

Once the database is up to date, the process ends. You can then run the update process again at regular intervals to keep your database up-to-date. You can use a cron job for that.

Important

Make sure you only have one instance of the update process running: having multiple instances will not make it faster.

Customizing the database prefix

You can define a custom prefix for all Redis keys managed by the SDK. This helps prevent key conflicts when the Redis database is shared across multiple projects. It also allows you to support multiple versions of your database by assigning a unique prefix to each version.

Caution

All components of the SDK must be configured with the same prefix. This includes both the Database instance and the Updater process.

While it is a common convention to separate the prefix from the rest of the key using the colon (:) character in redis, the SDK does not enforce any specific format. If you prefer to follow this convention, you can add a trailing colon to your prefix od: as a prefix.

If defined, the custom databse prefix must be used in:

The database configuration

You can define the key prefix during initialization using the appropriate helper function, like this:

import "gitlab.olfeo.tech/data-tools/nexus/sdk/database/redis"

...

func createDbInstance() {
   ...
   db, err := redis.NewDatabase(ctx, redisClient, redis.WithKeyPrefix("od:"))
You can define the key prefix during initialization using the appropriate helper function, like so:}

The updater process configuration

The updater process must also be aware of the prefix. You can set it using one of the following methods:

As an extra command line parameter:

$ docker run --rm -e "OLFEO_OEM_SDK_USERNAME=client" -e "OLFEO_OEM_SDK_PASSWORD=$password" \
  olfeo-oem-updater:v1.1.0 --redis-dsn rediss://my-redis-instance.local/0 --redis-prefix od:

As an environment variable:

$ docker run --rm -e "OLFEO_OEM_SDK_USERNAME=client" -e "OLFEO_OEM_SDK_PASSWORD=$password" \
  -e "OLFEO_OEM_SDK_REDIS_PREFIX=od:" olfeo-oem-updater:v1.1.0 \
  --redis-dsn rediss://my-redis-instance.local/0

Olfeo-tools

The olfeo-tool binary, used to inject custom domain data into the database for testing, also requires the prefix.

When using a prefix, make sure to include it in the command line, like this:

$ olfeo-tool add-test-domain --redis-dsn rediss://my-redis-instance.local/0 --redis-prefix od: ...

Resetting the database

This section outlines how to reset the local Redis database, including options for resetting the entire database or targeting specific parts of the dataset.

Reset the Complete Database

To completely reset the database and remove all stored data, use the following Redis command:

$ FLUSHDB

This will start the database from scratch, clearing all existing data.

Reset a Specific Part of the Dataset

If you only need to reset specific parts of the dataset, you can selectively manage the data associated with a given synchronization status.

The synchronization process status between the local database and the API server is stored under a specific key in the Redis database, identified as: status. To get the current status, run the following command:

$ HGETALL status

This key contains a map that associates data type IDs with the last synchronization dates. Each date indicates the most recent update for the corresponding data type.

The table below describes the existing data types and their respective IDs:

Table 1. Data Types and Their Identifiers

Data Type	Id
Domain	`d`
Domain Category	`c`
Theme	`t`
Application Category	`ca`
Application	`a`
Application Domain	`da`
IP Range Application	`ia`
Logo	`l`

Use the updater tool to resynchronize a specific data type by clearing the respective key in the status map.

For instance, to resynchronize all applications, the following command can be used:

$ HDEL status a

Olfeo OEM documentation