Olfeo OEM documentation

Beware: Domain field is hashed!

For security purposes, the domain field inside each Entry sub-message contains the hexadecimal digest of the hash of the actual domain name. This means that the stored value is not the plain FQDN itself but the result of hashing the domain using a specific algorithm and key.

This applies to the following message types: olfeo.database.v1.ExportedDomainInfo and olfeo.database.v1.ExportApplicationDomainInfo.

Default algorithm is xx64 as it represents a fine balance between compute needs, security and speed at runtime.

Obtaining the hashing algorithm and key

The hashing algorithm and key depend on the dataset you are working with (sample, production, legacy...).

Use the GetashInfo endpoint of Olfeo OEM's Update API to get the hashing algorithm and the hashing key that is specific to your files. This is stable and should not change over time.

However, it is a good idea to implement some way of checking hash method and value and updating it in your product.

NB: you will need to retrieve an API Key first in order to use this endpoint.

Hashing domains

Use teh appropriate hashing algorithm and key to hash the key and domain you want to get info about.. For instance, if you are using our sample file and want to get the category for the domain linuxcommand.org, you can do the following operation:

# The xxhash library provides an implementation of the xx64 hashing algorithm
import xxhash
KEY = "14dacd9b-5006-42f5-a8d6-825240a90b56" # This is the static key for all 
samples
DOMAIN = "linuxcommand.org"
hash = xxhash.xxh64_hexdigest(KEY + DOMAIN)
assert(hash == "9f688147060c5cab")

Our sample data always use the xx64 hashing algorithm ; hashing key is 14dacd9b-5006- 42f5-a8d6-825240a90b56

In order to keep the database manageable given the great number of domain names in the database, Olfeo OEM uses a hierarchical approach in which subdomains can inherit the category of their parent domain. However, in some cases this inheritance is not correct, the Olfeo database has a method to provide a more accurate results in such cases.

The general algorithm to find the category of a particular domain is to search the database starting from the full domain name to the top level domain, stopping as soon as a match is found. at the first match.

This process continue until a match is found.

Note: when a match occurs, you must then check the prefix field of the matching entry. If the value of this field is DOMAIN_PREFIX_STATUS_PREFIX, then the categoty of that domain does not apply to its subdomains. If you were looking for this domain specifically, then it is its category. However, if you were looking for one of its subdomains, the category do not apply and you should consider that subdomain as not categorized by the Olfeo database.

This mechanism exist to support domains that have very large breadth of content, such as blog hosting platforms, where the content of each subdomain can not be inferred from its hosting platform.