Skip to content

Dataset match subdomain/v1 #15058

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
antoineaboufayssal wants to merge 2 commits into from
Closed

Dataset match subdomain/v1 #15058

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4a72047
detect/dataset: add match subdomain option
antoineaboufayssal Mar 18, 2026
2aa43ba
doc/userguide: add dataset match subdomain documentation
antoineaboufayssal Mar 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 29 additions & 3 deletions doc/userguide/rules/datasets.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ Syntax::
dataset:<set|unset|isset|isnotset>,<name> \
[, type <string|md5|sha256|ipv4|ip>, save <file name>, load <file name>, state <file name>, memcap <size>, hashsize <size>
, format <csv|json|ndjson>, context_key <output_key>, value_key <json_key>, array_key <json_path>,
remove_key];
remove_key, match subdomain];

type <type>
the data type: string, md5, sha256, ipv4, ip
Expand Down Expand Up @@ -112,7 +112,11 @@ array_key <key>
remove_key
if set, the JSON object pointed by value key will be removed
from the alert event

match subdomain
if set to ``subdomain``, enables hierarchical domain matching.
On lookup, the dataset walks up the domain label hierarchy until
a match is found. Only valid with ``isset``/``isnotset`` commands
and ``type string``. Best used with the ``dotprefix`` transform.

.. note:: 'type' is mandatory and needs to be set.

Expand All @@ -137,6 +141,28 @@ on domain names to find TLDs in the dataset ``dns-tld-seen``:

.. image:: dataset-examples/detect-unique-tlds.png

3. Block domains and all their subdomains using a blocklist dataset:

.. container:: example-rule

reject dns any any -> any any (msg:"Blocked domain"; dns.query; dotprefix; dataset:isset,blocked-domains, type string, match subdomain, load blocked-domains.lst; sid:8000003; rev:1;)

The ``match subdomain`` option walks up the domain hierarchy on each
lookup. Combined with ``dotprefix``, a query for ``mail.evil.com``
becomes ``.mail.evil.com`` and is checked against the dataset as:
``.mail.evil.com``, ``.evil.com``, ``.com``. If ``.evil.com`` is in the
dataset, the rule matches.

The dataset file should contain entries with a leading dot::

LmV2aWwuY29tCg==

which is the base64 encoding of ``.evil.com``.

When using ``ndjson`` format, use the raw dotted value in the JSON::

{"domain": ".evil.com"}

Notice how it is not possible to do certain operations alone with datasets
(example 2 above), but, it is possible to use a combination of other rule
keywords. Keep in mind the cost of additional keywords though e.g. in the
Expand Down Expand Up @@ -184,7 +210,7 @@ Syntax::

dataset:<isset|isnotset>,<name> \
[, type <string|md5|sha256|ipv4|ip>, load <file name>, format <json|ndjson>, memcap <size>, hashsize <size>, context_key <json_key> \
, value_key <json_key>, array_key <json_path>];
, value_key <json_key>, array_key <json_path>, match subdomain];

Example rules could look like::

Expand Down
64 changes: 62 additions & 2 deletions src/detect-dataset.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,21 @@ void DetectDatasetRegister (void)
sigmatch_table[DETECT_DATASET].Free = DetectDatasetFree;
}

/** \brief walk up the domain hierarchy looking for a match in a JSON dataset */
static DataJsonResultType DatajsonLookupSubdomain(
Dataset *set, const uint8_t *data, const uint32_t data_len)
{
DataJsonResultType r = { .found = false, .json = { .value = NULL, .len = 0 }, .hashdata = NULL };
for (uint32_t i = 1; i < data_len; i++) {
if (data[i] == '.') {
r = DatajsonLookup(set, data + i, data_len - i);
if (r.found)
return r;
}
}
return r;
}

/*
1 match
0 no match
Expand All @@ -76,6 +91,9 @@ static int DetectDatajsonBufferMatch(DetectEngineThreadCtx *det_ctx, const Detec
// PrintRawDataFp(stdout, data, data_len);
DataJsonResultType r = DatajsonLookup(sd->set, data, data_len);
SCLogDebug("r found: %d, len: %u", r.found, r.json.len);
if (!r.found && sd->match_subdomain) {
r = DatajsonLookupSubdomain(sd->set, data, data_len);
}
if (!r.found)
return 0;
if (r.json.len > 0) {
Expand All @@ -101,6 +119,9 @@ static int DetectDatajsonBufferMatch(DetectEngineThreadCtx *det_ctx, const Detec
// PrintRawDataFp(stdout, data, data_len);
DataJsonResultType r = DatajsonLookup(sd->set, data, data_len);
SCLogDebug("r found: %d, len: %u", r.found, r.json.len);
if (!r.found && sd->match_subdomain) {
r = DatajsonLookupSubdomain(sd->set, data, data_len);
}
if (r.found) {
DatajsonUnlockElt(&r);
return 0;
Expand All @@ -113,6 +134,19 @@ static int DetectDatajsonBufferMatch(DetectEngineThreadCtx *det_ctx, const Detec
return 0;
}

/** \brief walk up the domain hierarchy looking for a match in a dataset */
static int DatasetLookupSubdomain(Dataset *set, const uint8_t *data, const uint32_t data_len)
{
for (uint32_t i = 1; i < data_len; i++) {
if (data[i] == '.') {
int r = DatasetLookup(set, data + i, data_len - i);

@catenacyber catenacyber Mar 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder if we should limit the number of calls to DatasetLookup in case someone tricks an input like .............................. many times

@antoineaboufayssal antoineaboufayssal Mar 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could skip consecutive dots, then .............................. will have 0 lookups. But it wouldn't cover the case where the domain is like .a.a.a.a.a.a.a.a.a.a.a..... DNS names max at 253 bytes, so worst case is ~126 lookups. Is this fine? What do you think?

@victorjulien victorjulien Mar 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can use this for buffers that come from other data, like http.host where there isn't necessarily a length limit enforced

@antoineaboufayssal antoineaboufayssal Mar 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I'll add a skip for consecutive dots and a cap for the lookups. What's a reasonable cap?

if (r == 1)
return 1;
}
}
return 0;
}

/*
1 match
0 no match
Expand All @@ -132,6 +166,9 @@ int DetectDatasetBufferMatch(DetectEngineThreadCtx *det_ctx,
case DETECT_DATASET_CMD_ISSET: {
//PrintRawDataFp(stdout, data, data_len);
int r = DatasetLookup(sd->set, data, data_len);
if (r != 1 && sd->match_subdomain) {
r = DatasetLookupSubdomain(sd->set, data, data_len);
}
SCLogDebug("r %d", r);
if (r == 1)
return 1;
Expand All @@ -140,6 +177,9 @@ int DetectDatasetBufferMatch(DetectEngineThreadCtx *det_ctx,
case DETECT_DATASET_CMD_ISNOTSET: {
//PrintRawDataFp(stdout, data, data_len);
int r = DatasetLookup(sd->set, data, data_len);
if (r != 1 && sd->match_subdomain) {
r = DatasetLookupSubdomain(sd->set, data, data_len);
}
SCLogDebug("r %d", r);
if (r < 1)
return 1;
Expand Down Expand Up @@ -168,7 +208,7 @@ static int DetectDatasetParse(const char *str, char *cmd, int cmd_len, char *nam
enum DatasetTypes *type, char *load, size_t load_size, char *save, size_t save_size,
uint64_t *memcap, uint32_t *hashsize, DatasetFormats *format, char *value_key,
size_t value_key_size, char *array_key, size_t array_key_size, char *enrichment_key,
size_t enrichment_key_size, bool *remove_key)
size_t enrichment_key_size, bool *remove_key, bool *match_subdomain)
{
bool cmd_set = false;
bool name_set = false;
Expand Down Expand Up @@ -220,6 +260,13 @@ static int DetectDatasetParse(const char *str, char *cmd, int cmd_len, char *nam
*remove_key = true;
} else
return -1;
} else if (strcmp(key, "match") == 0) {
if (strcmp(val, "subdomain") == 0) {
Comment thread
catenacyber marked this conversation as resolved.
*match_subdomain = true;
} else {
SCLogError("unknown match mode '%s'", val);
return -1;
}
} else if (strcmp(key, "type") == 0) {
SCLogDebug("type %s", val);

Expand Down Expand Up @@ -473,6 +520,7 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
char array_key[SIG_JSON_CONTENT_KEY_LEN] = "";
char enrichment_key[SIG_JSON_CONTENT_KEY_LEN] = "";
bool remove_key = false;
bool match_subdomain = false;

if (DetectBufferGetActiveList(de_ctx, s) == -1) {
SCLogError("datasets are only supported for sticky buffers");
Expand All @@ -488,7 +536,7 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
if (!DetectDatasetParse(rawstr, cmd_str, sizeof(cmd_str), name, sizeof(name), &type, load,
sizeof(load), save, sizeof(save), &memcap, &hashsize, &format, value_key,
sizeof(value_key), array_key, sizeof(array_key), enrichment_key,
sizeof(enrichment_key), &remove_key)) {
sizeof(enrichment_key), &remove_key, &match_subdomain)) {
return -1;
}

Expand All @@ -513,6 +561,17 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
return -1;
}

if (match_subdomain) {
if (cmd != DETECT_DATASET_CMD_ISSET && cmd != DETECT_DATASET_CMD_ISNOTSET) {
SCLogError("'match subdomain' only supports isset/isnotset commands");
return -1;
}
if (type != DATASET_TYPE_STRING) {
SCLogError("'match subdomain' only supports type string");
return -1;
}
}

if ((format == DATASET_FORMAT_JSON) || (format == DATASET_FORMAT_NDJSON)) {
if (strlen(save) != 0) {
SCLogError("json format is not supported with 'save' or 'state' option");
Expand Down Expand Up @@ -571,6 +630,7 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
cd->set = set;
cd->cmd = cmd;
cd->format = format;
cd->match_subdomain = match_subdomain;
if ((format == DATASET_FORMAT_JSON) || (format == DATASET_FORMAT_NDJSON)) {
strlcpy(cd->json_key, enrichment_key, sizeof(cd->json_key));
}
Expand Down
1 change: 1 addition & 0 deletions src/detect-dataset.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ typedef struct DetectDatasetData_ {
Dataset *set;
uint8_t cmd;
DatasetFormats format;
bool match_subdomain; /* walk up domain hierarchy on lookup */

@catenacyber catenacyber Mar 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: placing bool after uint8_t leads to a more efficient memory layout

DataJsonType json;
char json_key[SIG_JSON_CONTENT_KEY_LEN];
void *id; /* pointer to the triggering signature */
Expand Down
Loading