Skip to content

Dataset match subdomain/v2 #15073

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
antoineaboufayssal wants to merge 2 commits into from
Closed

Dataset match subdomain/v2 #15073

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6b2138d
detect/dataset: add match subdomain option
antoineaboufayssal Mar 20, 2026
4bff9c5
doc/userguide: add dataset match subdomain documentation
antoineaboufayssal Mar 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 29 additions & 3 deletions doc/userguide/rules/datasets.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ Syntax::
dataset:<set|unset|isset|isnotset>,<name> \
[, type <string|md5|sha256|ipv4|ip>, save <file name>, load <file name>, state <file name>, memcap <size>, hashsize <size>
, format <csv|json|ndjson>, context_key <output_key>, value_key <json_key>, array_key <json_path>,
remove_key];
remove_key, match subdomain];

type <type>
the data type: string, md5, sha256, ipv4, ip
Expand Down Expand Up @@ -112,7 +112,11 @@ array_key <key>
remove_key
if set, the JSON object pointed by value key will be removed
from the alert event

match subdomain
if set to ``subdomain``, enables hierarchical domain matching.
On lookup, the dataset walks up the domain label hierarchy until
a match is found. Only valid with ``isset``/``isnotset`` commands
and ``type string``. Best used with the ``dotprefix`` transform.

.. note:: 'type' is mandatory and needs to be set.

Expand All @@ -137,6 +141,28 @@ on domain names to find TLDs in the dataset ``dns-tld-seen``:

.. image:: dataset-examples/detect-unique-tlds.png

3. Block domains and all their subdomains using a blocklist dataset:

.. container:: example-rule

reject dns any any -> any any (msg:"Blocked domain"; dns.query; dotprefix; dataset:isset,blocked-domains, type string, match subdomain, load blocked-domains.lst; sid:8000003; rev:1;)

@jufajardini jufajardini Jun 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is leading to a CI failure for Check rules doc: Error: detect-dataset: failed to set up dataset 'blocked-domains'. [DetectDatasetSetup:detect-dataset.c:640]

https://github.com/OISF/suricata/actions/runs/27315514634/job/80695160734?pr=15601#step:22:12

@antoineaboufayssal antoineaboufayssal Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it because of load blocked-domains.lst? It's trying to find the file relative to the path of the rule file itself. Examples 1 and 2 use state instead of load.

@victorjulien victorjulien Jun 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this just may show a limitation of the check. I don't think it is aware of how datasets needs a file to load here. I'd say the check is wrong, not this doc example.

@catenacyber catenacyber Jun 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a fix #15605


The ``match subdomain`` option walks up the domain hierarchy on each
lookup. Combined with ``dotprefix``, a query for ``mail.evil.com``
becomes ``.mail.evil.com`` and is checked against the dataset as:
``.mail.evil.com``, ``.evil.com``, ``.com``. If ``.evil.com`` is in the
dataset, the rule matches.

The dataset file should contain entries with a leading dot::

LmV2aWwuY29tCg==

which is the base64 encoding of ``.evil.com``.

When using ``ndjson`` format, use the raw dotted value in the JSON::

{"domain": ".evil.com"}

Notice how it is not possible to do certain operations alone with datasets
(example 2 above), but, it is possible to use a combination of other rule
keywords. Keep in mind the cost of additional keywords though e.g. in the
Expand Down Expand Up @@ -184,7 +210,7 @@ Syntax::

dataset:<isset|isnotset>,<name> \
[, type <string|md5|sha256|ipv4|ip>, load <file name>, format <json|ndjson>, memcap <size>, hashsize <size>, context_key <json_key> \
, value_key <json_key>, array_key <json_path>];
, value_key <json_key>, array_key <json_path>, match subdomain];

Example rules could look like::

Expand Down
74 changes: 72 additions & 2 deletions src/detect-dataset.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@
#define DETECT_DATASET_CMD_ISNOTSET 2
#define DETECT_DATASET_CMD_ISSET 3

#define DATASET_SUBDOMAIN_MAX_LOOKUPS 126

@catenacyber catenacyber Mar 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, where did you find this constant ?

@antoineaboufayssal antoineaboufayssal Mar 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A valid domain name is limited to 253 bytes (whether it's dns, http host, tls sni, ...), giving at most 126 labels (1 char + 1 dot per label). Buffers like http.host are not length enforced, but anything beyond that limit is malformed or malicious and shouldn't match the dataset entry.

@catenacyber catenacyber Mar 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool

A valid domain name is limited to 253 bytes

Where did you find this ?

@antoineaboufayssal antoineaboufayssal Mar 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RFC 1035 defines it for DNS. (it's actually 255 bytes but a length byte and a null terminator byte are needed, hence the 253)

So maybe my statement up there wasn't 100% accurate :) While an http.host can exceed the 253 limit, it would not be a "globally resolvable domain". Perhaps used for an internal/private network?


static int DetectDatasetSetup (DetectEngineCtx *, Signature *, const char *);
void DetectDatasetFree (DetectEngineCtx *, void *);

Expand All @@ -61,6 +63,26 @@ void DetectDatasetRegister (void)
sigmatch_table[DETECT_DATASET].Free = DetectDatasetFree;
}

/** \brief walk up the domain hierarchy looking for a match in a JSON dataset */
static DataJsonResultType DatajsonLookupSubdomain(
Dataset *set, const uint8_t *data, const uint32_t data_len)
{
DataJsonResultType r = {
.found = false, .json = { .value = NULL, .len = 0 }, .hashdata = NULL
};
int lookups = 0;
for (uint32_t i = 1; i < data_len; i++) {
if (data[i] == '.' && data[i - 1] != '.') {
if (++lookups > DATASET_SUBDOMAIN_MAX_LOOKUPS)
break;
r = DatajsonLookup(set, data + i, data_len - i);
if (r.found)
return r;
}
}
return r;
}

/*
1 match
0 no match
Expand All @@ -76,6 +98,9 @@ static int DetectDatajsonBufferMatch(DetectEngineThreadCtx *det_ctx, const Detec
// PrintRawDataFp(stdout, data, data_len);
DataJsonResultType r = DatajsonLookup(sd->set, data, data_len);
SCLogDebug("r found: %d, len: %u", r.found, r.json.len);
if (!r.found && sd->match_subdomain) {
r = DatajsonLookupSubdomain(sd->set, data, data_len);
}
if (!r.found)
return 0;
if (r.json.len > 0) {
Expand All @@ -101,6 +126,9 @@ static int DetectDatajsonBufferMatch(DetectEngineThreadCtx *det_ctx, const Detec
// PrintRawDataFp(stdout, data, data_len);
DataJsonResultType r = DatajsonLookup(sd->set, data, data_len);
SCLogDebug("r found: %d, len: %u", r.found, r.json.len);
if (!r.found && sd->match_subdomain) {
r = DatajsonLookupSubdomain(sd->set, data, data_len);
}
if (r.found) {
DatajsonUnlockElt(&r);
return 0;
Expand All @@ -113,6 +141,22 @@ static int DetectDatajsonBufferMatch(DetectEngineThreadCtx *det_ctx, const Detec
return 0;
}

/** \brief walk up the domain hierarchy looking for a match in a dataset */
static int DatasetLookupSubdomain(Dataset *set, const uint8_t *data, const uint32_t data_len)
{
int lookups = 0;
for (uint32_t i = 1; i < data_len; i++) {
if (data[i] == '.' && data[i - 1] != '.') {
if (++lookups > DATASET_SUBDOMAIN_MAX_LOOKUPS)
break;
int r = DatasetLookup(set, data + i, data_len - i);
if (r == 1)
return 1;
}
}
return 0;
}

/*
1 match
0 no match
Expand All @@ -132,6 +176,9 @@ int DetectDatasetBufferMatch(DetectEngineThreadCtx *det_ctx,
case DETECT_DATASET_CMD_ISSET: {
//PrintRawDataFp(stdout, data, data_len);
int r = DatasetLookup(sd->set, data, data_len);
if (r != 1 && sd->match_subdomain) {
r = DatasetLookupSubdomain(sd->set, data, data_len);
}
SCLogDebug("r %d", r);
if (r == 1)
return 1;
Expand All @@ -140,6 +187,9 @@ int DetectDatasetBufferMatch(DetectEngineThreadCtx *det_ctx,
case DETECT_DATASET_CMD_ISNOTSET: {
//PrintRawDataFp(stdout, data, data_len);
int r = DatasetLookup(sd->set, data, data_len);
if (r != 1 && sd->match_subdomain) {
r = DatasetLookupSubdomain(sd->set, data, data_len);
}
SCLogDebug("r %d", r);
if (r < 1)
return 1;
Expand Down Expand Up @@ -168,7 +218,7 @@ static int DetectDatasetParse(const char *str, char *cmd, int cmd_len, char *nam
enum DatasetTypes *type, char *load, size_t load_size, char *save, size_t save_size,
uint64_t *memcap, uint32_t *hashsize, DatasetFormats *format, char *value_key,
size_t value_key_size, char *array_key, size_t array_key_size, char *enrichment_key,
size_t enrichment_key_size, bool *remove_key)
size_t enrichment_key_size, bool *remove_key, bool *match_subdomain)
{
bool cmd_set = false;
bool name_set = false;
Expand Down Expand Up @@ -220,6 +270,13 @@ static int DetectDatasetParse(const char *str, char *cmd, int cmd_len, char *nam
*remove_key = true;
} else
return -1;
} else if (strcmp(key, "match") == 0) {
if (strcmp(val, "subdomain") == 0) {
*match_subdomain = true;
} else {
SCLogError("unknown match mode '%s'", val);
return -1;
}
} else if (strcmp(key, "type") == 0) {
SCLogDebug("type %s", val);

Expand Down Expand Up @@ -473,6 +530,7 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
char array_key[SIG_JSON_CONTENT_KEY_LEN] = "";
char enrichment_key[SIG_JSON_CONTENT_KEY_LEN] = "";
bool remove_key = false;
bool match_subdomain = false;

if (DetectBufferGetActiveList(de_ctx, s) == -1) {
SCLogError("datasets are only supported for sticky buffers");
Expand All @@ -488,7 +546,7 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
if (!DetectDatasetParse(rawstr, cmd_str, sizeof(cmd_str), name, sizeof(name), &type, load,
sizeof(load), save, sizeof(save), &memcap, &hashsize, &format, value_key,
sizeof(value_key), array_key, sizeof(array_key), enrichment_key,
sizeof(enrichment_key), &remove_key)) {
sizeof(enrichment_key), &remove_key, &match_subdomain)) {
return -1;
}

Expand All @@ -513,6 +571,17 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
return -1;
}

if (match_subdomain) {
if (cmd != DETECT_DATASET_CMD_ISSET && cmd != DETECT_DATASET_CMD_ISNOTSET) {
SCLogError("'match subdomain' only supports isset/isnotset commands");
return -1;
}
if (type != DATASET_TYPE_STRING) {
SCLogError("'match subdomain' only supports type string");
return -1;
}
}

if ((format == DATASET_FORMAT_JSON) || (format == DATASET_FORMAT_NDJSON)) {
if (strlen(save) != 0) {
SCLogError("json format is not supported with 'save' or 'state' option");
Expand Down Expand Up @@ -571,6 +640,7 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst
cd->set = set;
cd->cmd = cmd;
cd->format = format;
cd->match_subdomain = match_subdomain;
if ((format == DATASET_FORMAT_JSON) || (format == DATASET_FORMAT_NDJSON)) {
strlcpy(cd->json_key, enrichment_key, sizeof(cd->json_key));
}
Expand Down
1 change: 1 addition & 0 deletions src/detect-dataset.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
typedef struct DetectDatasetData_ {
Dataset *set;
uint8_t cmd;
bool match_subdomain;
DatasetFormats format;
DataJsonType json;
char json_key[SIG_JSON_CONTENT_KEY_LEN];
Expand Down
Loading