Dictionaries and Iterators

[mgrant0]

I'm trying to extend the yara-json module which parses json using libjansson.

It currently defines 3 functions:

    declare_function("kv", "ss", "i", key_value);  /* return 1 if key==value otherwise 0 */
    declare_function("has_key", "s", "i", has_key);  /* return 1 if key exists */
    declare_function("has_key", "r", "i", has_key_r);  /* return 1 if regex of key exists */

Minimal, but a good base to start from. I'd like to return a dictionary of some subset of the json structure to iterate over in yara.

The docs don't seem to provide any example on how to do this though it says it's possible (https://yara.readthedocs.io/en/stable/writingrules.html?highlight=iterate#iterators). I looked at the pe module but it's not obvious to me what's going on.

Does anyone have a concise example how to make a dictionary in a module and how a function returns that or a pointer into that? It would be great to get an example into the docs for this because it's a great feature!

[wxsBSD]

If you're using a reasonable modern version you can use iterators, which are documented here: https://yara.readthedocs.io/en/stable/writingrules.html#iterators

It also talks about iterating over dictionaries. Here's an example of using it to print the pe.version_info dictionary:

import "console"
import "pe"

rule a {
  condition:
    for all k, v in pe.version_info:
      (console.log("Key: ", k) and console.log("\tValue: ", v))
}

[mgrant0]

I'm part way through trying to implement this and I hit a possible problem.

The module name is json.

If the JSON being read is like {"foo":"bar"} it's pretty clear to access foo, it's json.foo which is "bar".

But what if the JSON being read is an array or some other element without a name, for example [1,2,3]? How would you access the second element? json.[1]? it's a little weird. or i wonder if json[1] would be possible?

The situation becomes stranger when the JSON is a simple string or number such as "foo". Would you then access the value "foo" using json.?

I'm tempted to put a word in for the first part, for example "val". If the JSON is an object as in {"foo":"bar","baz":1} then the first part would be object and to access "foo" you would use json.val.foo. If the JSON contained just a string "foo", then you would use json.val and get "foo". Best I can think of at the moment. Any better suggestions?

[mgrant0]

I see our replies crossed, I just saw your reply after I posted my reply.

Yes, I totally agree now that I started digging into it. I see the problem with the names when there aren't names. I'm wondering if my idea of using "json.val" in front of things can work. Array of arrays like [1,2,[3,4]] get referred to like json.val.[2][1]. Your example [{"foo": 1}] would be referred to as json.val[0].foo. The top level object is an array named "val" and inside val, there is an element which is an object that is not named because it's array element 0, then the next level inside the object is an element whose name is "foo" and value 1.

[wxsBSD]

But the problem is this can get arbitrarily deep and arbitrarily complex. This is why I keep ending up at a selector language like what jq has for finding the parts of interest, rather than trying to model arbitrary json like you are.

Even if you use a selector function you need to be able to handle arbitrary return values which aren't known until after the selector has found the right values for you.

It's a complex problem for sure, and I don't mean to discourage you from exploring it. Just trying to point out potential other solutions that may result in a better user experience.

For example: {"foo": {"bar": 1}}

For that example I think this is a nice way to express it (and is directly inspired by jq):

json.query("foo.bar") == 1

This works when you know ahead of time that the query function will return an integer, but what happens when you use the same query on

{"foo": {"bar": "baz"}}

Now you need to overload the query function (which you can do) but you won't know which type to return (integer or string) until you actually parse the json in the query function.

I realize I barely answered your question so my apologies for that. Hopefully I've been able to shed some light on this and help you somehow. :)

[mgrant0]

You are helping. Just discussing this here is a huge help. Let's keep going...

Ok I now totally see your point about the types. Let's talk about using jq...

There is a not-so-well documented C API for jq: https://github.com/stedolan/jq/wiki/C-API:-jq-program-invocation. There's also a debian package libjq-dev which contains this lib. So it should in theory be easy to link in and call from a yara module in place of libjansson.

Yara supports only these return types from a function: string, integer, and float. But if you want to iterate over something, yara has dictionaries and arrays. There's also structure types but i'm not sure if they are iterable. But you can't return dictionaries, arrays or structures from functions.

This kind of gets back to one of my original questions: how do you return something iterable back to yara? From what I understood, a function can't do this (yet?).

From what I understand, to iterate over, or do things like 1 of them, the data has to be stored in something like json.val.foo.bar which is not access via a function which I could call something in libjq. This is where I got pushed down this path of unpacking the JSON into Yara's structures.

Another point against using json.val.foo.bar is you can't do regular expressions on the name like you can with a jq query or yara itself.

Do we need to propose some major change for yara 5.x to support this? Have any better suggestions?

[wxsBSD]

You are helping. Just discussing this here is a huge help. Let's keep going...

Ok I now totally see your point about the types. Let's talk about using jq...

There is a not-so-well documented C API for jq: https://github.com/stedolan/jq/wiki/C-API:-jq-program-invocation. There's also a debian package libjq-dev which contains this lib. So it should in theory be easy to link in and call from a yara module in place of libjansson.

Yup, this is exactly where I ended up. I wasn't convinced I liked the speed of jq compared to something like simdjson but as a POC I wasn't willing to care enough to change anything.

Yara supports only these return types from a function: string, integer, and float. But if you want to iterate over something, yara has dictionaries and arrays. There's also structure types but i'm not sure if they are iterable. But you can't return dictionaries, arrays or structures from functions.

Correct, and this was something I also identified. My initial thought was to limit it such that the function(s) only returned scalar types supported by YARA. If your query resulted in a list we would either flatten it to a string (which seems wrong) or return undefined. As a POC I thought that would be fine.

This kind of gets back to one of my original questions: how do you return something iterable back to yara? From what I understood, a function can't do this (yet?).

You identified a critical point I was hoping you would get to. ;)

You can't return an array or struct via a function (at least last time I looked).

From what I understand, to iterate over, or do things like 1 of them, the data has to be stored in something like json.val.foo.bar which is not access via a function which I could call something in libjq. This is where I got pushed down this path of unpacking the JSON into Yara's structures.

You are correct here too.

I think if you're willing to eliminate some functionality and only support queries that result in a scalar then you have a better shot at making it work and identifying areas of improvement (like returning arrays via functions). Basically, don't let perfect be the enemy of good while you're exploring here.

Do we need to propose some major change for yara 5.x to support this? Have any better suggestions?

It's possible but until you have a demonstrated need for them I wouldn't really pursue them beyond this thread. I think building it and demonstrating the current limitations is better than designing improvements that are theoretical.

[mgrant0]

Ok so I have confirmed more of what you said. I wrote some functions to unpack the json in the module_load() function and here's what I learned:

I learned that the definitions in module_definitions() are defined BEFORE the file is read in module_load(). One idea we talked about above was creating a structure like json.val.foo.bar. This does not work because the file isn't read (the __context isn't even created) until after the yara rules file is parsed. json.val.anything returns

error: rule "jsonrule" in test.yara(5): invalid field name "val"

Even though I could create val in the module_load() function, it's too late in the game. As you say, you need to pre-load the structure of the JSON for this to even stand a chance at working. Like a true catch-22. I'm abandoning this line of thought here.

It sure would be nice if functions could return something iterable. But let's see how far I get without that.

Correct, and this was something I also identified. My initial thought was to limit it such that the function(s) only returned scalar types supported by YARA. If your query resulted in a list we would either flatten it to a string (which seems wrong) or return undefined. As a POC I thought that would be fine.

At this point I have to agree. However, if the query resulted in something larger than a scalar type, I would return the JSON as a string. The reason I would do this is at minimum, it allows me to then stick that in a variable and then match a RE on that whole block of JSON. It's not ideal but it's functional.

libjansson does not support any query into the JSON it loads. I'm going to look around for a more robust JSON lib. Even if I find something that returns some structure, I can "flatten" it, or rather cat it together back into a single string and return that as a string (a "sub-JSON" string, a smaller JSON block). If you have any suggestions of a decent C JSON lib, I'd like to know.

With that, it should make yara quite minimally useful for my purpose (and perhaps others) with JSON formatted data.

[mgrant0]

Do you mean a file which could contain json formatted data as part of a file but not the whole file? For example something like:

; example of json embedded in some other file
foo = { [ { "bar": 1 } ] }

Running this through the module I wrote bombs out because the entire file isn't valid JSON.

What if, instead of reading in the file in module_load(), what if I provided a separate function to load the JSON from a string? Where would you call such a function? In the Strings definitions section? In the Condition section? Alternatively, and this might be the best solution (but certainly the slowest) would be to pass in the JSON each time you call the query function, for example:

import "json"

rule jsonrule {
    strings:
        $j = /\{.*\}/
    condition:
        json.query($j, "/foo/bar/%d/baz",0) == "hi"
}

so $j extracts some part of the file which has json, in this contrived example {...}. Then $j is fed into json.query which would first extract the JSON into a structure in memory and then run json_object_get_string on the in memory representation, then free the in memory representation. Clearly if you were going to call json.query() many times with the same $j it's wasteful but this could just be the price to pay for this functionality. I'm not sure how to extract the json into memory and then subsequently call json.query() on it.

[mgrant0]

By the way, I tried to call json.query() up in the strings: section with my existing code and it doesn't work. Apparently you can only call module functions in the condition: section. Is this your understanding as well?

[wxsBSD]

Sorry, been traveling a bit so I missed this.

Do you mean a file which could contain json formatted data as part of a file but not the whole file?

Nope, that would be a lot more work, to find valid JSON in arbitrary data and parse it. What I meant was how you can use the -x argument to wire up a file as module data, which can then be handled via the module. We were using this (at least I think we were) to write rules that combined content of the binary (using normal yara strings) with information from the sandbox report using $string and sandbox.write_file(/foo.exe/) conditions.

By the way, I tried to call json.query() up in the strings: section with my existing code and it doesn't work. Apparently you can only call module functions in the condition: section. Is this your understanding as well?

Yes, that is correct.

[mgrant0]

I think I see what cuckoo does to use the -x option. It stores the JSON in an already defined structure. From the point of view, it's the same as when loading in the actual file being read. You would need to have already defined that structure before reading module_load() was called. It's the same exact problem with reading in the file itself.

In my experimentation, the -x json:foo.json, what this does is pass foo.json in the module_data param of the module_load() function call.

As far as I can tell, the -x doesn't help. Have I missed something?

Do you know of any other languages like Yara that I might want to look at? Something more specific to JSON perhaps?

[wxsBSD]

I think I see what cuckoo does to use the -x option. It stores the JSON in an already defined structure. From the point of view, it's the same as when loading in the actual file being read. You would need to have already defined that structure before reading module_load() was called. It's the same exact problem with reading in the file itself.

Yes, you still have the problems associated with not knowing the structure of the json ahead of time. My point in suggesting why a generic json parser might be useful is you can use the module data trick to pass json to that module AND pass arbitrary data to the rest of yara. This way you can express things like "$string and json.foo()" where the string is searched for in the arbitrary data and the json can be something related to it (like say a sandbox report).

This won't solve your problems as you've found out but it does combine static detections from yara with arbitrary json expressions over the output of some other tool. For example, being able to express something like "pe.imphash() == "foo" and json.writes_file(/bar/)" is an interesting capability.

In my experimentation, the -x json:foo.json, what this does is pass foo.json in the module_data param of the module_load() function call.

Correct. But for the reason I highlighted above it is probably the best option here.

As far as I can tell, the -x doesn't help. Have I missed something?

It doesn't address the problem of arbitrary json structures, but it does open up possibilities if you can live with the limitations already identified.

Do you know of any other languages like Yara that I might want to look at? Something more specific to JSON perhaps?

jq? Maybe something that kicks off a yara scan and executed arbitrary jq expressions in parallel and then combines the results?