app-layer: websockets protocol support by catenacyber · Pull Request #9989 · OISF/suricata

catenacyber · 2023-12-06T21:11:30Z

Link to redmine ticket:
https://redmine.openinfosecfoundation.org/issues/2695

Describe changes:

app-layer: websockets protocol support

Just a draft to show started work

SV_BRANCH=pr/1519

OISF/suricata-verify#1519

#9987 with more TODOs done

What is left todo :

keywords
reassembly ?

suricata-qa · 2023-12-07T01:30:03Z

WARNING:

field	baseline	test	%
	SURI_TLPR1_stats_chk
.tcp.pseudo	2810	3014	107.26%

Pipeline 16907

catenacyber · 2023-12-07T09:01:09Z

Also #9870 should be merged before this to avoid a short-lived output-json-websockets.c file ;-)

catenacyber · 2023-12-07T09:47:53Z

QA : it is expected to have a rise of pseudo packets on the protocol change from HTTP1 to WebSocket

victorjulien

Nice work. Left some comments inline.

victorjulien · 2023-12-07T11:04:29Z

etc/schema.json

                                "tls": {
                                    "$ref": "#/$defs/stats_applayer_error"
+                                },
+                                "websockets": {


rfc 6455 refers to it as "WebSocket" (singular), so lets use singular in our code and docs

victorjulien · 2023-12-07T11:06:13Z

rust/src/websockets/logger.rs

+
+fn log_websockets(tx: &WebSocketsTransaction, js: &mut JsonBuilder) -> Result<(), JsonError> {
+    js.open_object("websockets")?;
+    js.set_bool("mask", tx.pdu.mask)?;


would it make sense to log the mask value?

I do not think so.
Frame can be used to detect on it...

victorjulien · 2023-12-07T11:09:08Z

rust/src/websockets/websockets.rs

+                    self.transactions.push_back(tx);
+                }
+                Err(nom::Err::Incomplete(_)) => {
+                    // Not enough data. just ask for one more byte.


as the header seems to have a payload_len, could we use that here?

Also what is the max payload size?

A frame, per spec can be up to 9,223,372,036,854,775,807 bytes I believe.

frames the size of the universe, of course

but I guess it means we need to consume all payload data directly, even if incomplete. Process them the same as other streaming buffers like http body and file.data

frames the size of the universe, of course

In practice, most implementations don't allow extremely large payloads, I think 16kb even being on the extreme.. But we of course still need to deal with universe sized frames somehow.

as the header seems to have a payload_len, could we use that here?

Indeed, I was just lazy :-p

frames the size of the universe, of course

half the size of the universe :-p

other streaming buffers like http body and file.data

Are there other examples than http body ?
Another way could be to set a config limit (and skip the rest of the payload)

victorjulien · 2023-12-07T11:12:18Z

src/app-layer-htp.c

+                        }
+                        SCReturnStruct(APP_LAYER_OK);
+                    } else if (bstr_cmp_c_nocase(h->value, "WebSocket") == 0) {
+                        if (AppLayerProtoDetectGetProtoName(ALPROTO_WEBSOCKETS) == NULL) {


maybe we can cache this as a flag in the htp_state? Seems like a fairly expensive lookup of a static value (same for http2 a few lines up)

What do you want to cache ?
The header Upgrade value being WebSocket ?
(we compare it only once)

AppLayerProtoDetectGetProtoName does a table lookup, but the answer will always be the same. So we could look it up once and store the result

Oh. Why is this "table lookup" fairly expensive ?
To me, it looks negligible next to the string comparison

victorjulien · 2023-12-07T11:13:59Z

src/app-layer-htp.c

+                            break;
+                        }
+                        hstate->slice = NULL;
+                        if (AppLayerExpectationCreate(f, STREAM_TOCLIENT | STREAM_TOSERVER, f->sp,


surprised to see this. Expectations are about related but unique flows (so different 5 tuple), that is not the case here, right?

Websocket cannot be recognized by a fixed pattern, neither by a probing parser... (and they should not as we should not recognize this protocol without the HTTP1 handshake )
So, expectation looks the only option left...

Another way would be AppLayerRequestProtocolChange-ish call to force the protocol change... (instead of running TcpProtocolDetect)

Expectations are about related but unique flows (so different 5 tuple), that is not the case here, right?

Here, we have one 5-tuple flow, but expectations work here even if they were meant for ftp-data, so for related but different flows. They also work for a single flow changing protocol.

TL;DR This is a POC/draft and I will reach out about this again in a good version if I do not find anything better until then ;-)

Expectations create a new flow, so shouldn't be used here. It's the wrong API for the problem.

Forcing the protocol change, perhaps with some validation that it indeed looks like websock, sounds better.

Forcing the protocol change, perhaps with some validation that it indeed looks like websock, sounds better.

Will look into it (validation will be checking 3 reserved bits are 0, not so great)

Expectations create a new flow, so shouldn't be used here. It's the wrong API for the problem.

The way I understood is that expectation do not create a new flow.
A packet creates a new flow, which may be checked against current expectations to recognize its protocol.
What is wrong in my understanding ?

Indeed, not a flow but something worse: a entry in the ip_pair table, pretty expensive operation.

Now, that looks evil 👿
Thanks for the explanation :-)

Got a new hack for next PR iteration

Ticket: 2695 Introduces a device EnumStringU8 to ease the use of enumerations in protocols : logging the string out of the u8, and for detection, parsing the u8 out of the string

catenacyber · 2023-12-08T14:46:03Z

Replaced by #10014

catenacyber mentioned this pull request Dec 6, 2023

app-layer: websockets protocol support #9987

Closed

victorjulien requested changes Dec 7, 2023

View reviewed changes

catenacyber mentioned this pull request Dec 7, 2023

Adds test for websocket OISF/suricata-verify#1528

Closed

app-layer: websockets protocol support

82550d9

Ticket: 2695 Introduces a device EnumStringU8 to ease the use of enumerations in protocols : logging the string out of the u8, and for detection, parsing the u8 out of the string

catenacyber force-pushed the websockets-2695-v2 branch from 17f6ef0 to 82550d9 Compare December 8, 2023 14:42

catenacyber mentioned this pull request Dec 8, 2023

app-layer: websockets protocol support #10014

Closed

catenacyber closed this Dec 8, 2023

catenacyber mentioned this pull request Dec 14, 2023

app-layer: websockets protocol support #10060

Closed

Comments

Conversation

catenacyber commented Dec 6, 2023

Uh oh!

suricata-qa commented Dec 7, 2023

Uh oh!

catenacyber commented Dec 7, 2023

Uh oh!

catenacyber commented Dec 7, 2023

Uh oh!

victorjulien left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

catenacyber commented Dec 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants