Smtp server detection 1125 v2.1

[catenacyber]

Link to ticket: https://redmine.openinfosecfoundation.org/issues/
https://redmine.openinfosecfoundation.org/issues/1125

Describe changes:

• smtp server detection (ie to_client)
• ftp server detection (ie to_client)

SV_BRANCH=OISF/suricata-verify#1850

#11125 with code review and rust clippy fixes

[suricata-qa]

WARNING:

field	baseline	test	%
	SURI_TLPW1_stats_chk
.app_layer.flow.smtp	7556	7825	103.56%
	SURI_TLPR1_stats_chk
.app_layer.flow.smtp	335817	347669	103.53%
.app_layer.flow.failed_tcp	178240	161735	90.74%
.app_layer.tx.ftp	101030	95200	94.23%
.app_layer.error.smtp.parser	527	603	114.42%
.ftp.memuse	10637	2906	27.32%

Pipeline 20770

[victorjulien]

@ct0br0 can you isolate some of the new smtp failures and share them with @catenacyber ?

[catenacyber]

Would indeed like all TLPW flows for both SMTP and FTP for this PR and for master

[ct0br0]

that's much easier :D ill dm you in a few

[catenacyber]

can you isolate some of the new smtp failures and share them with @catenacyber ?

So, I would not say these are failures.

First tcp stream is like a scan : 3way handshake,
then server banner
220 BLU0-SMTP155.phx.gbl Microsoft ESMTP MAIL Service, Version: 6.0.3790.4675 ready at Wed, 19 Feb 2014 11:02:34 -0800
And client resets the connexion without sending any data.

With this PR, we classify the flow as SMTP, when master keeps unknown

[catenacyber]

Also tcp.stream eq 2shows
client sends junk
SMTP server sens banner

220 iltc1.iltc.br ESMTP Sendmail 8.14.4/8.14.4; Wed, 19 Feb 2014 16:02:39 -0300

end then answers to the junk like

500 5.5.1 Command unrecognized: "..(r.jje.Q&p%7.f2.Yp?.spLi.qf5?qs.Xq"

[victorjulien]

Great, and I agree that that pattern should lead to flows that have app proto smtp.

Did you also check why smtp errors increased?

[catenacyber]

There is one and only one flow which goes from tls to smtp because client sends TLS handshake client hello, and server replies with 220 banner + 500 error

Great, and I agree that that pattern should lead to flows that have app proto smtp.

Did you also check why smtp errors increased?

Because we parse more flows as SMTP, and we error on more because of unrecognized reply codes like 535 cf https://redmine.openinfosecfoundation.org/issues/6821

I see

  15 lolb 943 535 Authentication failed. Restarting authentication process.
  24 lolb 943 535 5.7.8 Error: authentication failed: 
  31 lolb 943 535 Authentication failed
  36 lolb 943 535 5.7.8 Error: authentication failed: authentication failure
  47 lolb 943 535 Incorrect authentication data

Besides 535, I see

   8 lolb 943 522
  18 lolb 943 454
  26 lolb 943 521
  52 lolb 943 530

[catenacyber]

I also see

   1 lol 400 4.5.2 Error: bad syntax
   1 lol 472 unusualz@prg-dc.dhl.com DNS A-record is empty

But do not find doc about these 400 and 472 SMTP reply codes...

[catenacyber]

New PR in #11193