Pop3 protocol detection 6366 v4

[catenacyber]

Link to redmine ticket:
https://redmine.openinfosecfoundation.org/issues/6366

Describe changes:

• pop3 protocol detection

OISF/suricata-verify#1481

SV_BRANCH=pr/1481

Rebase of #9874

First preliminary part for #8892 and https://redmine.openinfosecfoundation.org/issues/1125

This will require a QA rebaseline

After that :

• See first commits of Smtp server detection 1125 v17 #8892 about generic protocol detection and see if we can craft tests to identify these bugs
• Make eve.json stats field about flows match the count of flow with app_proto because of so many corner cases
• Add FTP and SMTP server side detection

[codecov]

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (9fe00ff) 82.52% compared to head (b50f75a) 82.51%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10373      +/-   ##
==========================================
- Coverage   82.52%   82.51%   -0.01%     
==========================================
  Files         978      978              
  Lines      272148   272158      +10     
==========================================
- Hits       224595   224581      -14     
- Misses      47553    47577      +24     

Flag	Coverage Δ
fuzzcorpus	63.59% <71.42%> (+<0.01%)	⬆️
suricata-verify	61.87% <71.42%> (-0.02%)	⬇️
unittests	62.83% <50.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[suricata-qa]

Information:

ERROR: QA failed on SURI_TLPW2_autofp_suri_time.

ERROR: QA failed on IPS_AFP_drop_chk.

field	baseline	test	%
	SURI_TLPW2_autofp_stats_chk
.uptime	101	111	109.9%
	SURI_TLPW1_stats_chk
.app_layer.flow.ftp	52	43	82.69%
.app_layer.tx.ftp	819	188	22.95%
.app_layer.error.ftp.gap	2	0	-
.app_layer.error.ftp.parser	2	0	-
.ftp.memuse	348	3	0.86%
	SURI_TLPR1_stats_chk
.ftp.memuse	11385	10637	93.43%
	IPS_AFP_stats_chk
.ips.blocked	1395360	747360	53.56%
.ips.drop_reason.flow_drop	1296000	680400	52.5%
.ips.drop_reason.applayer_error	32400	0	-
.flow.end.state.established	583199	550799	94.44%
.flow.end.state.closed	1016272	1048672	103.19%
.flow.end.tcp_state.established	201960	169560	83.96%
.flow.end.tcp_state.closed	1016272	1048672	103.19%
.app_layer.flow.ftp	33480	1080	3.23%
.app_layer.tx.ftp	131760	2160	1.64%
.app_layer.error.ftp.parser	32400	0	-
	TREX_GENERIC_stats_chk
.app_layer.flow.ftp	14871	0	-
.app_layer.tx.ftp	59484	0	-
.app_layer.error.ftp.parser	14871	0	-

Pipeline 18416

[victorjulien]

I suspect the IPS drop differences are because of the exception policy no longer applying to the mis-identified traffic. But I think it would be helpful to have the exception policy stats merged first to confirm.

[catenacyber]

exception policy stats

cc @jufajardini are you the one working on this ? Ticket number to link ?

[jufajardini]

exception policy stats

cc @jufajardini are you the one working on this ? Ticket number to link ?

Hi there, here it is, sorry, missed the notification for this: https://redmine.openinfosecfoundation.org/issues/5816

[jufajardini]

exception policy stats

cc @jufajardini are you the one working on this ? Ticket number to link ?

Suri PR has been merged, I hope it helps. #10785
Please let me know if there's more I could do here :)

[catenacyber]

Cool, rebased in #10890