Next/20230530/v9

Makefile.am

@@ -54,7 +54,7 @@ endif
 	@echo "updated and managed with the suricata-update tool."
 	@echo ""
 	@echo "For more information please see:"
-	@echo "  https://suricata.readthedocs.io/en/latest/rule-management/index.html"
+	@echo "  https://docs.suricata.io/en/latest/rule-management/index.html"
 	@echo ""
 
 install-library:

README.md

@@ -12,12 +12,12 @@ Introduction
 Installation
 ------------
 
-https://suricata.readthedocs.io/en/latest/install.html
+https://docs.suricata.io/en/latest/install.html
 
 User Guide
 ----------
 
-You can follow the [Suricata user guide](https://suricata.readthedocs.io/en/latest/) to get started.
+You can follow the [Suricata user guide](https://docs.suricata.io/en/latest/) to get started.
 
 Contributing
 ------------

doc/INSTALL

@@ -2,4 +2,4 @@ See doc/userguide/install.rst
 
 An hosted version of this can be found at Read The Docs:
 
-https://suricata.readthedocs.io/en/latest/install.html
+https://docs.suricata.io/en/latest/install.html

doc/Setting_up_IPSinline_for_Linux.txt

@@ -1,5 +1,5 @@
 Autogenerated on 2012-11-29
-from - https://suricata.readthedocs.io/en/latest/setting-up-ipsinline-for-linux.html
+from - https://docs.suricata.io/en/latest/setting-up-ipsinline-for-linux.html
 
 
 Setting up IPS/inline for Linux

doc/userguide/configuration/includes.rst

@@ -0,0 +1,56 @@
+.. _includes:
+
+Includes
+========
+
+A Suricata configuration file (typically
+``/etc/suricata/suricata.yaml``) may include other files allowing a
+configuration file to be broken into multiple files. The *special*
+field name ``include`` is used to include one or more files.
+
+The contents of the *include* file are inlined at the level of the
+``include`` statement. *Include* fields may also be included at any
+level within a mapping.
+
+Including a Single File
+-----------------------
+
+::
+
+    include: filename.yaml
+
+Including Multiple Files
+------------------------
+
+::
+
+    include:
+      - filename1.yaml
+      - filename2.yaml
+
+Include Inside a Mapping
+------------------------
+
+::
+
+    vars:
+      address-groups:
+        include: address-groups.yaml
+
+where ``address-groups.yaml`` contains::
+
+    %YAML 1.1
+    ---
+    HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]"
+
+is the equivalent of::
+
+    vars:
+      address-groups:
+        HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]"
+
+.. note:: Suricata versions less than 7 required multiple ``include``
+    statements to be specified to include more than one file. While
+    Suricata 7.0 still supports this it will issue a deprecation
+    warning. Suricata 8.0 will not allow multiple ``include``
+    statements at the same level as this is not allowed by YAML.

doc/userguide/configuration/index.rst

@@ -11,3 +11,4 @@ Configuration
    dropping-privileges
    landlock
    systemd-notify
+   includes

doc/userguide/devguide/codebase/contributing/contribution-process.rst

@@ -203,7 +203,7 @@ If you are writing or updating *documentation pages*, please:
 * when adding diagrams or images, we prefer alternatives that can be generated
   automatically, if possible;
 * bear in mind that our documentation is published on `Read the Docs <https:/
-  /suricata.readthedocs.io/en/latest/#suricata-user-guide>`_ and can also be
+  /docs.suricata.io/en/latest/#suricata-user-guide>`_ and can also be
   built to pdf, so it is important that it looks good in such formats.
 
 

doc/userguide/devguide/codebase/unittests-c.rst

@@ -19,7 +19,7 @@ If you would like to compile Suricata with unit tests, enter the following durin
 
    ./configure --enable-unittests
 
-The unit tests specific command line options can be found at `Command Line Options <https://suricata.readthedocs.io/en/suricata-6.0.3/command-line-options.html#unit-tests>`_.
+The unit tests specific command line options can be found at `Command Line Options <https://docs.suricata.io/en/suricata-6.0.3/command-line-options.html#unit-tests>`_.
 
 Example:
 You can run tests specifically on flowbits. This is how you should do that::
@@ -42,7 +42,7 @@ This will be very verbose. You can also add the ``SC_LOG_OP_FILTER`` to limit th
     SC_LOG_LEVEL=Debug SC_LOG_OP_FILTER="(something|somethingelse)" suricata -u
 
 This example will show all lines (debug, info, and all other levels) that contain either something or something else.
-Keep in mind the `log level <https://suricata.readthedocs.io/en/latest/manpages/suricata.html#id1>`_  precedence: if you choose *Info* level, for instance, Suricata won't show messages from the other levels.
+Keep in mind the `log level <https://docs.suricata.io/en/latest/manpages/suricata.html#id1>`_  precedence: if you choose *Info* level, for instance, Suricata won't show messages from the other levels.
 
 Writing Unit Tests - C codebase
 ===============================

doc/userguide/devguide/extending/app-layer/app-layer-frames.rst

@@ -7,7 +7,7 @@ Application Layer Frame Support
 Baseline
 ========
 
-- `Suricata rules format <https://suricata.readthedocs.io/en/latest/rules/intro.html>`_
+- `Suricata rules format <https://docs.suricata.io/en/latest/rules/intro.html>`_
 
 General Concepts
 ================

doc/userguide/upgrade.rst

@@ -57,6 +57,15 @@ Logging changes
   ``ike.ikev2.errors`` and ``ike.ikev2.notify``.
 - FTP DATA metadata for alerts are now logged in ``ftp_data`` instead of root.
 - Alert ``xff`` field is now logged as ``alert.xff`` for alerts instead of at the root.
+- Protocol values and their names are built into Suricata instead of using the system's ``/etc/protocols`` file. Some names and casing may have changed
+  in the values ``proto`` in ``eve.json`` log entries and other logs containing protocol names and values.
+  See https://redmine.openinfosecfoundation.org/issues/4267 for more information.
+
+Deprecations
+~~~~~~~~~~~~
+- Multiple "include" fields in the configuration file will now issue a
+  warning and in Suricata 8.0 will not be supported. See
+  :ref:`includes` for documentation on including multiple files.
 
 Other changes
 ~~~~~~~~~~~~~
@@ -66,12 +75,6 @@ Other changes
 - SWF decompression in http has been disabled by default. To change the default see :ref:`suricata-yaml-configure-libhtp`. Users with configurations from previous releases may want to modify their config to match the new default.
   See https://redmine.openinfosecfoundation.org/issues/5632 for more information.
 
-Logging changes
-~~~~~~~~~~~~~~~
-- Protocol values and their names are built into Suricata instead of using the system's ``/etc/protocols`` file. Some names and casing may have changed
-  in the values ``proto`` in ``eve.json`` log entries and other logs containing protocol names and values.
-  See https://redmine.openinfosecfoundation.org/issues/4267 for more information.
-
 Upgrading 5.0 to 6.0
 --------------------
 - SIP now enabled by default

lua/fast.lua

@@ -10,7 +10,7 @@
 
 -- To learn more about all the API functions suricata provides for your lua scripts
 -- and the lua output extension in general see:
--- http://suricata.readthedocs.io/en/latest/output/lua-output.html
+-- http://docs.suricata.io/en/latest/output/lua-output.html
 
 function init()
     local needs     = {}

rust/src/dns/log.rs

@@ -399,7 +399,7 @@ pub fn dns_print_addr(addr: &Vec<u8>) -> std::string::String {
 
 /// Log SOA section fields.
 fn dns_log_soa(soa: &DNSRDataSOA) -> Result<JsonBuilder, JsonError> {
-    let mut js = JsonBuilder::new_object();
+    let mut js = JsonBuilder::try_new_object()?;
 
     js.set_string_from_bytes("mname", &soa.mname)?;
     js.set_string_from_bytes("rname", &soa.rname)?;
@@ -415,7 +415,7 @@ fn dns_log_soa(soa: &DNSRDataSOA) -> Result<JsonBuilder, JsonError> {
 
 /// Log SSHFP section fields.
 fn dns_log_sshfp(sshfp: &DNSRDataSSHFP) -> Result<JsonBuilder, JsonError> {
-    let mut js = JsonBuilder::new_object();
+    let mut js = JsonBuilder::try_new_object()?;
 
     let mut hex = Vec::new();
     for byte in &sshfp.fingerprint {
@@ -432,7 +432,7 @@ fn dns_log_sshfp(sshfp: &DNSRDataSSHFP) -> Result<JsonBuilder, JsonError> {
 
 /// Log SRV section fields.
 fn dns_log_srv(srv: &DNSRDataSRV) -> Result<JsonBuilder, JsonError> {
-    let mut js = JsonBuilder::new_object();
+    let mut js = JsonBuilder::try_new_object()?;
 
     js.set_uint("priority", srv.priority as u64)?;
     js.set_uint("weight", srv.weight as u64)?;
@@ -444,7 +444,7 @@ fn dns_log_srv(srv: &DNSRDataSRV) -> Result<JsonBuilder, JsonError> {
 }
 
 fn dns_log_json_answer_detail(answer: &DNSAnswerEntry) -> Result<JsonBuilder, JsonError> {
-    let mut jsa = JsonBuilder::new_object();
+    let mut jsa = JsonBuilder::try_new_object()?;
 
     jsa.set_string_from_bytes("rrname", &answer.name)?;
     jsa.set_string("rrtype", &dns_rrtype_string(answer.rrtype))?;
@@ -516,7 +516,7 @@ fn dns_log_json_answer(
     js.set_string("rcode", &dns_rcode_string(header.flags))?;
 
     if !response.answers.is_empty() {
-        let mut js_answers = JsonBuilder::new_array();
+        let mut js_answers = JsonBuilder::try_new_array()?;
 
         // For grouped answers we use a HashMap keyed by the rrtype.
         let mut answer_types = HashMap::new();
@@ -527,7 +527,7 @@ fn dns_log_json_answer(
                 match &answer.data {
                     DNSRData::A(addr) | DNSRData::AAAA(addr) => {
                         if !answer_types.contains_key(&type_string) {
-                            answer_types.insert(type_string.to_string(), JsonBuilder::new_array());
+                            answer_types.insert(type_string.to_string(), JsonBuilder::try_new_array()?);
                         }
                         if let Some(a) = answer_types.get_mut(&type_string) {
                             a.append_string(&dns_print_addr(addr))?;
@@ -540,31 +540,31 @@ fn dns_log_json_answer(
                     | DNSRData::NULL(bytes)
                     | DNSRData::PTR(bytes) => {
                         if !answer_types.contains_key(&type_string) {
-                            answer_types.insert(type_string.to_string(), JsonBuilder::new_array());
+                            answer_types.insert(type_string.to_string(), JsonBuilder::try_new_array()?);
                         }
                         if let Some(a) = answer_types.get_mut(&type_string) {
                             a.append_string_from_bytes(bytes)?;
                         }
                     }
                     DNSRData::SOA(soa) => {
                         if !answer_types.contains_key(&type_string) {
-                            answer_types.insert(type_string.to_string(), JsonBuilder::new_array());
+                            answer_types.insert(type_string.to_string(), JsonBuilder::try_new_array()?);
                         }
                         if let Some(a) = answer_types.get_mut(&type_string) {
                             a.append_object(&dns_log_soa(soa)?)?;
                         }
                     }
                     DNSRData::SSHFP(sshfp) => {
                         if !answer_types.contains_key(&type_string) {
-                            answer_types.insert(type_string.to_string(), JsonBuilder::new_array());
+                            answer_types.insert(type_string.to_string(), JsonBuilder::try_new_array()?);
                         }
                         if let Some(a) = answer_types.get_mut(&type_string) {
                             a.append_object(&dns_log_sshfp(sshfp)?)?;
                         }
                     }
                     DNSRData::SRV(srv) => {
                         if !answer_types.contains_key(&type_string) {
-                            answer_types.insert(type_string.to_string(), JsonBuilder::new_array());
+                            answer_types.insert(type_string.to_string(), JsonBuilder::try_new_array()?);
                         }
                         if let Some(a) = answer_types.get_mut(&type_string) {
                             a.append_object(&dns_log_srv(srv)?)?;