build: namespace generated headers with zephyr/

[ycsin]

Namespaced the generated headers with zephyr/ to prevent potential conflict with other headers.

Introduced a Kconfig (CONFIG_LEGACY_GENERATED_INCLUDE_PATH) to support legacy generated header include paths in the mean time.

Documentation preview

Fixes #60377
Fixes #68035

• Architecture WG
• Missing changes to the N repos that depend on this are required in order to merge the PR
  • sof
    • zephyr: namespace the generated version.h - take 1 thesofproject/sof#8459
    • zephyr: namespace the generated version.h sof#34
  • percepio
    • zephyr: namespace the generated version.h percepio/percepio#5
    • zephyr: namespace the generated version.h percepio/percepio#6
    • zephyr: namespace the generated version.h percepio#10
    • zephyr: prefix generated header path with zephyr/ percepio/percepio#9
    • zephyr: prefix generated header path with zephyr/ percepio/percepio#8
    • zephyr: prefix generated header path with zephyr/ percepio#13
  • zephyr: remove autoconf.h includes hal_stm32#211
  • zephyr: prefix autoconf.h with zephyr/ hal_nxp#395
  • zephyr: prefix autoconf.h with zephyr/ lvgl#53
  • mcuboot
    • zephyr: prefix generated header path with zephyr/ mcu-tools/mcuboot#1965
    • zephyr: prefix generated header path with zephyr/ mcuboot#117

Automation script:

#!/usr/bin/env python

from pathlib import Path
import re

EXTENSIONS = ("c", "h", "cpp", "rst", "md", "S", "ld", "py", "svg")

HEADERS = [
    "app_version",
    # "autoconf",
    "cmake_intdef",
    "core-isa-dM",
    "devicetree_generated",
    "driver-validation",
    "kobj-types-enum",
    "linker-kobject-prebuilt-data",
    "linker-kobject-prebuilt-priv-stacks",
    "linker-kobject-prebuilt-rodata",
    "mcuboot_version",
    "offsets",
    "otype-to-size",
    "otype-to-str",
    "strerror_table",
    "strsignal_table",
    "syscall_list",
    "version",
    "zsr",
]

HEADERS_H = [hdr + ".h" for hdr in HEADERS]

for p in Path(".").glob("zephyr/**/*"):
    if not p.is_file() or p.suffix and p.suffix[1:] not in EXTENSIONS:
        continue

    # skip git & github folders
    if (str(p).startswith("zephyr/.git")):
        continue

    # skip release docs
    if (str(p).startswith("zephyr/doc/releases")):
        continue

    content = ""
    with open(p, 'r', newline='', errors="surrogateescape") as f:
        for line in f:
            # for lines in sources
            m = re.match(r"^(.*)#include <(.*)>(.*)$", line)
            # for lines in docs
            n = re.match(r"^(.*)include/generated/(.*)\.h(.*)$", line)
            if (m and (m.group(2).startswith("syscalls/") or m.group(2) in HEADERS_H)):
                content += (m.group(1) + "#include <zephyr/" +
                            m.group(2) + ">" + m.group(3) + "\n")
            elif (n and n.group(2) in HEADERS):
                content += (n.group(1) + "include/generated/zephyr/" +
                            n.group(2) + ".h" + n.group(3) + "\n")
            else:
                content += line

    with open(p, "w", newline='') as f:
        f.write(content)

Requirements for Treewide Changes

• The zephyr repository must apply the ‘treewide’ GitHub label to any issues or pull requests that are treewide changes
• The person proposing a treewide change must create an RFC issue describing the change, its rationale and impact, etc. before any pull requests related to the change can be merged
• The project’s Architecture Working Group (WG) must include the issue on the agenda and discuss whether the project will accept or reject the change before any pull requests related to the change can be merged (with escalation to the TSC if consensus is not reached at the WG)
• The Architecture WG must specify the procedure for merging any PRs associated with each individual treewide change, including any required approvals for pull requests affecting specific subsystems or extra review time requirements
• The person proposing a treewide change must email [email protected] about the RFC if it is accepted by the Architecture WG before any pull requests related to the change can be merged

[ycsin]

this PR is getting more involved than I initially expected

EDIT:
And CI is complaining about some lines that are not modified in this PR:

Run if [[ ! -s "compliance.xml" ]]; then
Error: See https://www.pylint.org/ for more details

 C03[25](https://github.com/zephyrproject-rtos/zephyr/actions/runs/6542835611/job/17766473397?pr=63973#step:9:26):Unnecessary parens after '=' keyword (superfluous-parens)
File:scripts/kconfig/kconfiglib.py
Line:2931
Column:0
 C0325:Unnecessary parens after '=' keyword (superfluous-parens)
File:scripts/kconfig/kconfiglib.py
Line:4310
Column:0
 C0325:Unnecessary parens after '=' keyword (superfluous-parens)
File:scripts/kconfig/kconfiglib.py
Line:4448
Column:0
Error: Process completed with exit code 1.

zephyr/scripts/kconfig/kconfiglib.py

Line 2931 in b272760

node.is_menuconfig = (t0 is _T_MENUCONFIG)

zephyr/scripts/kconfig/kconfiglib.py

Line 4310 in b272760

self._write_to_conf = (vis != 0)

zephyr/scripts/kconfig/kconfiglib.py

Line 4448 in b272760

self._write_to_conf = (vis != 0)

This is fixed in the last commit

[henrikbrixandersen]

I am not fond of having such a warning enabled by default. This will be emitted on every build, right?

[marc-hb]

Wait... there are other people who look at build warnings? @henrikbrixandersen, you and I should start a club!

I've seen Kconfig and Device tree warnings pretty much every time I build anything more complex than samples/hello_world/... neither of these tools seems to have any -Werror, which in my experience is the only way to make anyone care.

[marc-hb]

The warning will be printed only for people who should fix their code ASAP so printing it every time looks good to me.

[ycsin]

Otherwise the CONFIG_LEGACY_GENERATED_INCLUDE_PATH which is enabled by default now will just make everything passes silently. Most people might not notice that the headers are now relocated to zephyr/ unless they looked at their build directory or the migration guide.

On the other extreme, if we disable CONFIG_LEGACY_GENERATED_INCLUDE_PATH by default, the downstream apps will fail to build immediately, which I don't think is the right way to do in an OS of this scale.

The current implementation IMHO is the middle approach - we don't want to break downstream applications, but at the same time we want to make sure that developers are made aware of the changes, and given enough time to do their migration (as we do ours - some of the PRs aren't merged, west.yml not updated to point at them).

[ycsin]

@cfriedt @nordicjm @finikorg @fabiobaltieri @de-nordic @dcpleung @mbolivar-ampere
This PR is now ready (again), could you please take another look?

@gmarull this PR should align with #41543 and consistent with the header migration PR that you have done in the past (~v3.1.0?), could you please review?

[de-nordic]

Changes to nRF QSPI NOR look ok.

[fabiobaltieri]

This may use some coordination help from someone from @zephyrproject-rtos/release, plenty of dependencies, plus probably something will pop up after merge as well, worth keeping it in mind.

[ycsin]

ping assignee @tejlmand, PTAL, thanks!

[ycsin]

ping assignee @tejlmand, PTAL, thanks!

another ping @tejlmand

[ycsin]

🥳🍀🤞

[marc-hb]

Just FYI https://github.com/thesofproject/sof/actions/runs/9278328176/job/25529164408

FileNotFoundError:  No such file or directory:
  '/zep_workspace/build-lnl/zephyr/include/generated/autoconf.h'

That's because one Python script in SOF extracts values from autoconf.h after the build.

The C compilation was fine and the CONFIG_LEGACY_GENERATED_INCLUDE_PATH warning was printed by CMake as expected.

Just FYI.