remove unnecessary env corrections and use correct include paths

[WinterMute]

base_tools was provided so people don't have to manually set path & variables for build tools.

The env variable substitution is wrong and producing incorrect paths on windows. If there's a problem with these variables then it's most likely because the wrong make binary is being used. We provide msys2 for the unix tools there and attempting to use different tools can & will cause problems.

The include paths weren't being set properly, I've corrected that too

supercedes #14

[Leseratte10]

Now that I get the chance to comment on such an issue again without the comment immediately being deleted and locked, I would like to write a couple sentences about that (commenting this both in the netslug repo and in the brainslug PR):

The binaries that clamtram linked to are not random binaries someone found on the internet. These are exactly identical original official binaries you have released in the past, either through github or through pacman, and they are 100% identical to the environment that is or was used to compile brainslug. Look, I apologize if my comments (on that gbatemp thread or on some github issues) from about a year ago have been a bit rude, but is it really necessary to A) block, delete, and lock any mention of previous versions, and B) block and ban me from ever commenting on any devkitPro repositories ever again just because I have an opinion that you don't agree with?

I appreciate the work you do with devkitPPC and the other packages included in the devkitPro toolchain, I really do. I also would like to thank you for fixing open-source tools like brainslug to work with newer versions. I gave you a "like" on your comment for that. However what I would appreciate even more if the devkitPro community could become a bit less hostile regarding the usage of older versions of the tools and libraries your toolchain provides. And I simultaneously gave you a dislike for the two critic points A) and B) mentioned above.

In regards to your comment about the Makefile - I know for a fact that MrBean used to compile Brainslug and netslug on a Windows machine, so while I agree with you that the path opt:/devkitpro is definitely wrong, it seems to have worked fine in the past. And it looks like your PR breaks it again, at least for the Linux-based CI (I didn't look at it too closely yet).

[asiekierka]

However what I would appreciate even more if the devkitPro community could become a bit less hostile regarding the usage of older versions of the tools and libraries your toolchain provides.

The principial problem is that it perpetuates fixed bugs and issues. Consider devkitARM and libnds, where old versions of the toolchain would not create DSi mode-compatible homebrew; or devkitPPC and libogc, where old versions of the toolchain would not create homebrew compatible with the Wii Motion Plus. This creates a bad experience for the end user.

Another issue is that it perpetuates bad practices. Due to the lack of high-quality (expensive) documentation, users will often look to open-source, publicly available projects for advice. Poor-quality code, as well as workarounds made for older toolchain/library versions both in code and the buildsystem, effectively lead them astray - towards worse-quality homebrew.

There's a lot of effort you've put into maintaining the mirror, bringing up CI and other stuff, but I often feel like it'd be more beneficial to work towards keeping the homebrew codebases up to date. This would benefit end users and other developers alike.

Regarding the "hostility" - even if I'm not a fan of the extreme measures sometimes taken, I also have to admit that devkitPro is a group which has consistently provided some of the highest-quality homebrew toolchains and libraries I have used (and I've worked with quite a few platforms), and the ruthless attitude towards trying to work around it rather than with it might have something to do with it.

[Leseratte10]

I have to say, that I/we do keep homebrew up-to-date. If you (and/or Wintermute) check the CI config file this repository uses, you can see that with the exception of devkitPPC (which is one version behind ...) all the libraries we are using are the most-recent, up-to-date versions. Nothing outdated or ancient at all.

And, I have to say, to WinterMute: Keeping your Homebrew Applications up-to-date is WAY easier if you have a safe and easy way to switch back and forth between the old (known-working) version and the new version (which you want to use in the future). That's way easier using a Github CI or a Dockerfile with hardcoded versions where you can keep one Dockerfile for the old and one Dockerfile for the new version, rather than having to update the full toolchain on your development machine and have no (supported) way of going back to the old versions.

[asiekierka]

I have to say, that I/we do keep homebrew up-to-date. If you (and/or Wintermute) check the CI config file this repository uses, you can see that with the exception of devkitPPC (which is one version behind ...) all the libraries we are using are the most-recent, up-to-date versions. Nothing outdated or ancient at all.

Which means your version of newlib may be out of date - and many areas of the libraries you updated by hand (for instance file I/O logic) rely on communicating with newlib for that purpose.

Frankenstein toolchains are a plague in many communities - there are so many obscure CPUs and platforms where the only way to get things working is a magical incantation of binutils, GCC, and tooling versions that's barely documented... If and when avoidable, it's best to avoid them.

[Leseratte10]

Hang on a minute, you misunderstood what I said. I never mismatched gcc and library versions. At some point in time (a couple months ago or so), all the versions we are currently using in the CI were the most recent versions, so there's no mismatching and/or library differences. I'm just saying that some time after that point at which we "froze" the versions, one new version of devkitPPC and no other versions of libogc and such have been released.

Also, to my knowledge, both gcc and newlib are included in the same "devkitPPC" package, so even if I did mismatch other library versions I'd still be using the correct newlib version that fits the compiler version.

[asiekierka]

That approach sounds more reasonable. It still doesn't resolve the initial notes I've had, however, and I disagree with it being called "up to date" when it isn't quite that.

Let me explain the problem with examples, though. If you search the URL of your mirror online, you'll find advice such as:

Replace the devkitarm folder with the devkitarm r45
<URL - snip>
Extract it to DevkitPro/devkitARM

Al descargarlo, les quedara un autoextractor, le dan clic encima y seleccionan el lugar donde lo extraerán.
O si quieren, le dan clic derecho encima del archivo y seleccionan "Extraer aquí" y se extraerá una carpeta llamada "devkitARM".

These encourage people to overwrite an existing, new installation of devkitPro tools only partially, based on the mirrors as provided - which can easily cause troubleshooting issues later on when said users inevitably seek advice on official platforms like the GitHub.

since this post, devkitpro updated. the devkitpro github only allows you to download the latest version. leseratte has backups of devkitpro 1.0.0 and 1.0.1 at <URL - snip>

This encourages people to follow an outdated tutorial by downgrading the version of the package manager utilized, as opposed to updating the knowledge in the tutorial.

I can see why a project maintainer would be annoyed by users relying on old toolchains guided there on misleading information. I know you're trying to warn people not to use your mirrors for bodges, but outside of areas such as old, carefully assembled homebrew CI it seems that they are absolutely used for bodges in the general case. Conversely, providing enterprise-grade LTS support - which means a bit more than just "keeping old versions available", as evident above - is certainly out of devkitPro's budget, and a lot of attempts to tell the users that what they're trying to do is a bodge that will hurt them in the long-term are often misread as hostility.

devkitPro is not even the only project where you can't easily downgrade. Many Linux distributions follow a "rolling release" model - which itself has proponents and critics - where there aren't concrete "releases" and no guaranteed way to roll your system back to how it was at a specific date unless you kept the packages and information around yourself.

[Leseratte10]

I agree that people overwriting random parts of their toolchain is a problem. But other than removing my mirror, there is nothing I can do to prevent that. On my download page, there's a note at the bottom of every page that people aren't supposed to use these for up-to-date projects. Maybe I should add a note there that you shouldn't mix toolchain versions either.

I clearly remember that before I made my mirror page in 2018, the situation was not better. Yes, the instructions didn't say "go to Leseratte's mirror and download X", they said "This tool has last been compiled with devkitPPC version X", and usually had a file hoster link to the windows-only versions of that devkitPPC version. I'd argue that that is worse than a clean, structured mirror.

In the end, if there is a tutorial that says "Download X and then do Y with this downloaded X" - if Y is something that is stupid (like overwriting and messing up your toolchain), it's not my fault for providing X. I didn't make these tutorials, and you won't find a single instance where I recommend messing up your existing toolchain. That's why you have the DEVKITPRO and DEVKITPPC environment variables - so you can have multiple toolchain versions in different locations, and use version A for one project and version B for another.

Also, it is correct that linux distributions like Arch and Manjaro are rolling-release and you can't easily downgrade - that's why it's really rare that you see Arch or Manjaro running on important servers, or in build environments where you want to compile reproducibly. For that, either Debian, or other commercial, non-rolling Linux distributions are used.

For building software you want reproduciblity: https://reproducible-builds.org/
You don't want one single commit in your source code to build differently yesterday vs today, because the toolchain Docker container was updated ...

[WinterMute]

In the end, if there is a tutorial that says "Download X and then do Y with this downloaded X" - if Y is something that is stupid (like overwriting and messing up your toolchain), it's not my fault for providing X. I didn't make these tutorials, and you won't find a single instance where I recommend messing up your existing toolchain. That's why you have the DEVKITPRO and DEVKITPPC environment variables - so you can have multiple toolchain versions in different locations, and use version A for one project and version B for another.

You providing X is facilitating this inappropriate procedure. Many projects will compile just fine with latest tools with minor modifications.

DEVKITPRO & DEVKITPPC environment variables are not provided for this purpose and should not be used in this way. These exist for legacy reasons from before we even had a basic installer. They're not actually needed any more and only still exist in our installations because they used to exist and removing them causes confusion.

Please stop taking our work and rehosting it elsewhere then using it to cause support issues like this.

For building software you want reproduciblity: https://reproducible-builds.org/
You don't want one single commit in your source code to build differently yesterday vs today, because the toolchain Docker container was updated ...

This has debateable merit. We do however currently have dated docker images although ideally code should be updated to work with latest releases where possible.

[asiekierka]

Maybe I should add a note there that you shouldn't mix toolchain versions either.

Given that these links were generally DDL links, these notes will have limited effectiveness.

In the end, if there is a tutorial that says "Download X and then do Y with this downloaded X" - if Y is something that is stupid (like overwriting and messing up your toolchain), it's not my fault for providing X.

Correct, but it doesn't mean the maintainers are not allowed to be frustrated with you for providing X which, more often than not, causes Y. It's like that note in rxvt-unicode about not providing support to Gentoo users - at some point, the author realized an incredibly high percentage of Gentoo setups were quite bodged and difficult to reproduce, so he just ceased providing support to Gentoo users period. Does this mean Gentoo, or many Gentoo users who worked with their system cleanly, did something wrong? No. Does that mean the author did something wrong? Likewise, no.

Also, it is correct that linux distributions like Arch and Manjaro are rolling-release and you can't easily downgrade - that's why it's really rare that you see Arch or Manjaro running on important servers, or in build environments where you want to compile reproducibly. For that, either Debian, or other commercial, non-rolling Linux distributions are used.

That is correct, but there is no non-rolling or commercial equivalent of devkitPro's toolchains, and using an outdated version of devkitPro's toolchains is more akin to using an outdated version of Arch Linux than to using an outdated version of RHEL.

[Leseratte10]

You providing X is facilitating this inappropriate procedure. Many projects will compile just fine with latest tools with minor modifications.

Me providing X is using the rights granted to me under the GPL and under other open-source licenses. It's not facilitating anything. If many projects would compile fine with the latest tools, then why do the tutorial creators write stuff like "go to this mirror and download this and that" when they could as well say "install devkitPPC the official way"? Why rely on an unofficial mirror if the official way worked?

We do however currently have dated docker images although ideally code should be updated to work with latest releases where possible.

That sounds like a solution that would be cleaner than what's currently done in this repository. It would mean using official sources to download the toolchain so you can be sure there's no mess (version mixup) involved, but still reproducibility and the assurance that you will still be able to build a project unmodified in a couple years (as long as the old Docker container versions don't also get deleted after some time). Now this is not my repository so it's not my decision to make, but I would assume that Chadderz would be more likely to merge this if it was pinned to a fixed version to ensure it works in the future, too, when the container's latest tag is updated. (Also, your CI config is missing the part where the resulting binaries are handled / uploaded).

If Docker containers for old versions of devkitPPC and other devkitPro toolchains had existed in 2018 I might not even have created my archive. The point of this archive never was to cause you additional work. I have nothing against you personally. It's just an unfortunate side effect. The point was that I wanted to be able to - and I wanted to give others the ability to - compile old Homebrew without spending hours digging into ancient code to figure out why it doesn't compile.

Given that these links were generally DDL links, these notes will have limited effectiveness.

That may be true, but the rest (what people write into their tutorials) is not something that's under my control.

Correct, but it doesn't mean the maintainers are not allowed to be frustrated with you for providing X which, more often than not, causes Y. It's like that note in rxvt-unicode about not providing support to Gentoo users - at some point, the author realized an incredibly high percentage of Gentoo setups were quite bodged and difficult to reproduce, so he just ceased providing support to Gentoo users period. Does this mean Gentoo, or many Gentoo users who worked with their system cleanly, did something wrong? No. Does that mean the author did something wrong? Likewise, no.

That is true, and it's a good point. Wintermute could add a note to his website / forum / issue tracker / whatever that people do not get any support unless they downloaded their toolchain from the official pacman in the most recent version. I would have absolutely no problem with that. I'm not doing something wrong by providing old versions of the devkitPro toolchain, users are not doing something wrong by downloading them if they NEED them to compile old homebrew, and Wintermute would not be doing anything wrong if he said "You are using an old version, please update and come back if the issue persists". Just like it would happen if you used a two-year old copy of Arch Linux.

What did NOT happen is: The developer of rxvt-unicode - I assume - didn't shout at the Gentoo developers to stop providing Gentoo because it causes him support issues. He just told the users that he can't support Gentoo. But that's not what happens here. What happens here would be similar to the rxvt-unicode dev requesting that Gentoo take down their OS because it causes him support issues.

That is correct, but there is no non-rolling or commercial equivalent of devkitPro's toolchains, and using an outdated version of devkitPro's toolchains is more akin to using an outdated version of Arch Linux than to using an outdated version of RHEL.

Correct. But even then, there's nobody who tries to PREVENT me from running outdated Arch Linux. All that happens is when I want support, they say "Hey, would you mind updating first, and report back if the issue still occurs".

[WinterMute]

Me providing X is using the rights granted to me under the GPL and under other open-source licenses. It's not facilitating anything.

You providing X is absolutely facilitating the use of X.

If many projects would compile fine with the latest tools, then why do the tutorial creators write stuff like "go to this mirror and download this and that" when they could as well say "install devkitPPC the official way"? Why rely on an unofficial mirror if the official way worked?

In a lot of cases because they simply don't understand how the toolchains work and either make incorrect assumptions about how things work or because people like you come along and paper over the cracks instead of fixing the actual underlying issue. The bodges and workarounds then get perpetuated everywhere. This is incredibly frustrating, horribly tedious and takes time away from actually maintaining the tools.

Now this is not my repository so it's not my decision to make, but I would assume that Chadderz would be more likely to merge this if it was pinned to a fixed version to ensure it works in the future, too, when the container's latest tag is updated. (Also, your CI config is missing the part where the resulting binaries are handled / uploaded).

The update to use docker was made to check that it actually works. We can update it to upload artifacts if that's needed.

If Docker containers for old versions of devkitPPC and other devkitPro toolchains had existed in 2018

The docker containers have been provided since 2018.

That is true, and it's a good point. Wintermute could add a note to his website / forum / issue tracker / whatever that people do not get any support unless they downloaded their toolchain from the official pacman in the most recent version. I would have absolutely no problem with that. I'm not doing something wrong by providing old versions of the devkitPro toolchain, users are not doing something wrong by downloading them if they NEED them to compile old homebrew, and Wintermute would not be doing anything wrong if he said "You are using an old version, please update and come back if the issue persists". Just like it would happen if you used a two-year old copy of Arch Linux.

You don't understand the problem. People using old tools and libraries and providing instructions to obtain unsupported binaries in order to compile a project that, in most cases, needs minor adjustments to work with latest tools causes support issues. Users don't turn up with problems using old tools, they sometimes turn up with problems compiling old projects using latest tools when we can, given the opportunity, usually provide the minor tweaks required.

We don't want to withhold support from the victims of bad tutorials and inappropriate practices. We want to provide tools that work OOB and allow people to get on with collaborating on homebrew without having to worry about the tools. When you provide old binaries, extract them manually, and create build environments that differ from the official tools you actively damage the relationship we have with homebrewers. Please stop.

[Leseratte10]

The docker containers have been provided since 2018.

Docker containers for DevkitPro versions released in or after 2018 have been provided since 2018, that's true ... but not for older versions from pre-2018 that I would have needed in 2018 to compile pre-2018 homebrew without updating it.

People ... providing instructions to obtain unsupported binaries in order to compile a project that,

Both the BUILDING instructions and the Readme in this repository, as well as the instructions and readme in the netslug repository, only provide links to your official page(s) and not to my mirror. Why would it matter to the end user what data source the CI system is using to build the binaries (which is the only place in this repo that even mentions my mirror)?

Project maintainers who say stuff like "go download old version X from Leseratte's mirror" are part of the issue. If my mirror didn't exist, they'd link to a filehoster where they upload version X themselves. Taking down my mirror wouldn't magically make developers update their software they haven't touched in years. It'd mean they'd include a file hoster download link to that particular toolchain version themselves.

[WinterMute]

What we'd like, as I said, is for people to be able to compile pretty much everything with a stock install. When people provide old binaries and instruct people to use them without even bothering to check if things still work with latest tools then users get locked to old versions for no good reason. If CI is using some random collection of binaries we may or may not have released at some point arranged so that it doesn't match a stock install then CI isn't ensuring that it works on the enviroment we supply. In this case in particular the Makefile was in fact broken and should have been fixed. If CI was working properly then the build should have failed.

Project maintainers who say stuff like "go download old version X from Leseratte's mirror" are part of the issue.

Of course they are. And you providing the mirror is another part. You would do more good by learning how things work and helping people use latest tools. Instead you facilitate and perpetuate practices which ultimately hinder everyone.

Please stop maintaining your mirror and encouraging people to use the binaries there.

[Chadderz121]

Well that was a lot of reading! I'm not going to comment on the wider debate here, but I'll focus on the specifics of this PR.

base_tools was provided so people don't have to manually set path & variables for build tools.

Perfect! That wasn't present when I wrote this Makefile - just base_rules which did a little too much. Very happy to change to using base_tools

The env variable substitution is wrong and producing incorrect paths on windows. If there's a problem with these variables then it's most likely because the wrong make binary is being used. We provide msys2 for the unix tools there and attempting to use different tools can & will cause problems.

Yes; the workaround was specifically included because (at the time I tested it) it worked with BOTH the supplied msys make in devkitPro as well as other makes people had lying around. It does however seem to have bit rotted. The workaround changes paths such as /c/devkitPro to c:/devkitPro but nowadays it's clearly changing paths such as /opt/devkitPro into opt:/devkitPro. Clearly it should be removed and users with multiple makes need to fix the underlying issue.

The include paths weren't being set properly, I've corrected that too

I will test later as this project has a bit of a funky setup. Specifically it comprises two parts; a normal Homebrew application and a series of 'modules' which are designed to run concurrently with an official Nintendo SDK game. Therefore, the include paths have to be quite carefully curated to separate out includes for libogc (et al.) and includes that mirror the SDK. Includes for the compiler such as <stdint.h> should be available to both.

The update to use docker was made to check that it actually works. We can update it to upload artifacts if that's needed

Yes; I don't think we should lose any functionality if we're modifying the CI script unless its truly necessary. In particular the new CI script isn't testing the modules can be built with modern devkitPro, which could still be an issue. Furthermore, I'll need to test it all actually still works as funky setups such as this project are liable to break silently more easily.

[WinterMute]

Yes; the workaround was specifically included because (at the time I tested it) it worked with BOTH the supplied msys make in devkitPro as well as other makes people had lying around. It does however seem to have bit rotted. The workaround changes paths such as /c/devkitPro to c:/devkitPro but nowadays it's clearly changing paths such as /opt/devkitPro into opt:/devkitPro. Clearly it should be removed and users with multiple makes need to fix the underlying issue.

We don't support the use of any make other than msys2 make on windows. We supply a preconfigured msys2 setup and also offer instructions for people who prefer to add the devkitPro repositories to an existing msys2 setup. We do this to help ensure a standardised build environment across all the OSes we support which helps to ensure people can collaborate on homebrew without worrying about or being hindered by differences in the OS they choose to use. Attempting to support people who want to use different setups just leads to fragmentation & the inability to build a project with a stock install.

Yes; I don't think we should lose any functionality if we're modifying the CI script unless its truly necessary. In particular the new CI script isn't testing the modules can be built with modern devkitPro, which could still be an issue. Furthermore, I'll need to test it all actually still works as funky setups such as this project are liable to break silently more easily.

Everything should now be cleaned up and working as before, including artifact upload. Let me know if anything is broken.

Please note: devkitPro is the organisation that supplies and maintains devkitPPC and the supporting libraries. We supply over 200 components to assist with homebrew development on around 9 consoles atm. You don't install devkitPro or build things with devkitPro. You use our tools and libraries for development.