✅ We are removing SEU colour support support in 3.0.0 ✅

[worksofliam]

By the time Code for IBM i 3.0.0 releases, we are planning on removing SEU colour support.

This is what it looked like.

We are still sometime away from 3.0.0. If you take a look at our milestones, there is a planned 2.11.0 release which still has some time to go. 3.0.0 will likely be sometime this year (2024). It also means that any new issues that arise with colour support will be closed, and going forward suggest disabling SEU colour support in the VS Code settings.

This means when you open up source code in Visual Studio Code, instead of the colours appearing, you will instead see the character equivalent of the byte.

You can track the issue here: #2056

Why are we doing this?

As time has gone by, SEU is really one of the only editors that has support for a colour system like this. We are not trying to be like SEU, so therefore we don't feel that it is worth the time maintaining such a system in Code for IBM i.

To fit in better with the general development ecosystem, as well as the move to UTF8, colour bytes (which is typically ebcdic) just don't fit in well.

As @JonFP pointed out:

There never was any "support" - it was just an accidental by-product of the way the 5250 datastream works

What are your options?

1. You can continue to use an older version of Code for IBM i that will have SEU colour support.
2. You can (and should) look into removing SEU colour bytes from your source code.

Leave feedback

If you are affected by this change, you are welcome to leave a comment here. We also welcome comments about how users can migrate away from SEU colors (e.g. stripped out the bytes to remove them).

[chrjorgensen]

You can easily remove SEU colour codes from source members by creating and running a SQL procedure - for more information, see this comment.

[JonFP]

I would suggest not to use wording like "that has support for a colour system" in any announcement. There never was any "support" - it was just an accidental by-product of the way the 5250 datastream works. Like many others I had been working with SEU for eons before I even knew it could be done!

In other words just tell 'em it doesn't work!

[thomprl]

So if you open a source with this hex code color will it still compile or have to be removed? And if it still compiles will it save the hex code and if someone opens in SEU still be there?

[chrjorgensen]

@thomprl I know two of the hexcodes used for SEU colouring is converted into the same UTF-8 value - so when the member is saved, the colour codes will be the same, i.e. two colours made be changed into only one colour.

This should not affect compilation, though, since colour codes were only used in comment lines.

[starbuck5250]

To clarify a little, columns the compiler ignores everything in any comment area, including columns 1-5 of fixed RPG, and right-hand comments. I have plenty of code where someone put an attribute byte in column 1 of say an F-spec. Ugh.

On a marketing note, I 100% agree with Jon. These aren't colour codes, they're 5250 attribute bytes. They were never WYSIWYG. Good riddance.

[jkyeung]

First, I want to reinforce Jon and Buck: To call it "supported" in SEU is... well, bluntly, I consider that to be a complete misunderstanding.

Even among experienced SEU users who deliberately used attribute bytes, a disturbing percentage of them didn't understand what they were doing.

Perhaps the clearest illustration of how accidental this "feature" is: Let's say you put an attribute byte in column 1 of the source, like Buck mentioned. If it's an RPGLE member, then by default SEU opens it horizontally scrolled such that column 6 is the leftmost visible column on the screen. The vast majority of SEU users do not bother to scroll those first five columns into view, they just leave it like that (which is, after all, what the SEU designers were intending and expecting). In this circumstance, any attribute bytes in the first five columns are not displayed, and thus are not in the 5250 data stream, and thus have no effect. Those lines will simply be the default green! (Well, characters on that line will be displayed with default attributes up until an on-screen attribute byte is encountered, which would still affect subsequent characters.)

Moreover, users who do not scroll those first five columns into view are prone to unknowingly propagating any junk that's in there (whether it's attribute bytes, coder initials, revision ID, or whatever), via copying those lines in the course of doing maintenance work on the member. (This is perhaps my least favorite quirk of SEU, and why putting stuff in columns 1 to 5 is such a horrible practice.)

[jkyeung]

Despite the strong consensus that VS Code should not attempt to emulate 5250 behavior when encountering attribute bytes, there is still a sense in which I believe they need to be "supported", which is: Since they are a reality in existing codebases, there should be some sensible defined handling of them.

Some issues:

1. They need to be reliably detected. They are not "colour codes" and they are not EBCDIC, per se. They are a specific range of hex values (x'20' through x'3F') and they need to be intercepted before any attempt to translate them to any text encoding, EBCDIC or otherwise.
2. We need to decide what to do with them when we encounter them. The options that come to mind are
   1. Convert them to blanks upon reading, which is destructive but simple. This action removes the need for any other decision-making or implementation. No special handling is required to write the blanks upon save. (I hope it goes without saying that closing the buffer without saving should leave the original source untouched, attribute bytes and all.)
   2. Leave them intact and write them back out unchanged upon save. I think this is how it's supposed to be working now, with 5250 color emulation disabled, but it's not working quite right. In particular, x'25' and x'3F' do not make the round trip. (x'25' becomes a linefeed and x'3F' becomes '\Z'.)

So, we're already seemingly close to supporting (ii). I hope it is not too hard to fix.

Incidentally, this procedure is also not quite right. I don't know why some of the possible attribute byte values were left out of the TRANSLATE operation.

[jkyeung]

Smartly enough, VS Code doesn't support blinking text.

I know. That is what @chrjorgensen said. I contend that (1) you could have simply done what ACS did and not blink those, and (2) you seem to have attempted to implement nondisplay (even though he said you didn't), which to me is no better or worse than attempting to implement blinking. Why not just be complete, if you're going to do it at all?

The bigger question right now, which I am posing to the community:

Is the current behavior when 5250 color emulation is disabled sufficient?

In my opinion, it isn't. I strongly believe that every single byte of source code on the IBM i should be able to survive a round trip (as though untouched), with no bytes added, changed, or removed unless explicitly done so by the user. The extension could offer facilities to make it easy to replace them with blanks, but that should either be explicitly invoked by the user, or if automated, preceded by a warning notification and an option to abort or open in read-only mode.

[worksofliam]

I do believe in Code for IBM i, we are not going to do anything with those bytes. We are just going to let VS Code render them as it used to (it used to display SUB, SYS, etc, characters in the place of them) and it can be up to the user to determine what to do with them.

Though, @chrjorgensen does provide a nice SQL routine to remove them.

[jkyeung]

I do believe in Code for IBM i, we are not going to do anything with those bytes. We are just going to let VS Code render them as it used to (it used to display SUB, SYS, etc, characters in the place of them) and it can be up to the user to determine what to do with them.

I don't know that that is sufficient. As I reported, there are at least 2 codes that do not survive the round trip. From my point of view, this does not satisfy the very basic principle that if you open a member in VS Code, and do nothing other than save it right back, the contents should be exactly the same as before, byte for byte. (I can see good arguments for updating the timestamp or leaving the timestamp alone; either would be fine with me.)

At the very, very minimum, the extension should give the user a warning along the lines of "sorry, we found 5250 attribute bytes in this code and we cannot guarantee that they will be written back correctly; do you want to proceed anyway, or abort?".

Though, @chrjorgensen does provide a nice SQL routine to remove them.

Except that it doesn't remove all of them. It should remove all of them.

[JDubbTX]

Just chiming in with my 2 cents.

• I actually liked that Code4i handled the color bytes, even if the implementation was incomplete. I'm not going to shed a tear over it, but it gave me a bullet point when talking to RDi folks in the category of here is something that RDi doesn't do. It was also kind of cool to see when opening up 30 year old source members something that someone long since retired set up just to make their comments a bit more impactful.
• I agree that the best alternative is to leave them alone. Don't alter them, full stop. The goofy representation that shows up at least makes you aware and gives you the choice of how to handle them, either use fancy sql to strip them out or just deal with it. It's really not anything to lose sleep over.

[jkyeung]

• I agree that the best alternative is to leave them alone. Don't alter them, full stop.

See my new issue.

[BrianGodsend]

As @JDubbTX points out, if this feature goes away, so be it. I personally find that folks that color tend to over-color to the point of dramatically reducing readability. However, every once in awhile, coloring is used sparingly and has a very impactful effect.

That said, as others have pointed out, simply opening and re-saving a member should not change 1-byte of the source code ... including the 5250 display attributes.

So, can you please clarify what removal of support really means? Are the 5250 display attributes going to be stripped from source? If not, will the full range of all display attribute codes (x20 - x3F and xA0-xBF) be preserved? If the codes are not being stripped and not being preserved, will you add a warning about codes being altered?

[worksofliam]

1. what removal of support really means? We will no longer highlight lines based on the hex codes.
2. Are the 5250 display attributes going to be stripped from source? No, but we will be offering a way to strip them in the UI.

[BrianGodsend]

Thank you for the clarification.

@chrjorgensen stated that some of the hex codes are not being preserved round-trip. That is, simply opening the source member and saving it may cause some of the hex codes to be changed. Will (can?) this behavior be fixed to preserve the hex codes? If the hex codes cannot be preserved, then will (can?) a warning be given to let the user know that they are about to make a change.

As for offering a way to strip the codes, I would suggest not only adding something in the UI, but also adding an option to silently allow the codes to be stripped automatically.

Again, thank you for taking the time to work through this issue and all the other support of this wonderful tool. Stepping back from this issue, it seems almost comical that we are discussing what level of support for 5250 display attributes embedded in source comments should be included in the extension. So, I can only imagine how difficult it is to not simply state it is not supported and you get-what-you-get (and don't through a fit) and move on to more productive/impactful issues.

[worksofliam]

Will (can?) this behavior be fixed to preserve the hex codes?

Honestly, no. I don't plan on supporting this at all. We cannot confirm that it will work due to the fact that the conversion between EBCDIC -> UTF8 (which happens on the server) removes those characters. The only way we could work around it is if we moved the EBCDIC translation to the client, which there is no way that we will ever do this due to the fact that there are a lot of EBCDIC code-pages and we've really only just solved a lot of major CCSIDs related to this.

We will add warnings where applicable. Thanks for these excellent questions.

cc @chrjorgensen

[jkyeung]

We cannot confirm that it will work due to the fact that the conversion between EBCDIC -> UTF8 (which happens on the server) removes those characters. The only way we could work around it is if we moved the EBCDIC translation to the client, which there is no way that we will ever do this due to the fact that there are a lot of EBCDIC code-pages and we've really only just solved a lot of major CCSIDs related to this.

I agree that it's not worth the effort. I can imagine ways to accomplish it, but they are all fairly complicated.

Even "moving the translation to the client" is more fraught than it sounds, because it's not really an issue of having to support lots of code pages. That part is kind of arbitrary grunt work that either has already been done (e.g. like this) or can be accomplished in relatively straightforward (if tedious) ways. The big issue I see is that these attribute bytes are intrinsically not part of ANY valid text encoding. (In other words, no matter how many encodings you support, these attribute bytes would break ALL of them!) That's the whole point! They aren't supposed to be valid characters. They abuse a feature specific to the 5250 protocol. In order to work with them in any sane way, you either have to be operating directly with the 5250 data stream (i.e. through a full-blown 5250 emulator) or do some pretty intense custom programming.

We will add warnings where applicable.

This is probably the only reasonable way to go. There are still some specifics to decide (like whether to automatically (with warning) replace those bytes with spaces, or make stripping them an on-demand user action, or make the behavior configurable via settings). I hope they get discussed here before being finalized.