Stack Error?

[Candurin]

Just started getting this error while away on vacation this week:

ApiError: Message cleared from outbound buffer: 165,33,15,16,168,36,15,0,0,36,0,0,0,0,0,0,0,0,0,0,0,17,0,0,0,0,0,0,0,0,0,0,0,0,255,255,255,255,0,0,1,0,5,205
at RS485Port.clearOutboundBuffer (/home/pi/nodejs-poolController/controller/comms/Comms.ts:916:23)
at RS485Port.clearBuffer (/home/pi/nodejs-poolController/controller/comms/Comms.ts:901:61)
at RS485Port.closeBuffer (/home/pi/nodejs-poolController/controller/comms/Comms.ts:902:64)
at RS485Port.closeAsync (/home/pi/nodejs-poolController/controller/comms/Comms.ts:809:37)
at RS485Port.openAsync (/home/pi/nodejs-poolController/controller/comms/Comms.ts:535:26)
at Timeout. (/home/pi/nodejs-poolController/controller/comms/Comms.ts:831:17)

If you have any ideas that would be great. I have rebooted pentair intellicenter as well as my pi numerous times to no effect.

[rstrouse]

Which IntelliCenter fw version is installed? This is simply trying to turn on a circuit. I hope they didn't start pushing updates. The last one was pulled because it was a bleeding booger.

[Candurin]

Still on 1.064. No changes made on my end.

[rstrouse]

Please check that at the OCP.

…

Sent from my iPhone

On Jul 23, 2023, at 4:06 PM, Candurin ***@***.***> wrote:  Still on 1.064. No changes made on my end. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[Candurin]

I’m not sure how to capture anything… how do I do that? Sorry!

[rstrouse]

https://github.com/tagyoureit/nodejs-poolController/wiki/How-to-capture-all-packets-for-issue-resolution#packet-capture

[Candurin]

Hopefully this works!
replay.zip

And of course, now I don't know how to get my configuration back (that part is not in the instructions) as it seems to have been reset to a default state.

[rstrouse]

The configuration will return as soon as we figure out why there are no communications on RS485. It is empty as in nothing is being transmitted over the hardware. Lets start with the which adapter do you have and how is it connected?

[Candurin]

I have this RS485 connector:

https://www.amazon.com/gp/product/B07SD65BVF

and I am using this to change over to CAT5 for the long run:

https://www.amazon.com/gp/product/B07WKKVZRF

I am also using these for surge protection:

https://www.amazon.com/gp/product/B07GBLFFNK

I have not changed anything hardware-wise since installing all this 3 years ago.

[Candurin]

Here are some photos to explain further:

[rstrouse]

Unfortunately something isn't working as it was. The RS485 data coming through is almost all 1s with no 0s. I suspect the USB rs485 adapter. Pull it from the pi, wait about 15 seconds then plug it back in. If you go into the gears menu in dashPanel then select Controller --> Comms tab and expand the Primary port you can view its status.

[Candurin]

When I took out the cable I got the disconnected error and here is what I see now:

[Candurin]

I am accessing it through VNC. I have turned off bluetooth as well.

[rstrouse]

What are the results of the above commands?

[Candurin]

So... I did go back into the directory and did "nom start". I am seeing this:

pi@pi:~/nodejs-poolController $ npm start

[email protected] start
npm run build && node dist/app.js

[email protected] build
tsc

Init state for Pool Controller
[7/26/2023, 9:34:08 PM] info: The current git branch output is master
[7/26/2023, 9:34:08 PM] info: The current git commit output is a8acb2a
[7/26/2023, 9:34:09 PM] info: Starting up SSDP server
[7/26/2023, 9:34:09 PM] info: Server is now listening on 0.0.0.0:4200 - 192.168.1.50:4200
[7/26/2023, 9:34:09 PM] info: Serial port: /dev/ttyUSB0 request to open successful 9600b 8-none-1
[7/26/2023, 9:34:09 PM] info: Checking njsPC versions...
[7/26/2023, 9:34:09 PM] info: Starting Pool System intellicenter
[7/26/2023, 9:34:09 PM] info: Auto-backup initialized Last Backup: 1969-12-31T19:00:00.000-0500
[7/26/2023, 9:34:21 PM] info: [9:34:21 PM] 192.168.1.42 GET /state/all?null {}
[7/26/2023, 9:34:21 PM] info: [9:34:21 PM] 192.168.1.42 GET /app/config/log {}
[7/26/2023, 9:34:21 PM] info: [9:34:21 PM] 192.168.1.42 GET /config/all?null {}
[7/26/2023, 9:34:21 PM] info: [9:34:21 PM] 192.168.1.42 GET /app/config/options/backup?null {}
[7/26/2023, 9:34:21 PM] info: [9:34:21 PM] 192.168.1.42 GET /state/appVersion?null {}
[7/26/2023, 9:34:25 PM] info: [9:34:25 PM] 192.168.1.42 GET /config/all?null {}
[7/26/2023, 9:34:25 PM] info: [9:34:25 PM] 192.168.1.42 GET /config/options/general?null {}
[7/26/2023, 9:34:27 PM] info: [9:34:27 PM] 192.168.1.42 GET /app/options/interfaces?null {}
[7/26/2023, 9:34:29 PM] info: [9:34:29 PM] 192.168.1.42 GET /config/options/rs485?null {}
[7/26/2023, 9:35:28 PM] info: [9:35:28 PM] 192.168.1.42 GET /state/all?null {}
[7/26/2023, 9:35:28 PM] info: [9:35:28 PM] 192.168.1.42 GET /app/config/log {}
[7/26/2023, 9:35:28 PM] info: [9:35:28 PM] 192.168.1.42 GET /config/all?null {}
[7/26/2023, 9:35:28 PM] info: [9:35:28 PM] 192.168.1.42 GET /app/config/options/backup?null {}
[7/26/2023, 9:35:28 PM] info: [9:35:28 PM] 192.168.1.42 GET /state/appVersion?null {}
[7/26/2023, 9:35:33 PM] info: [9:35:33 PM] 192.168.1.42 GET /state/all?null {}
[7/26/2023, 9:35:33 PM] info: [9:35:33 PM] 192.168.1.42 GET /app/config/log {}
[7/26/2023, 9:35:33 PM] info: [9:35:33 PM] 192.168.1.42 GET /config/all?null {}
[7/26/2023, 9:35:33 PM] info: [9:35:33 PM] 192.168.1.42 GET /app/config/options/backup?null {}
[7/26/2023, 9:35:33 PM] info: [9:35:33 PM] 192.168.1.42 GET /state/appVersion?null {}

That help at all?

[Candurin]

It rebuilt no problem. I do see this error at the end after "npm start":

/home/pi/nodejs-poolController/config/Config.ts:52
this._cfg = fs.existsSync(this.cfgPath) ? JSON.parse(fs.readFileSync(this.cfgPath, "utf8")) : {};
^
SyntaxError: Unexpected end of JSON input
at JSON.parse ()
at FSWatcher. (/home/pi/nodejs-poolController/config/Config.ts:52:76)
at FSWatcher.emit (node:events:513:28)
at FSEvent.FSWatcher._handle.onchange (node:internal/fs/watchers:215:12)

[rstrouse]

Are you running PM2?

[rstrouse]

The good news is that it looks like it detected IntelliCenter.

Are you running PM2?

[Candurin]

I am.

[rstrouse]

Type pm2 monit

Then open a new console and type
pm2 stop njsPC
pm2 restart njsPC

Then lets monitor the log for njsPC.

[Candurin]

Doing it now. What's funny is that all the lights are blinking on my USB RS485 adapter

[Candurin]

[Candurin]

[rstrouse]

Go into the port settings in dashPanel and press Save Port.

[Candurin]

I did.. Nothing happened.

[Candurin]

[rstrouse]

It isn't receiving garbage anymore but I don't see a connection to njsPC either. In the hamburger menu what is selected as the njsPC instance?

[Candurin]

Posting this as well. Cause why not :)

[Candurin]

This what you wanted?

[Candurin]

[rstrouse]

Change the address from 192.168.1.50 to 127.0.0.1

[Candurin]

Done

[rstrouse]

Press apply. Did the not connected message go away?

[Candurin]

Nope. Still there.

[rstrouse]

Perhaps it is time to re-install the pi. This is so bizarre. The port opens reads a few bytes off it then promptly closes because there is no traffic. It doesn't look like you run any of the REM extensions so all of your configuration is on the IntelliCenter.

If you have another SD card you can simply install a new one of those from scratch with the new OS and go from there.

[Candurin]

Does a multimeter (continuity) help me diagnose any of the com/RS485 ports on the IntelliCenter at all?

[tagyoureit]

It won't tell you anything more then the blinking lights. If you had more advanced graphing meters you could see the high and low bits but that level of detail is rarely helpful.

Can you try stopping all of the PM2/njsPC services and running the od -x command as outlined in the wiki? Let's see if you get any data outside of njsPC.

Edit (add):
Also, make sure you aren't running the njsPC services through PM2 and pi@raspberrypi:/nodejs-poolController $ npm start& at the same time. This would definitely cause problems.

[Candurin]

At this point in time I am still at 5 (c) in the wiki (the last commands were cd nodejs-poolController then npm I then npm start&) . It is currently in a continual cycle because the RS485 communication keeps opening and closing with errors. Should I stop that process and continue on?

[tagyoureit]

The od -x command is an OS level command that just snoops on the packets coming in. It can be used to test the adapter at any point, but njsPC should not be running (you don't want multiple applications to access the adapter at the same time). So, yes, just pause where you are and show us what the output of that command is.

[Candurin]

Nothing is happening (I am entering these commands on my pi):

pi@raspberrypi:~/nodejs-poolController $ od -x < /dev/ttyUSB0
^C

[1]+ Stopped npm start
pi@raspberrypi:/nodejs-poolController $ exit
exit
There are stopped jobs.
pi@raspberrypi:/nodejs-poolController $ od -x < /dev/ttyUSB0

[Candurin]

I'm rebooting it now.

[tagyoureit]

You can use the command ps -ef | grep node and it will return all of the running node processes. This is a good way to check that you don't have njsPC running from an unknown source.

[Candurin]

pi@raspberrypi:~ $ ps -ef | grep node
pi 1428 1422 0 11:20 pts/2 00:00:00 grep --color=auto node

[Candurin]

So... Entering either od -x < /dev/ttyUSB0 or od -x does nothing. It just sits there.

[tagyoureit]

As @rstrouse said, something is not right with either the adapter or the wiring. I would try new wiring and/or a new adapter (even in just a test bed scenario) and figure out the hardware issue first.

[Candurin]

I have tried two different RS485 usb adapters as well as different ethernet cables. I even connect both RS485 adapters directly to the intellicenter, all with similar, failing, results.

[tagyoureit]

When you say "directly", do you mean just the RS485 adapter & wires without the long ethernet cable, surge protector, and breakout connector?

[Candurin]

Correct. I had posted a picture of it yesterday, if that helps.

[Candurin]

If either of you are interested, I'd be glad to forward a port for you to diagnose remotely, if that helps (would certainly help me :) )

Or at least let me know what I can be looking for hardware-wise on my intellicenter? I know that everything is working EXCEPT I am noticing that the Salt Level is reporting as "0" on the intellicenter system, yet, I'm still generating proper chlorine and the leds on the salt cell are functioning properly. Maybe related?

[rstrouse]

The salt level from IntelliChlor will only report on interval after startup. The timeframe is dependent upon the version. It may take an entire day to get caught up. It is likely unrelated if you can control it from IntelliCenter.

As for remoting. I am completely booked through the week at this point. But have you tried another USB port on the pi?

[Candurin]

Where I am now, is that the lights are actively blinking on the RS485 usb adapter and DashPanel shows this:

Seems a bit closer, but it's not getting my configuration from my intellicenter panel.

[Candurin]

OK. I tested the lan cable (https://www.amazon.com/gp/product/B00HEM653S) and it turns out that there is an open failure on 4 & 5 in the cable (which is odd as the cable is fully intact start to finish). As of the start of this, The A-/B+ data cables ran though the outermost pins on the breakout cable. I have also tried using "ports" 1-3 and 6-8 without success.

I would have assumed that as long as I am NOT using cable 4 or 5 to transmit the data it would be fine, no?

Does it seem as though I need to run a new 100' cable?

[rstrouse]

Well that is disturbing but earlier you had this cable completely removed from the situation. Typically a short or open on a previously good cable will exhibit damage.

The command od -x < /dev/ttyUSB0 is simply listening to the data lines of the adapter. In this instance we are seeing nothing as in no data at all on the line. You did have IntelliCenter running when you ran this command didn't you?

[Candurin]

Yes. I haven’t shut down intellicenter at all.

Are we thinking it’s a hardware issue at my panel? I have no clue where to even look for issues now.

[rstrouse]

If your IntelliCenter is controlling the pumps and IntelliChlor then there is nothing wrong with it. The problem is likely either the adapter or the wiring.

[Candurin]

But which wiring? This is what isn’t making any sense.

I’ve tried direct to the intellicenter with no luck.

I’ve tried both rs485 adapters as well.

[Candurin]

More feedback on current situation:

pi@raspberrypi:~ $ ps -ef | grep node
pi 1067 867 1 12:59 ? 00:00:30 node /home/pi/.pm2/modules/pm2-logrotate/node_modules/pm2-
pi 1288 1091 0 13:00 ? 00:00:00 sh -c npm run build && node dist/app.js
pi 1289 1097 0 13:00 ? 00:00:00 sh -c npm run build && node dist/app.js
pi 1490 1288 0 13:00 ? 00:00:04 node dist/app.js
pi 1501 1289 0 13:02 ? 00:00:15 node dist/app.js
pi 1834 1799 0 13:31 pts/0 00:00:00 grep --color=auto node

[rstrouse]

The od command is simply a dump of what is on the wire. Try another port on the IntelliCenter. Double up the wires in the connector.

…

On Jul 28, 2023, at 6:57 PM, Candurin ***@***.***> wrote:  But which wiring? This is what isn’t making any sense. I’ve tried direct to the intellicenter with no luck. I’ve tried both rs485 adapters as well. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[Candurin]

What do you mean by double up the wires?

[rstrouse]

Put 2 wires in each terminal green and yellow on the main board. Please flip the breaker to cut IntelliCenter power when you are wiring rs485.

[Candurin]

This is using the doubled wiring:

pi@raspberrypi:~ $ od -x < /dev/ttyUSB0
0000000 fffd ffdc fff7 f5d7 ffeb ffff fffd bfff
0000020 f9a0 ff4c f0ff fffb 7fff eefe fffd 6aff
0000040 fffd efef ffd6 ffbf abfe f7ff ffff fbeb
0000060 b4fa fae3 ff7f fffe fbb8 7ad6 e6db fafe
0000100 feeb ff7f fbf7 fdde eaff fb7e eed7 fffb
0000120 dffd f77f ffff ffff ffe7 fffd f7ff ffff
0000140 f7b7 ffff dfb5 ffff ffdf 7fff efff b77f
0000160 ffff ffff 7fff bf75 f7ff ffff ff7f ffff
0000200 fdff ffff ffb7 ffff ff6f ffef fffd ffff

However, this is the incorrect wiring (no ffa5ff) at my adapter (green to B-, yellow to A+), which I know. If I wire the adapter CORRECTLY I never get any results from the od - x command.

[rstrouse]

That looks like only one side of the rs485 wiring is working. Rs485 is a differential circuit when you flip it around it never pulls high so no bits. The other adapter exhibits the same behavior?

[Candurin]

The other adapter isn’t working at all.

I am going to move everything over to my pi4 (currently on a 3B now) to rule that out tomorrow.

Other than that, I’m out of options.

[Candurin]

Moved the installation over to my Mac this morning.

Both usb adapters are exhibiting the same behavior:

[7/29/2023, 9:50:19 AM] warn: Inactivity timeout for 0 serial port /dev/tty.usbserial-A10LKG26 after 10 seconds
[7/29/2023, 9:50:19 AM] info: Serial Port 0 - /dev/tty.usbserial-A10LKG26 has been closed.
[7/29/2023, 9:50:19 AM] info: Successfully closed portId 0 for serial port /dev/tty.usbserial-A10LKG26
[7/29/2023, 9:50:19 AM] info: Serial port: /dev/tty.usbserial-A10LKG26 request to open successful 9600b 8-none-1
[7/29/2023, 9:50:29 AM] warn: Inactivity timeout for 0 serial port /dev/tty.usbserial-A10LKG26 after 10 seconds
[7/29/2023, 9:50:29 AM] info: Serial Port 0 - /dev/tty.usbserial-A10LKG26 has been closed.
[7/29/2023, 9:50:29 AM] info: Successfully closed portId 0 for serial port /dev/tty.usbserial-A10LKG26
[7/29/2023, 9:50:29 AM] info: Serial port: /dev/tty.usbserial-A10LKG26 request to open successful 9600b 8-none-1
[7/29/2023, 9:50:39 AM] warn: Inactivity timeout for 0 serial port /dev/tty.usbserial-A10LKG26 after 10 seconds
[7/29/2023, 9:50:39 AM] info: Serial Port 0 - /dev/tty.usbserial-A10LKG26 has been closed.
[7/29/2023, 9:50:39 AM] info: Successfully closed portId 0 for serial port /dev/tty.usbserial-A10LKG26
[7/29/2023, 9:50:39 AM] info: Serial port: /dev/tty.usbserial-A10LKG26 request to open successful 9600b 8-none-1
[7/29/2023, 9:50:49 AM] warn: Inactivity timeout for 0 serial port /dev/tty.usbserial-A10LKG26 after 10 seconds
[7/29/2023, 9:50:49 AM] info: Serial Port 0 - /dev/tty.usbserial-A10LKG26 has been closed.
[7/29/2023, 9:50:49 AM] info: Successfully closed portId 0 for serial port /dev/tty.usbserial-A10LKG26
[7/29/2023, 9:50:49 AM] info: Serial port: /dev/tty.usbserial-A10LKG26 request to open successful 9600b 8-none-1
[7/29/2023, 9:50:59 AM] warn: Inactivity timeout for 0 serial port /dev/tty.usbserial-A10LKG26 after 10 seconds
[7/29/2023, 9:50:59 AM] info: Serial Port 0 - /dev/tty.usbserial-A10LKG26 has been closed.
[7/29/2023, 9:50:59 AM] info: Successfully closed portId 0 for serial port /dev/tty.usbserial-A10LKG26
[7/29/2023, 9:50:59 AM] info: Serial port: /dev/tty.usbserial-A10LKG26 request to open successful 9600b 8-none-1

od -x < /dev/tty.usbserial-A10LKG26 is not reporting back any data.

[Candurin]

Is it at all possible that something went wrong with this board:

I am now seeing in intellicenter that salt is no longer being reported. I am not sure if there is different sets of data across the green and yellow, but it is notable that salt was always being reported before and now no longer.

[tagyoureit]

If that is the intellichlor board, I would start with the RS485 ports directly on the Intellicenter panel. These boards can do some strange things because they can be used standalone. Try doubling up on the wires that go to your pump, instead.

[Candurin]

Well! That fixed it (well, 90% at least)! I doubled up on the pump (connected to COM-A (J5) on the main control board.

I am still getting no communications from the IC40 and it doesn't seem to remember that my AUX circuit is named "microbrites" to control the lights. I do get the output and correct light names in IntelliCenter app though.

I am still thinking that my IC circuit/surge board has suffered some damage during a storm while we were gone, which explains why NONE of those comm terminals were reporting correctly.

[rstrouse]

To fix the circuit name go into the circuit in either the mobile app or the OCP and save it. Just to be certain that there is something simple going on make sure the connection that jumps between the main board and the surge board is tight and secure. You can pull these connectors straight up off the board to check their connection. It is remarkable that it somehow affected the RS485 bus connectors only on this board.

[Candurin]

I’ve checked each one. Once I move the connections to any comm port on the IC board, no dice.

Still not seeing the salt levels, which always worked before this issue started, which also makes me thing there’s a board issue.

[rstrouse]

If you change the percentage on the app while the pool is running do the lights change on the chlorinator to reflect the change? The salt level reporting is very difficult to catch since it will not report this out of the cell right away. From your screenshot above it does not appear that the cell is reporting anything.

[Candurin]

The lights do not change. In fact, there’s no lights on for the percentage value at all.

The salt value had always been working before.

[rstrouse]

Can you control the cell by sliding the cover up and pressing the more button?

[Candurin]

I found the problem. It’s comm port B on the board. Seems to have failed. I moved the salt cell comm to the ICG board and started working immediately.

[rstrouse]

Wow that is an obscure failure point. Glad it is all working! I guess each comm port must have its own TVS.

[Candurin]

@rstrouse @tagyoureit

Sincerely, thank you both for the help. I certainly have a much better understanding of the inner workings of the IC now (I’ve had it for 3 years!). I’ll call pentair just to inquire about it and see what they say.

I can confirm that it is ONLY Com-B (port J4) that no longer works. I moved around the cables and whatever is attached to J4 (on the main panel) doesn’t function. If I connect the IC control board to that port none of the IC com ports work.

This seems to be related to a storm we had at the same time my panel stopped communicating (I checked ring doorbell time and date to confirm). It’s odd because I have surge protectors on the Ethernet cables, the entire sub-panel is protected as well (we have a whole house protector on our main house panel).

You guys are fantastic! Thanks again.