Add the possibility to write values using IDM modbus

[Colleoni]

Hi,
before all, thanks for this integration.
I performed different tests and this integration (at least with my setup) represents the only way to correctly setup the
modbus communication with the IDM heatpump.

Here I suggest what I think is a useful integration

Is your feature request related to a problem? Please describe.

IDM heat pump offers the possibility to also write values using modbus. I found two sensors to be particularly interesting.

• sensor.idm_warmepumpe_aktueller_pv_produktion
• sensor.idm_warmepumpe_aktueller_pv_uberschuss

Setting this two values, you can inform the heat pump about how much energy you have from a photovoltaic system and how much of that energy is available as it does exceed your current consumption. As far as I understood, this way the heat pump can prepare hot water taking advantage of the energy that otherwise will be unused. Saving energy later as the water will be already hot.

Describe the solution you'd like

The perfect solution will be to select another sensor as input, and one of the two variables as output, repeating the process for both variables.

• sensor.my_inverter_pv_produktion -> sensor.idm_warmepumpe_aktueller_pv_produktion
• sensor.my_inverter_aktueller_pv_uberschuss -> sensor.idm_warmepumpe_aktueller_pv_uberschuss

Two considerations:

1. aktueller_pv_uberschuss can be te result of an arithmetical operation, current production - current energy usage, which can be obtained using aliases if I correctly understood.
2. Some inverters automatically shout down at night, that should be took into consideration when reading input from the sensor, thus performing something like sensor.my_inverter_pv_produktion or 0.0

Describe alternatives you've considered

I tried to setup everything manually following the really precious instructions on this repo Homeassistant-IDM-Integration, however I encounter some problems with the home assistant modbus implementation as every few reads it encounters an error, setting the values to unavailable until the next read.
I have investigated a little bit in the code as home assistant core uses the same pymodbus 3.1.3, but I did non find the cause of the problem.
I never tested write operations as I do not feel confident of the solution until I can get rid of the problem during reading.

Error:
[homeassistant.components.modbus.modbus] Pymodbus: idm_wp: Modbus Error: [Connection] ModbusTcpClient(192.168.61.6:502): Connection unexpectedly closed 0.0067907405853271484 seconds into read of 8 bytes without response from unit before it closed connection

[kodebach]

First of all thanks for the interest in the integration.

I know of the writable registers the IDM heat pumps provide. However, I have not implemented them, because the heat pump I used to develop this integration is connected to a Fronius Solar Inverter, which uses a proprietary protocol through which the heat pump can access even more information than otherwise. Therefore, I sadly have no way of testing the modbus way of setting solar energy values. Without a way to test things, I'm not comfortable to write the necessary code.

If you feel comfortable writing the necessary code yourself, I am open to pull requests. I can also point you in the right direction (which code probably needs changes), but you'd have to do most of the work.

Otherwise, I can only suggest you look into connecting the heat pump to your solar inverter through some other means. The IDM multiple different inputs for this, IDM support may be able to help you with that and you can also find some info via Google.

every few reads it encounters an error, setting the values to unavailable until the next read.

Yes, that in fact is part of the reason why I developed this integration. It seems the IDM heat pump doesn't like to keep TCP sockets open and refuses to respond after a while. The integration solves that by trying to reconnect when a read fails. Only if the reconnect also fails is the value set to unavailable.

[Colleoni]

Yes I am considering a pull request but I still trying to understand the code. You did a really great job and looking into your code I am learning a lot of things. Last time I tested I was looking into the docker setup you made with visual studio code. I will try to implement something as soon as I understand the code better.

For the moment since you know a lot of the problem on the TCP socket, what do you think opening an issue on homeassistant/core repo to explain the problem? and suggesting them the same fix? Probably it can solve the situation also for others.
The strange thing is that this problem seems to not occur in every installation. I asked to the developer who wrote the repo I linked and he seems to have zero problems. While another person told me that after every correct read the next will be a bad one. In my case instead it seems more a random pattern, and my main hypothesis is that this was caused by the fact that I was using different scan intervals.

[kodebach]

The VS Code / devcontainer setup is based on https://github.com/ludeeus/integration_blueprint, which has a bit more documentation on it.

what do you think opening an issue on homeassistant/core repo to explain the problem? and suggesting them the same fix? Probably it can solve the situation also for others.

There is already a recent issue home-assistant/core#87265 for similar problem with different device. The developers of Homeassistant decided that the "Connection unexpectedly closed" problem is not an error in the modbus integration and therefore they cannot do anything.

It seems the config in home-assistant/core#87265 (comment) (somewhat) helped the person with the original issue. Maybe it works for you too.

Otherwise, I would suggest you open a new issue yourself. You can tag me, or link to this issue and I'll gladly explain to the homeassistant devs what I did in this integration. But I don't think it would be much help, if I opened the issue myself. I don't have any logs with the error right now. I also can't test any fixes without reconfiguring my (working) homeassistant setup, which I'd like to avoid.

The strange thing is that this problem seems to not occur in every installation.

If the IDM heat pump does use a timeout for idle connections, the read intervals configured in homeassistant would definitely have an effect on the issue. If you read very often, you may not see the error at all. It could also be that different IDM models act differently or that it depends on your local LAN setup, especially WiFi might cause unstable connections.

[Colleoni]

Thank you very much I found the documentation and I started following it yesterday. Let's see if I can come up with something useful.

I have read the issue you linked multiple times, but at the end the owner found out the problem which was a NodeRed instance. After reading the issue I tested different setups hoping that also in my case was something network related. I disconnected everything and to be sure there was nothing even on my pc (which I use to monitor) I changed all the ips, but nothing.
Then I found out other IDM and homeassitant owners have the same problem and thanks to this I found your repo.
From your response I am finally convinced that it is something related to timeout of the TCP socket. I wrote a simple python script to test reads and only after I read your answer I realized it was working because I was opening and closing a new connection every time, now I will try to keep it open to test if there is a timeout. I have several different ideas to test.

I found also an issue home-assistant/core#87265 which is more related to RTU but that suggest in June there will be an update of pymodbus.

[georgbachmann]

@Colleoni If you have progress on the writing data to the heatpump and you need testers... I'd be in... would be nice to control heating temperature in winter via home assistant!
And... also... like you said better using the PV-Überschuss (solar energy that would go to the grid... don't know the english word) would be amazing!!!! Cause currently I have huge spikes when e.g. heating water... if that could be done in a smarter way, that would be great!

Also THANKS a lot for the integration so far. I used to read the values manually using node-red and their modbus integration. But this is just so much easier and gives me so much more sensor data! Love it! Thanks a lot!!!

[AndyNew2]

Hi all, closing connection and sockets may be related to limited resources and is a way to keep the system running endless. Therefore this behavior may be partly intended by IDM and I can understand why they did so. @kodebach: Your code fixes that issue and I think this is the prefered way anyway. You never can expect sockets are endless connected and need always be prepared for a error handling and reconnect code. Well done, just keep it that way, even modbus implementation changes later. That is good practise anyway.
For the writing of the two PV values, I am interested as well and can test.
@Collenoi: Your initial considerations about calculation of these values, this is your job in the automation of your Setup. This will never be part of this integration, because this integration can't take care of your solar setup. I like it a lot more to have this separated.

[kodebach]

closing connection and sockets may be related to limited resources and is a way to keep the system running endless

Yeah that all makes sense. However, the way it is implemented is still very strange. The heat pump doesn't actually terminate the TCP socket AFAICT, it just stops responding and/or sends empty responses.

[Colleoni]

@georgbachmann @AndyNew2, unfortunately I did not have time to look any further into the setup. So still at the starting point and currently using a manual schedule which I setup to what I expect to be sunny hours during the day. Simply by manually looking into PV production stats (sure it does not consider the possibility of cloudy days, so not a real solution)

For the initial considerations, sure it depends on personal setup I do not expect to find them integrated (or at least not strictly required), I just thought they could be useful to explain the issue and the possibilities.
In any case, I think you can create your own aliases and operations in homeassitant.

[KVS62]

@Colleoni - @SimowN implemented the writing of the PV values already. See https://github.com/simowNgithub

[AndyNew2]

@kv62: I struggle to understand the work from SimowN: Does this mean the provided script directly writes the values via modbus to the iDM Heatpump. Because the provided github link is no home assistant integration. It is just little elements. The description is poor. So nothing to install, just copy and past the script and adapt to my needs? For example I do not need this calculation and currently just want to copy the values from Solarmodul to iDM.

[simowNgithub]

closing connection and sockets may be related to limited resources and is a way to keep the system running endless

Yeah that all makes sense. However, the way it is implemented is still very strange. The heat pump doesn't actually terminate the TCP socket AFAICT, it just stops responding and/or sends empty responses.

@kodebach should we check it together to get it work? at the moment i tuned my script for restarting the HA modbus service after every write register to the IDM. Because after updating the modbus libs with HA 2023.09.* the old script was broken.

[kodebach]

Does this mean the provided script directly writes the values via modbus to the iDM Heatpump. Because the provided github link is no home assistant integration. It is just little elements.

AFAICT the repo at https://github.com/simowNgithub/Homeassistant-IDM-Integration contains example configuration files that use the existing modbus integration for Home Assistant to communicate with the heat pump. I had problems with frequent disconnections and "unavailable" values with that approach, which is why I wrote this custom integration. I believe the modbus integration has since been improved and now also handles automatic reconnects.

should we check it together to get it work? at the moment i tuned my script for restarting the HA modbus service after every write register to the IDM. Because after updating the modbus libs with HA 2023.09.* the old script was broken.

The integration already has code to automatically restart it's internal modbus client, if a read fails, so there shouldn't be any issues in this integration. I believe the improvement to the underlying pymodbus library that would help here is the retry_on_empty option. I didn't test this, because I solved it differently in my integration. But in the standard modbus integration this option was exposed as retry_on_empty in 2023.9.x, but has been deprecated in 2023.10.x. Now it is automatically enabled, so maybe with 2023.10.x your scripts work better again.

In #40 I added some basic logic for writing values via a service exposed by the integration. It is almost entirely untested, but should in theory allow write the values requested here. Feel free to test and provide feedback in #40.

[AndyNew2]

Yes that is a lot better understandable. When I read the description from simowNgithub, I expected to see something similar you did. But this is not the case. The the sentense "Use my script for getting the available power for the heatpump and sync the values to the heatpump" means actually a little bit rework and not just using the script. The main work is actually in the configuration file holding the code and needs adaptation. Nothing described, but the integration in iDM which is described in the iDM paper anyway, is very detailed. That is confusing, since you expect the same detail level on the HA integration as well. I am quite busy in the moment but at a certain point will try #40