Add the possibility to write values using IDM modbus

[Colleoni]

Hi,
before all, thanks for this integration.
I performed different tests and this integration (at least with my setup) represents the only way to correctly setup the
modbus communication with the IDM heatpump.

Here I suggest what I think is a useful integration

Is your feature request related to a problem? Please describe.

IDM heat pump offers the possibility to also write values using modbus. I found two sensors to be particularly interesting.

• sensor.idm_warmepumpe_aktueller_pv_produktion
• sensor.idm_warmepumpe_aktueller_pv_uberschuss

Setting this two values, you can inform the heat pump about how much energy you have from a photovoltaic system and how much of that energy is available as it does exceed your current consumption. As far as I understood, this way the heat pump can prepare hot water taking advantage of the energy that otherwise will be unused. Saving energy later as the water will be already hot.

Describe the solution you'd like

The perfect solution will be to select another sensor as input, and one of the two variables as output, repeating the process for both variables.

• sensor.my_inverter_pv_produktion -> sensor.idm_warmepumpe_aktueller_pv_produktion
• sensor.my_inverter_aktueller_pv_uberschuss -> sensor.idm_warmepumpe_aktueller_pv_uberschuss

Two considerations:

1. aktueller_pv_uberschuss can be te result of an arithmetical operation, current production - current energy usage, which can be obtained using aliases if I correctly understood.
2. Some inverters automatically shout down at night, that should be took into consideration when reading input from the sensor, thus performing something like sensor.my_inverter_pv_produktion or 0.0

Describe alternatives you've considered

I tried to setup everything manually following the really precious instructions on this repo Homeassistant-IDM-Integration, however I encounter some problems with the home assistant modbus implementation as every few reads it encounters an error, setting the values to unavailable until the next read.
I have investigated a little bit in the code as home assistant core uses the same pymodbus 3.1.3, but I did non find the cause of the problem.
I never tested write operations as I do not feel confident of the solution until I can get rid of the problem during reading.

Error:
[homeassistant.components.modbus.modbus] Pymodbus: idm_wp: Modbus Error: [Connection] ModbusTcpClient(192.168.61.6:502): Connection unexpectedly closed 0.0067907405853271484 seconds into read of 8 bytes without response from unit before it closed connection

[kodebach]

First of all thanks for the interest in the integration.

I know of the writable registers the IDM heat pumps provide. However, I have not implemented them, because the heat pump I used to develop this integration is connected to a Fronius Solar Inverter, which uses a proprietary protocol through which the heat pump can access even more information than otherwise. Therefore, I sadly have no way of testing the modbus way of setting solar energy values. Without a way to test things, I'm not comfortable to write the necessary code.

If you feel comfortable writing the necessary code yourself, I am open to pull requests. I can also point you in the right direction (which code probably needs changes), but you'd have to do most of the work.

Otherwise, I can only suggest you look into connecting the heat pump to your solar inverter through some other means. The IDM multiple different inputs for this, IDM support may be able to help you with that and you can also find some info via Google.

every few reads it encounters an error, setting the values to unavailable until the next read.

Yes, that in fact is part of the reason why I developed this integration. It seems the IDM heat pump doesn't like to keep TCP sockets open and refuses to respond after a while. The integration solves that by trying to reconnect when a read fails. Only if the reconnect also fails is the value set to unavailable.

[Colleoni]

Yes I am considering a pull request but I still trying to understand the code. You did a really great job and looking into your code I am learning a lot of things. Last time I tested I was looking into the docker setup you made with visual studio code. I will try to implement something as soon as I understand the code better.

For the moment since you know a lot of the problem on the TCP socket, what do you think opening an issue on homeassistant/core repo to explain the problem? and suggesting them the same fix? Probably it can solve the situation also for others.
The strange thing is that this problem seems to not occur in every installation. I asked to the developer who wrote the repo I linked and he seems to have zero problems. While another person told me that after every correct read the next will be a bad one. In my case instead it seems more a random pattern, and my main hypothesis is that this was caused by the fact that I was using different scan intervals.

[kodebach]

The VS Code / devcontainer setup is based on https://github.com/ludeeus/integration_blueprint, which has a bit more documentation on it.

what do you think opening an issue on homeassistant/core repo to explain the problem? and suggesting them the same fix? Probably it can solve the situation also for others.

There is already a recent issue home-assistant/core#87265 for similar problem with different device. The developers of Homeassistant decided that the "Connection unexpectedly closed" problem is not an error in the modbus integration and therefore they cannot do anything.

It seems the config in home-assistant/core#87265 (comment) (somewhat) helped the person with the original issue. Maybe it works for you too.

Otherwise, I would suggest you open a new issue yourself. You can tag me, or link to this issue and I'll gladly explain to the homeassistant devs what I did in this integration. But I don't think it would be much help, if I opened the issue myself. I don't have any logs with the error right now. I also can't test any fixes without reconfiguring my (working) homeassistant setup, which I'd like to avoid.

The strange thing is that this problem seems to not occur in every installation.

If the IDM heat pump does use a timeout for idle connections, the read intervals configured in homeassistant would definitely have an effect on the issue. If you read very often, you may not see the error at all. It could also be that different IDM models act differently or that it depends on your local LAN setup, especially WiFi might cause unstable connections.

[Colleoni]

Thank you very much I found the documentation and I started following it yesterday. Let's see if I can come up with something useful.

I have read the issue you linked multiple times, but at the end the owner found out the problem which was a NodeRed instance. After reading the issue I tested different setups hoping that also in my case was something network related. I disconnected everything and to be sure there was nothing even on my pc (which I use to monitor) I changed all the ips, but nothing.
Then I found out other IDM and homeassitant owners have the same problem and thanks to this I found your repo.
From your response I am finally convinced that it is something related to timeout of the TCP socket. I wrote a simple python script to test reads and only after I read your answer I realized it was working because I was opening and closing a new connection every time, now I will try to keep it open to test if there is a timeout. I have several different ideas to test.

I found also an issue home-assistant/core#87265 which is more related to RTU but that suggest in June there will be an update of pymodbus.

[georgbachmann]

@Colleoni If you have progress on the writing data to the heatpump and you need testers... I'd be in... would be nice to control heating temperature in winter via home assistant!
And... also... like you said better using the PV-Überschuss (solar energy that would go to the grid... don't know the english word) would be amazing!!!! Cause currently I have huge spikes when e.g. heating water... if that could be done in a smarter way, that would be great!

Also THANKS a lot for the integration so far. I used to read the values manually using node-red and their modbus integration. But this is just so much easier and gives me so much more sensor data! Love it! Thanks a lot!!!

[AndyNew2]

Hi all, closing connection and sockets may be related to limited resources and is a way to keep the system running endless. Therefore this behavior may be partly intended by IDM and I can understand why they did so. @kodebach: Your code fixes that issue and I think this is the prefered way anyway. You never can expect sockets are endless connected and need always be prepared for a error handling and reconnect code. Well done, just keep it that way, even modbus implementation changes later. That is good practise anyway.
For the writing of the two PV values, I am interested as well and can test.
@Collenoi: Your initial considerations about calculation of these values, this is your job in the automation of your Setup. This will never be part of this integration, because this integration can't take care of your solar setup. I like it a lot more to have this separated.

[kodebach]

closing connection and sockets may be related to limited resources and is a way to keep the system running endless

Yeah that all makes sense. However, the way it is implemented is still very strange. The heat pump doesn't actually terminate the TCP socket AFAICT, it just stops responding and/or sends empty responses.

[Colleoni]

@georgbachmann @AndyNew2, unfortunately I did not have time to look any further into the setup. So still at the starting point and currently using a manual schedule which I setup to what I expect to be sunny hours during the day. Simply by manually looking into PV production stats (sure it does not consider the possibility of cloudy days, so not a real solution)

For the initial considerations, sure it depends on personal setup I do not expect to find them integrated (or at least not strictly required), I just thought they could be useful to explain the issue and the possibilities.
In any case, I think you can create your own aliases and operations in homeassitant.

[KVS62]

@Colleoni - @SimowN implemented the writing of the PV values already. See https://github.com/simowNgithub

[AndyNew2]

@kv62: I struggle to understand the work from SimowN: Does this mean the provided script directly writes the values via modbus to the iDM Heatpump. Because the provided github link is no home assistant integration. It is just little elements. The description is poor. So nothing to install, just copy and past the script and adapt to my needs? For example I do not need this calculation and currently just want to copy the values from Solarmodul to iDM.

[simowNgithub]

closing connection and sockets may be related to limited resources and is a way to keep the system running endless

Yeah that all makes sense. However, the way it is implemented is still very strange. The heat pump doesn't actually terminate the TCP socket AFAICT, it just stops responding and/or sends empty responses.

@kodebach should we check it together to get it work? at the moment i tuned my script for restarting the HA modbus service after every write register to the IDM. Because after updating the modbus libs with HA 2023.09.* the old script was broken.

[kodebach]

Does this mean the provided script directly writes the values via modbus to the iDM Heatpump. Because the provided github link is no home assistant integration. It is just little elements.

AFAICT the repo at https://github.com/simowNgithub/Homeassistant-IDM-Integration contains example configuration files that use the existing modbus integration for Home Assistant to communicate with the heat pump. I had problems with frequent disconnections and "unavailable" values with that approach, which is why I wrote this custom integration. I believe the modbus integration has since been improved and now also handles automatic reconnects.

should we check it together to get it work? at the moment i tuned my script for restarting the HA modbus service after every write register to the IDM. Because after updating the modbus libs with HA 2023.09.* the old script was broken.

The integration already has code to automatically restart it's internal modbus client, if a read fails, so there shouldn't be any issues in this integration. I believe the improvement to the underlying pymodbus library that would help here is the retry_on_empty option. I didn't test this, because I solved it differently in my integration. But in the standard modbus integration this option was exposed as retry_on_empty in 2023.9.x, but has been deprecated in 2023.10.x. Now it is automatically enabled, so maybe with 2023.10.x your scripts work better again.

In #40 I added some basic logic for writing values via a service exposed by the integration. It is almost entirely untested, but should in theory allow write the values requested here. Feel free to test and provide feedback in #40.

[AndyNew2]

Yes that is a lot better understandable. When I read the description from simowNgithub, I expected to see something similar you did. But this is not the case. The the sentense "Use my script for getting the available power for the heatpump and sync the values to the heatpump" means actually a little bit rework and not just using the script. The main work is actually in the configuration file holding the code and needs adaptation. Nothing described, but the integration in iDM which is described in the iDM paper anyway, is very detailed. That is confusing, since you expect the same detail level on the HA integration as well. I am quite busy in the moment but at a certain point will try #40

[KVS62]

Thanks to @kodebach for the improved integration. So far I have used @simowNgithub 's script/config and adapted this for my situation (surplus solar power provided from a P1-connector to my smart meter + calculated solar production from my clamps used by my Myenergi-car charger(some problems with the latter)). I get values to the heat pump and can see that it reacts based on this.

I am now considering to use @kodebach 's integration as the basis. To make this work for my situation, I wanted to ask whether it is correct that I can do the following:

1. Create a config file to define + do the required calculations for surplus solar & calculated solar production (assume that all other variables required are defined in the @kodebach integration)
2. Use @simowNgithub 's script & automation to write these values to the heat pump

Appreciate if you could confirm that in principle this would work + I assume that the @kodebach integration will be extended with the write option later on, so that only 1. will be required.

[kodebach]

In theory your idea should work. I think you can just remove the sensors part of the modbus config for the heat pump (you need the rest for writing). In the scripts and automations you still use modbus.write_register to write to the heat pump, but you swap the IDs from the removed modbus: sensors config with the corresponding IDs from this integration.

[simowNgithub]

Thanks to @kodebach for the improved integration. So far I have used @simowNgithub 's script/config and adapted this for my situation (surplus solar power provided from a P1-connector to my smart meter + calculated solar production from my clamps used by my Myenergi-car charger(some problems with the latter)). I get values to the heat pump and can see that it reacts based on this.

I am now considering to use @kodebach 's integration as the basis. To make this work for my situation, I wanted to ask whether it is correct that I can do the following:

1. Create a config file to define + do the required calculations for surplus solar & calculated solar production (assume that all other variables required are defined in the @kodebach integration)
2. Use @simowNgithub 's script & automation to write these values to the heat pump

Appreciate if you could confirm that in principle this would work + I assume that the @kodebach integration will be extended with the write option later on, so that only 1. will be required.

it is exactly what i‘m doing. and exaktly the case why i‘m active in this issue, because for long term i want to get everything work by only using the idm integration package from @kodebach.

[AndyNew2]

Anybody tried yet this beta? It is a little bit hassle to install that version.
@kodebach: Would you mind to create a standard beta release? I will still try but need to find more time...

[geryyyyyy]

Hi All,
I was using the IDM HACS in combination with the modbus write_register scripts for several months. But from time to time it happens that the value at the IDM site doesn't get updated anymore. Therefore I am also very interested into getting this done all in one stable integration.

I am about to test the write feature. I was able to successfully implement the write-values branch into HA. But I struggle with the adaption of the scripts.yaml.

In theory your idea should work. I think you can just remove the sensors part of the modbus config for the heat pump (you need the rest for writing). In the scripts and automations you still use modbus.write_register to write to the heat pump, but you swap the IDs from the removed modbus: sensors config with the corresponding IDs from this integration.

I am not sure with what I have to replace the data (hub, unit, address) part with?

idm_update_pv_values:
  alias: Update Photovoltaik values to IDM Headpump
  sequence:
  - service: modbus.write_register
    data:
      hub: idm_heatpump
      unit: 1
      address: 74
    data_template:
      value: >
        [ {{ unpack(pack(states("sensor.wp_aktueller_pv_uberschuss")|float, ">f"), ">H", offset=2) }}, 
          {{ unpack(pack(states("sensor.wp_aktueller_pv_uberschuss")|float, ">f"), ">H") }} ]
  - service: modbus.write_register
    data:
      hub: idm_heatpump
      unit: 1
      address: 78
    data_template:
      value: >
        [ {{ unpack(pack(states("sensor.pv_aktuelle_produktion")|float, ">f"), ">H", offset=2) }}, 
          {{ unpack(pack(states("sensor.pv_aktuelle_produktion")|float, ">f"), ">H") }} ]
  mode: single

My sensor names from the integration are default:

• sensor.idm_warmepumpe_aktueller_pv_produktion
• sensor.idm_warmepumpe_aktueller_pv_uberschuss

Could someone please give me an example what exactly this has to look like?
Thanks.

[simowNgithub]

Hi All, I was using the IDM HACS in combination with the modbus write_register scripts for several months. But from time to time it happens that the value at the IDM site doesn't get updated anymore. Therefore I am also very interested into getting this done all in one stable integration.

I am about to test the write feature. I was able to successfully implement the write-values branch into HA. But I struggle with the adaption of the scripts.yaml.

In theory your idea should work. I think you can just remove the sensors part of the modbus config for the heat pump (you need the rest for writing). In the scripts and automations you still use modbus.write_register to write to the heat pump, but you swap the IDs from the removed modbus: sensors config with the corresponding IDs from this integration.

I am not sure with what I have to replace the data (hub, unit, address) part with?

idm_update_pv_values:
  alias: Update Photovoltaik values to IDM Headpump
  sequence:
  - service: modbus.write_register
    data:
      hub: idm_heatpump
      unit: 1
      address: 74
    data_template:
      value: >
        [ {{ unpack(pack(states("sensor.wp_aktueller_pv_uberschuss")|float, ">f"), ">H", offset=2) }}, 
          {{ unpack(pack(states("sensor.wp_aktueller_pv_uberschuss")|float, ">f"), ">H") }} ]
  - service: modbus.write_register
    data:
      hub: idm_heatpump
      unit: 1
      address: 78
    data_template:
      value: >
        [ {{ unpack(pack(states("sensor.pv_aktuelle_produktion")|float, ">f"), ">H", offset=2) }}, 
          {{ unpack(pack(states("sensor.pv_aktuelle_produktion")|float, ">f"), ">H") }} ]
  mode: single

My sensor names from the integration are default:

• sensor.idm_warmepumpe_aktueller_pv_produktion
• sensor.idm_warmepumpe_aktueller_pv_uberschuss

Could someone please give me an example what exactly this has to look like? Thanks.

Hi, i had this issues with the new HA releases. At the moment I always restart the modbus HA component before the write register…

idm_update_pv_values:
  alias: Update Photovoltaik values to IDM Headpump
  sequence:
  - service: modbus.restart
    data:
      hub: idm_heatpump
    enabled: true
  - service: modbus.write_register
    data:
      hub: idm_heatpump
      unit: 1
      address: 74
    data_template:
      value: "[ {{ unpack(pack(states(\"sensor.wp_aktueller_pv_uberschuss\")|float,
        \">f\"), \">H\", offset=2) }}, \n  {{ unpack(pack(states(\"sensor.wp_aktueller_pv_uberschuss\")|float,
        \">f\"), \">H\") }} ]\n"
  - service: modbus.write_register
    data:
      hub: idm_heatpump
      unit: 1
      address: 78
    data_template:
      value: "[ {{ unpack(pack(states(\"sensor.pv_aktuelle_produktion\")|float, \">f\"),
        \">H\", offset=2) }}, \n  {{ unpack(pack(states(\"sensor.pv_aktuelle_produktion\")|float,
        \">f\"), \">H\") }} ]\n"
  mode: single

[kodebach]

I am not sure with what I have to replace the data (hub, unit, address) part with?

See #40 (comment)

[AndyNew2]

A question to people already have written to the registers.
Could you please share the experience for these values:

1. pv_produktion is obvious, and you calculate the value to match kW, correct? So if my Source tell me this value in W and need to divide by 1000
2. pv_Ueberschuss is complicated. It seems easy in the first place, but what does it mean in detail:
   a) When the heatpump starts, does the heatpump expect this value is reduced by the own consumption? So when the heatpump starts and needs 1kW the value should go down by 1kW? This is usually automatically the case, but not for me. I need to substract in that case the estimated own consumption.
   b) Is it expected to be negative in case the pv production is too less and external energy is consumed. This is used for iDM statisticts, correct?

[AndyNew2]

Actually for trying the branch I am a little bit behind the topic. Do I need to have git installed to do so? I have no idea how to get the clone done. I guess it is too error prone to do it file by file and transfer this to the HA installation?
Really look forward for a beta version...

[kodebach]

When the heatpump starts, does the heatpump expect this value is reduced by the own consumption? So when the heatpump starts and needs 1kW the value should go down by 1kW?

That seems to be the case. At least from what I can tell with my setup, where the heat pump is communicating directly with the Fronius solar inverter. I can still read the "PV Übschuss" value from the heat pump and as soon as the heat pump switches on this value drops.

I assume the the idea behind the two value is that you have two power meters. One that measure the power from the solar array and another that measures the power leaving your house towards the grid (sometimes you can use the smart meter provided by the power company for that). The heat pump then wants the raw value from the first power meter as well as the difference between the two meters.

The Modbus documentation from IDM also says that you can use the "Aktuelle Leistungsaufnahme Wärmepumpe" value if necessary and that the heat pump will start as soon as "PV Überschuss" > "Aktuelle Leistungsaufnahme Wärmepumpe" and will then follow the settings in the PV menu. I assume there is also some hysteresis involved and that the heat pump compares against an expected power value, if it is not currently running.

Is it expected to be negative in case the pv production is too less and external energy is consumed. This is used for iDM statisticts, correct?

Not sure. The integration would currently show negative values as "unavailable". But you can just look at the debug logs, when your PV is running and producing power, but there is no surplus. If the raw value for power_solar_surplus is negative, I should probably change the integration to allow that.

But the difference between 0 and negative is probably just what shows up in the IDM UI.

[geryyyyyy]

I am not sure with what I have to replace the data (hub, unit, address) part with?

See #40 (comment)

Thanks for the explanation. In the developer section it is working fine and looks for example like this:

service: idm_heatpump.set_power
data:
  target: sensor.idm_warmepumpe_aktueller_pv_produktion
  value: 0.123

But I am was not able to successfully add my sensor data to the value. Maybe it is a problem with the format of my data preparation?
I have a GoodWe GW10K-ET, using the GoodWe Inverter Integration to receive the data.
In the configuration.yaml I use an adapted version of this config: https://github.com/simowNgithub/Homeassistant-IDM-Integration/blob/main/configuration.yaml

template:
  - sensor:
      - name: "PV aktuelle Produktion"
        unique_id: "idm_aktuelle_pv_produktion"
        unit_of_measurement: "kW"
        state: >
          {% if states('sensor.pv_power') == 'unavailable' %}
            0.0
          {% else %}
            {{ '%.2f'%(states('sensor.pv_power')|float / 1000|float| float) }}
          {% endif %}
        state_class: measurement
        device_class: power
        icon: mdi:solar-power
      - name: "WP aktueller PV Überschuss"
        unique_id: "idm_aktueller_pv_ueberschuss"
        unit_of_measurement: "kW"
        state: >
          {% if states('sensor.pv_power')|float == 0|float %}
            0.0
          {% elif states('sensor.meter_active_power_total') == 'unavailable' %}
            0.0
          {% elif states('sensor.meter_active_power_total')|float < 0|float %}
            0.0
          {% else %}
            {{ '%.2f'%(states('sensor.meter_active_power_total')|float / 1000|float| float) }}
          {% endif %}
        state_class: measurement
        device_class: power
        icon: mdi:solar-power

The two values are prepared as float and rounded to two decimal places. It is working fine with the "modbus.write_register" script, but somehow I cannot use it in the "idm_heatpump.setpower" service.

service: idm_heatpump.set_power
data:
  target: sensor.idm_warmepumpe_aktueller_pv_produktion
  value: {{ states('sensor.pv_aktuelle_produktion') }}

Fehler beim Aufrufen des Diensts idm_heatpump.set_power. Unknown error

Does someone have an idea what I am doing wrong?

2. a) When the heatpump starts, does the heatpump expect this value is reduced by the own consumption? So when the heatpump starts and needs 1kW the value should go down by 1kW? This is usually automatically the case, but not for me. I need to substract in that case the estimated own consumption.

As the uberschuss it the actual value fed into the grid measured by the smart meter, it will be for sure getting lower if the power is consumed by the heat pump. I think the IDM is regulating the actual usage by using both values (produktion and uberschuss), but not sure how exactly it works. I assume that it regulates the power usage of the IDM always close to a uberschuss of almost 0.

2. b) Is it expected to be negative in case the pv production is too less and external energy is consumed. This is used for iDM statisticts, correct?

These values are above prepared in a way, that it will never be below 0.

Actually for trying the branch I am a little bit behind the topic. Do I need to have git installed to do so? I have no idea how to get the clone done. I guess it is too error prone to do it file by file and transfer this to the HA installation?

I simply downloaded the branch (ZIP) and copy pasted (replaced) the files of the idm integration: \config\custom_components\idm_heatpump

[AndyNew2]

In the IDM GUI in settings - Photovoltaik - PV Leistung
you see three values Überschuss - PV Leistung - Hausverbrauch
So how are you writing "Hausverbrauch"? Either a negativ Überschuss ;-) or is there a hidding register to write to?

[kodebach]

So how are you writing "Hausverbrauch"? Either a negativ Überschuss ;-) or is there a hidding register to write to?

I assume "Hausverbrauch" is calculated by the heat pump as Hausverbauch = PV-Leistung - Überschuss. Possibly it also subtracts it's own usage.

but somehow I cannot use it in the "idm_heatpump.setpower" service.

@geryyyyyy can we please continue this discussion in #40. Otherwise it gets complicated with so many people discussing different things in one thread. See #40 (comment)

[AndyNew2]

I have temporary the script from simon running. I see value for PV Generation and PV Überschuss, but Hausverbrauch is always 0.
I don't see any further statistics either. Seems this is a functionality they dedicate to known Solar Generation Systems they support natively...

[kodebach]

The new release v0.5.0-beta02 now supports writing the PV values. Anyone using #40, please switch back to installing via HACS so you'll get future updates. Everyone else, please update and report any issues.

It would also be helpful, if you could post a list of other sensors which you'd like to set from Home Assistant. Some would be pretty easy to add. Others come with caveats in the Modbus docs from IDM, where I'd like to come up with some kind of safety mechanism first to avoid users (unknowingly) damaging their heat pump by using this integration.

Please report the sensors in #50 to avoid spamming this issue too much.