Debian(RPM later): Restart OpenSearch/Dashboards Service after Upgrading the Package

[msproact]

Describe the bug
I accept you to not consider this a bug at all and argue that this is desired behavior. I still like to bring this up for your consideration.

On upgrading from OpenSearch 2.7 to 2.8 the pre removal script stopped the OpenSearch service. However it is then not restarted automatically after the update.

To Reproduce

1. Have a running OpenSearch 2.7 installation.
2. Run apt update
3. Run apt full-upgrage to upgrade to OpenSearch 2.8 (or later I bet).

Expected behavior
One can argue what the best behavior would be here.

What I see with upgrades of Elasticsearch Debian packages is that it does not stop the Elasticsearch service at all. This gives me an opportunity to reduce downtime. I usually get away with first doing the package upgrade while the Elasticsearch service is still running. Then I review configuration changes with vimdiff. And then I either restart the service or in case I have a kernel upgrade as well the whole system, probably after disabling shard allocation and/or pausing background work¹. One could consider this approach as being unclean or unsafe. However in practice it worked for Elasticsearch just fine for me.

Another approach would be to still stop the OpenSearch service but then start it after the update in case the Systemd service is enabled.

I'd prefer the first approach to leave the service state alone during the upgrade. That leaves it up to the admin to decide how to approach the upgrade. Of course the documentation should have some best practice hints.

[1] See https://www.elastic.co/guide/en/elastic-stack/8.8/upgrading-elasticsearch.html

Plugins
Not relevant for this topic.

Screenshots
Excerpt of console output:

Preparing to unpack .../opensearch_2.8.0_amd64.deb ...
Running OpenSearch Pre-Removal Script
Stop existing opensearch.service
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.8.0) over (2.7.0) ...
Setting up opensearch (2.8.0) ...
Installing new version of config file /etc/opensearch/opensearch-notifications-core/notifications-core.yml ...
Installing new version of config file /etc/opensearch/opensearch-performance-analyzer/opensearch_security.policy ...
Installing new version of config file /etc/opensearch/opensearch-security/roles.yml ...

Configuration file '/etc/opensearch/opensearch.yml'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** opensearch.yml (Y/I/N/O/D/Z) [default=N] ?
Running OpenSearch Post-Installation Script
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd

State of Systemd service after upgrade from OpenSearch 2.7 to 2.8:

% systemctl status opensearch
● opensearch.service - OpenSearch
     Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Wed 2023-06-07 09:36:59 CEST; 1min 40s ago
       Docs: https://opensearch.org/
   Main PID: 5887 (code=exited, status=143)
        CPU: 22min 18.888s  

[…]
Jun 07 09:36:59 somehost systemd[1]: Stopping OpenSearch...
Jun 07 09:36:59 somehost systemd[1]: opensearch.service: Succeeded.
Jun 07 09:36:59 somehost systemd[1]: Stopped OpenSearch.
Jun 07 09:36:59 somehost systemd[1]: opensearch.service: Consumed 22min 18.888s CPU time

Host/Environment (please complete the following information):

• OS: Debian
• Version 11

Additional context
Not required here I think.

[prudhvigodithi]

[Untriage]
Adding @peterzhuamazon, peter can you please add your thoughts?

[slalomsk8er]

I can confirm the problem and our Graylog test server was non functional for 3 weeks.

[slalomsk8er]

it happend again...

[peterzhuamazon]

Hi @slalomsk8er ,

That is expected behavior as our service file does not enable opensearch by default.
Even if you manually enable systemctl enable opensearch.service I believe it will just restart service on (re)boot.
We do not restart the service on upgrade.

In our design we do not start opensearch service by default as we want users to control the runs of the software. I think you can write in your upgrade script to restart the service after upgrade if that is suitable.

Thanks.

[slalomsk8er]

Hi @peterzhuamazon,

This is very unfortunate as this is the only service known to me, that isn't restarted after update on a Debian system.
The consequence is that my installation of Graylog isn't working properly, every time this package gets updated.

What is the reasoning to have a service and want to have the users to control the runs of the software? To me those are mostly mutually exclusive.

I will consult with my integrator about this issue but I guess, if it isn't save enough for the opensearch-project to automatically restart the service, we will switch to manual updates for this package.

Thanks for any clarification.

[peterzhuamazon]

Hi @slalomsk8er,

Thanks for the responses.

The initial design was based on our initial thoughts, and a restart on update complete, as you mentioned, is surely convenient to the end users.

Since this is an open source project, we welcome community feedbacks and improvements.

We welcome you to send a possible PR if you are able to suggest a fix to this.

Or I can take a look on this when I have some time / bandwidth.

Let me know,

Thanks.

[slalomsk8er]

I ran out of time but found some nice diagrams that could help MaintainerScripts.
I'm not sure if the postinst needs to handle it or a trigger and I couldn't find the answer in openssh_9.2p1-2.debian.tar.xz so maybe it's a implicit file trigger or I'm way off.

[peterzhuamazon]

I ran out of time but found some nice diagrams that could help MaintainerScripts. I'm not sure if the postinst needs to handle it or a trigger and I couldn't find the answer in openssh_9.2p1-2.debian.tar.xz so maybe it's a implicit file trigger or I'm way off.

To my understanding if I put the trigger in postinst, then both install and upgrade will be affected unless I have conditions around them. This would make the script way too complicated to maintain.

I am trying to find out if there is any systemd service related changes so if a service is already started, then an upgrade will just auto trigger the restart.

[slalomsk8er]

I think I misremembered and Debian doesn't auto restart services.
They just don't get stoppend and tools like https://wiki.debian.org/UnattendedUpgrades manage restarting of services and rebooting. Just keep the service running on update?

If you insist in stopping the service I think the post install script receives a argument that lets you know if it's a install or update.

[peterzhuamazon]

I think I misremembered and Debian doesn't auto restart services. They just don't get stoppend and tools like https://wiki.debian.org/UnattendedUpgrades manage restarting of services and rebooting. Just keep the service running on update?

If you insist in stopping the service I think the post install script receives a argument that lets you know if it's a install or update.

It is more like the postrm script. I add a service stop there so it will not have cases where user uninstall the pkg but the service still running with a zombie process in bg.

[peterzhuamazon]

I do, however, can try to see if we can add a condition to understand if it is a install/upgrade. But that might be different case between deb and rpm. If I have to change deb I have to change rpm as well to keep the consistency.

Will check.

[slalomsk8er]

https://www.debian.org/doc/manuals/debian-faq/pkg-basics.en.html#maintscripts

postinst
This script typically completes any required configuration of the package foo once foo has been unpacked from its Debian archive (".deb") file. Often, 'postinst' scripts ask users for input, and/or warn them that if they accept default values, they should remember to go back and re-configure that package as needed. Many 'postinst' scripts then execute any commands necessary to start or restart a service once a new package has been installed or upgraded.

Maybe I wasn't hallucinating self restarting services after all.

https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html

postinst configure most-recently-configured-version

So maybe the most-resently-configured-version < then current version means a update?

[smortex]

Not starting the service on installation makes sense because on first start OS would bootstrap some config and data before the end-user had the opportunity to tune the system (e.g. change default insecure passwords). While these changes can be made later, this is a rather painful process, so violating the Debian policy of starting services on installation makes sense in some cases like this one. I believe this behavior should be kept.

But if the OpenSearch service is running, it should be restarted during the update to minimize downtime.

A similar behavior of not starting on install but restarting on upgrade is implemented in the following non-debian-provided packages I install on my Debian nodes for configuration management: Puppet & Choria: both need some "seeding" before they can start (the puppet agent needs to know which server it should talk to, and Choria must be configured via Puppet). The following PR implemented such a behavior for Choria in order to mimic the Puppet one: choria-io/go-choria#1392. I believe we can adopt a similar behavior.

This can be part of the current WIP to improve OS / OSD packaging.

[peterzhuamazon]

Adding @wieso-itzi to the conversation as he seems to have experience with this.

• [Bug]: Debian Package - no service restart after apt upgrade #4515

Thanks 😄

[wieso-itzi]

Hi @peterzhuamazon , thanks for adding me to this issue. I've had a look at your Debian maintainer scripts and tweaked them a bit on my local system to check them out. I can provide 2 different working solutions depending on what the preferred behavior is: Restarting the service if it is active or restarting the service if it is enabled.

• Restarting the service if it was active: basically this would be accomplished by changing prerm and preinst to not stop the service on upgrades and letting postinst handle the full restart of the service (e. g. if systemctl is-active returns true). The big drawback of this approach is that on an upgrade the prerm script of the already installed package version is called (see https://www.debian.org/doc/debian-policy/ap-flowcharts.html) which stops the service, so "systemctl is-active" in postinst would return false. This makes it so that this approach would be effective earliest in 2 releases (or rather in 2 "apt upgrades", which could be the same as 2 releases or more if a user skips upgrades), but if that's ok to you (since it would just mean the existing behaviour would be prolonged for one additional release) and it is a requirement for the service to be restarted if it was active it can be done pretty easily.

• Restarting the service if it was enabled: this is more straightforward since it only requires a change in postinst to check whether the service is enabled and if yes to start the service. Also it would be effective immediately since this functionality is not broken by the behavior of the maintainer scripts of already installed package versions. But of course this would also mean that if the service was stopped after being enabled and an upgrade was installed, the service would be started which might not be the desirable behavior in every circumstance (and judging by the previous messages here restarting the service if it was previously active may be the more intuitive functionality)

Please let me know which of the 2 you prefer and I'll provide you with a PR.

Thanks

[peterzhuamazon]

Hi @peterzhuamazon , thanks for adding me to this issue. I've had a look at your Debian maintainer scripts and tweaked them a bit on my local system to check them out. I can provide 2 different working solutions depending on what the preferred behavior is: Restarting the service if it is active or restarting the service if it is enabled.

* Restarting the service if it was active: basically this would be accomplished by changing prerm and preinst to not stop the service on upgrades and letting postinst handle the full restart of the service (e. g. if systemctl is-active returns true). The big drawback of this approach is that on an upgrade the prerm script of the already installed package version is called (see https://www.debian.org/doc/debian-policy/ap-flowcharts.html) which stops the service, so "systemctl is-active" in postinst would return false. This makes it so that this approach would be effective earliest in 2 releases (or rather in 2 "apt upgrades", which could be the same as 2 releases or more if a user skips upgrades), but if that's ok to you (since it would just mean the existing behaviour would be prolonged for one additional release) and it is a requirement for the service to be restarted if it was active it can be done pretty easily.

* Restarting the service if it was enabled: this is more straightforward since it only requires a change in postinst to check whether the service is enabled and if yes to start the service. Also it would be effective immediately since this functionality is not broken by the behavior of the maintainer scripts of already installed package versions. But of course this would also mean that if the service was stopped after being enabled and an upgrade was installed, the service would be started which might not be the desirable behavior in every circumstance (and judging by the previous messages here restarting the service if it was previously active may be the more intuitive functionality)

Please let me know which of the 2 you prefer and I'll provide you with a PR.

Thanks

Hi @wieso-itzi,

Thanks for the comments. From my point of view, the if enabled option is more in line with the original design. The original design will always check if service is started and attempt to stop service when running prerm. It is good to ensure that service is fully stopped before removing the pkg, same with upgrades.

And this also gives users an option to decide if they want such behavior to happen.

Note that if this works in deb I would also love to copy it into rpm in similar ways so rpm and deb is consistent with this behavior.

cc: @bbarani @prudhvigodithi on the above proposal and give some opinions.

Thanks.

[peterzhuamazon]

More comments in:

• Automatic Service Restart when Service is Enabled (Debian) #4530

[peterzhuamazon]

More discussion:

• Automatic Service Restart when Service is Enabled (Debian) #4530 (comment)

[peterzhuamazon]

Re-open for RPM.

[peterzhuamazon]

After testing, this method does not seem to scale well with RPM due to RPM is actually verifying the old installation after new upgrades is already installed.

Several times the postin restart of the new pkg would be interrupted by the preun stop from the old pkg, such as:

Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : opensearch-dashboards-2.13.0-1.x86_64                                                                                                                                                                                                                                                                    1/2
Restarting opensearch-dashboards.service after upgrade
### Breaking change in packaging since 2.13.0
 In 2.13.0 and later releases of OpenSearch Dashboards, we have changed the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch Dashboards configuration files, we recommend you add the user that runs these tools to the 'opensearch-dashboards' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043
Stop existing opensearch-dashboards.service
  Cleanup    : opensearch-dashboards-2.11.1-1.x86_64                                                                                                                                                                                                                                                                    2/2
  Verifying  : opensearch-dashboards-2.13.0-1.x86_64                                                                                                                                                                                                                                                                    1/2
  Verifying  : opensearch-dashboards-2.11.1-1.x86_64                                                                                                                                                                                                                                                                    2/2

Updated:
  opensearch-dashboards.x86_64 0:2.13.0-1

Note that the 2.11.1 stop happened actually after 2.13.0 start.

Will close this issue for now as rpm is not processing this as stable as deb.

Thanks.

[peterzhuamazon]

• Update deb/rpm autorestart service after upgrade documentation documentation-website#6720

[pitch-d]

how can i disable the restart after upgrade?
i'm using ansible for deploy and upgrade but now the restart when upgraded will fail the playbook

[wieso-itzi]

Hi @pitch-d ,

the condition that is checked for whether the service will be restarted after an upgraded is the "enabled" state of the SystemD-Unit (as returned by systemctl is-enabled opensearch.service). So as of now, I guess, a possibility would be to add tasks for disabling/re-enabling the service to the playbook to get the behaviour previous to OpenSearch 2.13.0