Mitigate enterprise anti-malware problems

[gnachman]

Summary

Every time I update my app, I get many responses from users whose upgrades failed because SentinelOne quarantined the new version. They have to beg their administrators to add the app to an allow-list and try again. It would be nice if there were a mechanism to detect this problem and educate the users about it instead of leaving them with a broken installation.

Possible Fix

After unzipping the app, SentinelOne will delete the main app binary. I propose that an app which uses sparkle be given the ability to opt into an extra verification step. After unzipping the payload, wait for a configurable delay and then check that it is intact (could check code signature or verify that the files in the app bundle match the files in the zip). If it detects a problem, show a message like "It looks like the app update was quarantined by your anti-malware. Please contact your system administrator".

[zorgiepoo]

I've been bitten by SentinelOne as well, although through a different issue.

I don't like the idea of adding a delayed/periodic code signing check in the installer tool (but if one were to be added I think it'd be automatic and not configurable -- and I think the message ought to be reworded; it could also be malware doing this, or something else going catastrophically wrong).

I am also not confident an additional check will help much, if such software can/will damage the app after Sparkle has installed the new app in /Applications, or wherever the old app was originally, instead of in the staging (Caches) area, which can be a race. (And there may be a lack of proof it's damaging the app before Sparkle moves it versus after the move. I understand you are getting reports from the app being damaged after the user manually unzips a downloaded app but that may not be the same circumstance).

There are some potential app tampering protections but they are kind of moot as software like this is typically given full access / permissions to the system.

[zorgiepoo]

After some more investigation, I noticed that Sparkle can install a corrupt bundle (corrupted by SentinelOne) by this path:

I'm aware of this logic. If (Ed)DSA signature verification succeeds (update is trusted at that point), Sparkle does a shallow code signing check to validate basic integrity and that the developer didn't make a mistake shipping their app with an obviously malformed bundle. It checks if the update is signed since there are apps that are not code signed. (This is not a security check, and Sparkle does not intend to fully validate the bundle or identify who signed it in this case). The update validation in Sparkle is complicated, partly due to supporting rotation of (Ed)DSA and Apple code signing. Changes to this path will require careful consideration.

At any rate, if this problem is fixed then it's possible that Sparkle could already detect SentinelOne's shenanigans without having to introduce other changes.

This does not prove that SentinelOne actually does corrupt the update before Sparkle moves it. And if it does, there could still be a realistic race at the point of time verification is done. I'm also aware there may be some differences between Sparkle 1 and 2 regarding where the update is staged (depending if authorization is used) or when verification is done. The Sparkle 1 path is no longer being iterated on.

[gnachman]

Race conditions are unavoidable when trying to solve this problem. I don't believe a 100% perfect solution is possible, but I would happily settle for a 95% solution.

Assuming SentinelOne watches the caches folder that Sparkle extracts to: if an app could opt in to requiring code signing for updates via an Info.plist key then some amount of SentinelOne's damage would be mitigated. The actual amount won't be known until the code is out in the world. Introducing a configurable delay would let developers trade off between performance and broken installs.

Since I run a fork of Sparkle anyway, I've decided to take a more aggressive approach which I don't think is appropriate to upstream but I'll share it in case anyone else is interested: gnachman@218c669

It'll be two updates before I know if this approach is successful. I'll report back when I know.

[zorgiepoo]

Reflecting more, regardless of this issue, I think modifying Sparkle's policy to do (something like)

if (hostIsCodeSigned && !updateIsCodeSigned) {
        SULog(SULogLevelError, @"The original application is code signed but the update does not have a valid signature.");
        return NO;
    }

..should be pretty reasonable if Ed(DSA) verification succeeded in general. This shouldn't trip up adhoc signatures too, which should be used these days for those that have no Apple code signing certificate.

[zorgiepoo]

Since I run a fork of Sparkle anyway, I've decided to take a more aggressive approach which I don't think is appropriate to upstream but I'll share it in case anyone else is interested: gnachman@218c669

Some of these changes are overkill, even for a fork. Sleeping for 10 sec on the main thread (or any other thread) is a very bad idea (particularly also for users that have that SentinelOne set up in a way that allow-listed your app). Since your fork is based on Sparkle 1, I believe Sparkle 1 validates the update when the user clicks install and relaunch, which for it is "as late as possible" within your app but something could theoretically go wrong while installing the app still or I guess if the user instantly clicks the button (this is also partly a reason why installing updates is so slow in Sparkle 1 and can cause a short hang on install/quit for validating the update if you are observant). Better to check if sentientel one is present after checking the update is not code signed, too.

edit: I would also caution you against hardcoding "SentinelOne" strings or looking if their processes are alive in your program. That sounds like a great potential way to be even more flagged than you currently are.

[zorgiepoo]

If a currently impacted user can run (if they're allowed to):

sudo fs_usage -w | grep $YOUR_APP_NAME > ~/Desktop/logs.txt

then, download and install an update in your app, have SentinelOne tamper the app,

and then Control-C that process, I believe we get diagnostics for at least which part of the update process SentinelOne is interfering and replacing the binary on their machine.