diff --git a/README.md b/README.md index 16ebd00..e8fa5bc 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,14 @@ -# Email OAuth 2.0 Proxy +# Email OAuth 2.0 Proxy Transparently add OAuth 2.0 support to IMAP/POP/SMTP client applications, scripts or any other email use-cases that don't support this authentication method. -## Motivation and capabilities +## Motivation and capabilities Email services that support IMAP, POP and/or SMTP access are increasingly requiring the use of OAuth 2.0 to authenticate connections, but not all clients support this method. -This script creates a simple local proxy that intercepts the traditional IMAP/POP/SMTP authentication commands and transparently replaces them with the appropriate SASL (X)OAuth 2.0 commands and credentials. +This tool creates a simple local proxy that intercepts the traditional IMAP/POP/SMTP authentication commands and transparently replaces them with the appropriate SASL (X)OAuth 2.0 commands and credentials. Your email client can continue to use the `login` or `auth`/`authenticate` options, with no need to make it aware of OAuth's existence. -The proxy works in the background with a menu bar/taskbar helper or as a system service, and is compatible with macOS, Windows and Linux. +The proxy works in the background with a menu bar/taskbar helper or as a headless system service, and is compatible with macOS, Windows and Linux. -### Example use-cases +### Example use-cases - You need to use an Office 365 email account, but don't get on with Outlook. The email client you like doesn't support OAuth 2.0, which became mandatory [in January 2023](https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-deprecation-in-exchange-online-september/ba-p/3609437). - You used to use Gmail via IMAP/POP/SMTP with your raw account credentials (i.e., your real password), but cannot do this now that Google has disabled this method, and don't want to use an [App Password](https://support.google.com/accounts/answer/185833) (or cannot enable this option). @@ -17,32 +17,31 @@ You can edit the server details, but the client forces you to delete and re-add - You have made your own script or application that sends or receives email, but it doesn't support OAuth 2.0, and you don't want to have to modify it to implement this. - You work with multiple services or applications that use IMAP/POP/SMTP, and you don't want to have to set up OAuth 2.0 independently for each one. -In all of these cases and more, this proxy script can help. -Follow the instructions below to get started, and please [open an issue](https://github.com/simonrob/email-oauth2-proxy/issues) with any problems or suggestions. +In all of these cases and more, this proxy can help. +Follow the instructions here to get started, and please [open an issue](https://github.com/simonrob/email-oauth2-proxy/issues) with any problems or suggestions. For commercial support or feature requests, please also consider [sponsoring this project](https://github.com/sponsors/simonrob?frequency=one-time). -## Getting started +## Getting started Begin by downloading the proxy via one of the following methods:
    -
  1. Pick a pre-built release for your platform (no installation needed); or,
    2. -
  2. Install from PyPI: python -m pip install emailproxy[gui] to set up, then python -m emailproxy to run; or,
    3. -
  3. Clone or download (and star :-) the GitHub repository, then: python -m pip install -r requirements-core.txt -r requirements-gui.txt to install requirements, and python emailproxy.py to run
    4. +
  4. Pick a pre-built release for your platform (macOS or Windows; no installation needed); or,
    5. +
  5. Install from PyPI: set up using python -m pip install emailproxy[gui], download the sample emailproxy.config file, then python -m emailproxy to run; or,
    6. +
  6. Clone or download (and star :-) the GitHub repository, then: python -m pip install -r requirements-core.txt -r requirements-gui.txt to install requirements, and python emailproxy.py to run.
-If you choose download option (A) or (B), you should also [download the sample `emailproxy.config` file](https://github.com/simonrob/email-oauth2-proxy/raw/main/emailproxy.config) and place this into the directory you will run the proxy from. -Next, edit the `emailproxy.config` file to add configuration details for each email server and account that you want to use with the proxy. -[Guidance and example account configurations](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) are provided for Office 365, Gmail and several other providers, though you will need to insert your own client credentials for each one (see the [documentation below](https://github.com/simonrob/email-oauth2-proxy/#oauth-20-client-credentials)). +Next, edit the sample `emailproxy.config` file to add configuration details for each email server and account that you want to use with the proxy. +[Guidance and example account configurations](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) are provided for Office 365, Gmail and several other providers, though you will need to insert your own client credentials for each one (see the [client credentials documentation](#oauth-20-client-credentials) for guidance). You can remove details from the sample configuration file for services you don't use, or add additional ones for any other OAuth 2.0-authenticated IMAP/POP/SMTP servers you would like to use with the proxy. -You can now start the proxy: depending on which installation option you chose, either open the application or use the appropriate run command listed above. +You can now start the proxy: depending on which installation option you chose, either launch the application or use the appropriate run command listed above. A menu bar/taskbar icon should appear. -If instead of the icon you see an error notification, it is likely that your system is missing dependencies for the `pywebview` or `pystray` packages. -See the [dependencies and setup](https://github.com/simonrob/email-oauth2-proxy/#dependencies-and-setup) section below for help resolving this, and also the [advanced configuration](https://github.com/simonrob/email-oauth2-proxy/#advanced-configuration) section for additional options (including fully headless deployments and integration with a secrets manager). +If this does not happen, see the [dependencies and setup](#dependencies-and-setup) section for help resolving this. +For additional options, including fully headless deployments and integration with a secrets manager, see the [optional arguments](#optional-arguments-and-configuration) and [advanced configuration](#advanced-configuration)) sections. Finally, open your email client and configure its server details to match the ones you set in the proxy's configuration file. -The correct server to use with an account is identified using the port number you select in your client – for example, to use the sample Office 365 details, this would be `localhost` on port `1993` for IMAP, port `1995` for POP and port `1587` for SMTP. +The correct server to use with an account is identified using the port number you select in your client – for example, to use the sample Office 365 details, this would be `127.0.0.1` on port `1993` for IMAP, port `1995` for POP and port `1587` for SMTP. The proxy supports multiple accounts simultaneously, and all accounts associated with the same provider can share the same proxy server. The local connection in your email client should be configured as unencrypted to allow the proxy to operate, but the connection between the proxy and your email server is always secure (implicit SSL/TLS for IMAP and POP; implicit or explicit (STARTTLS) SSL/TLS for SMTP). See the [sample configuration file](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) for additional documentation about advanced features, including local encryption, account configuration inheritance and support for running in a container. @@ -50,48 +49,48 @@ See the [sample configuration file](https://github.com/simonrob/email-oauth2-pro The first time your email client makes a request you should see a notification from the proxy about authorising your account. Click the proxy's menu bar icon, select your account name in the `Authorise account` submenu, and then log in via the popup browser window that appears. The window will close itself once the process is complete. -See the various [optional arguments](https://github.com/simonrob/email-oauth2-proxy/#optional-arguments-and-configuration) below for completing authentication if running without a GUI. +See the various [optional arguments](#optional-arguments-and-configuration) for support completing authentication if running without a GUI. After successful authentication and authorisation you should have IMAP/POP/SMTP access to your account as normal. -Make sure you keep the proxy running at all times to allow it to authorise your email client's background activity – enable `Start at login` from the proxy's menu, or see the [instructions below](https://github.com/simonrob/email-oauth2-proxy/#starting-the-proxy-automatically) about how to configure this in various different setups. +Make sure you keep the proxy running at all times to allow it to authorise your email client's background activity – enable `Start at login` from the proxy's menu, or see the [auto-start instructions](#starting-the-proxy-automatically) about how to configure this in various different setups. After your accounts are fully set-up and authorised, no further proxy interaction should be required unless your account needs authorising again. It will notify you if this is the case. -### OAuth 2.0 client credentials +### OAuth 2.0 client credentials As part of the proxy setup process you need to provide an OAuth 2.0 `client_id` and `client_secret` to allow it to authenticate with email servers on your behalf. If you have an existing client ID and secret for a desktop app, you can use these directly in the proxy. If this is not possible, you can also reuse the client ID and secret from any email client that supports IMAP/POP/SMTP OAuth 2.0 authentication with the email server you would like to connect to (such as [the](https://github.com/mozilla/releases-comm-central/blob/812b7c9068ca5cac0580b0ddbea8e34c141cd441/mailnews/base/src/OAuth2Providers.jsm) [various](https://github.com/Foundry376/Mailspring/blob/master/app/internal_packages/onboarding/lib/onboarding-constants.ts) [open](https://gitlab.gnome.org/GNOME/evolution-data-server/-/blob/master/CMakeLists.txt) [source](https://gitlab.gnome.org/GNOME/gnome-online-accounts/-/blob/master/meson_options.txt) [clients](https://github.com/M66B/FairEmail/blob/master/app/src/main/res/xml/providers.xml) with OAuth 2.0 support), but please do this with care and restraint as access through reused tokens will be associated with the token owner rather than your own client. If you do not have access to credentials for an existing client you will need to register your own. -The process to do this is different for each provider, but the registration guides for several common ones are linked below. +The process to do this is different for each provider, but the registration guides for several common ones are linked here. In all cases, when registering, make sure your client is set up to use an OAuth scope that will give it permission to access IMAP/POP/SMTP as desired. It is also highly recommended to use a scope that will grant "offline" access (i.e., a way to [refresh the OAuth 2.0 authentication token](https://oauth.net/2/refresh-tokens/) without user intervention). The [sample configuration file](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) provides example scope values for several common providers. -- Office 365: register a new [Microsoft identity application](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app) +- Office 365: register a new [Microsoft identity application](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) - Gmail / Google Workspace: register a [Google API desktop app client](https://developers.google.com/identity/protocols/oauth2/native-app) - AOL and Yahoo Mail (and subproviders such as AT&T) are not currently allowing new client registrations with the OAuth email scope – the only option here is to reuse the credentials from an existing client that does have this permission. -The proxy also supports the [client credentials grant](https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow) and [resource owner password credentials grant](https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth-ropc) OAuth 2.0 flows if needed. +The proxy also supports the [client credentials grant](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow) and [resource owner password credentials grant](https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth-ropc) OAuth 2.0 flows if needed. Please note that currently only Office 365 is known to support these methods. In addition, when using the client credentials grant flow, Office 365 only supports IMAP/POP, [_not_ SMTP](https://learn.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth#use-client-credentials-grant-flow-to-authenticate-imap-and-pop-connections) (use [smtp2graph](https://github.com/EvanTrow/smtp2graph) instead here). See the [sample configuration file](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) for further details. -## Optional arguments and configuration +## Optional arguments and configuration When starting the proxy there are several optional arguments that can be set to customise its behaviour. - `--no-gui` will launch the proxy without an icon, which allows it to be run as a `systemctl` service as demonstrated in [this example](https://github.com/simonrob/email-oauth2-proxy/issues/2#issuecomment-839713677), or fully headless as demonstrated in [various](https://github.com/michaelstepner/email-oauth2-proxy-aws) [other](https://github.com/blacktirion/email-oauth2-proxy-docker) subprojects. -Please note that on its own this mode is only of use if you have already authorised your accounts through the proxy in GUI mode, or are importing a pre-authorised proxy configuration file from elsewhere. -Unless this option is used in conjunction with `--external-auth` or `--local-server-auth`, accounts that have not yet been authorised (or for whatever reason require re-authorisation) will time out when authenticating, and an error will be printed to the log. +Please note that unless you also specify one of the authorisation options below, this mode is only of use if you have already authorised your accounts through the proxy in GUI mode, or are importing a pre-authorised proxy configuration file from elsewhere. +If you do not set `--external-auth` or `--local-server-auth`, accounts that have not yet been authorised (or for whatever reason require re-authorisation) will time out when authenticating, and an error will be printed to the log. - `--external-auth` configures the proxy to present an account authorisation URL to be opened in an external browser and wait for you to copy+paste the post-authorisation result. -In GUI mode this can be useful in situations where the script's own browser window does not have access to some required authentication attribute of your typical setup. -In no-GUI mode this option allows you to authenticate accounts entirely externally (unlike `--local-server-auth`, which starts a local web server), though you will need to monitor the proxy's log for authentication notifications. +In GUI mode this can be useful in situations where the proxy's own browser window does not have access to some required authentication attribute of your typical setup. +In no-GUI mode this option allows you to authenticate accounts entirely externally (unlike `--local-server-auth`, which starts a local web server), though you will need to monitor the proxy's output and/or log for authentication notifications. - After visiting the link provided and authorising account access, paste the final URL from your browser's address bar back into the script's popup window (GUI mode) or the terminal (no-GUI mode) to give it access to transparently proxy your login. + After visiting the link provided and authorising account access, paste the final URL from your browser's address bar back into the proxy's popup window (GUI mode) or the terminal (no-GUI mode) to give it access to transparently proxy your login. You should ignore any browser error message that is shown (e.g., `unable to connect`); the important part is the URL itself. This argument is identical to enabling external authorisation mode from the `Authorise account` submenu of the proxy's menu bar icon. @@ -102,60 +101,61 @@ Please note that while authentication links can actually be visited from anywher See the [sample configuration file](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) for advanced options to configure this (via `redirect_listen_address`). - `--config-file` allows you to specify the location of a [configuration file](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) that the proxy should load. -By default, the proxy also saves its cached OAuth 2.0 tokens back to this file, so it must be writable. +If this argument is not provided, the proxy will look for `emailproxy.config` in its working directory. +By default, the proxy also saves cached OAuth 2.0 tokens back to this file, so it must be writable. See the `--cache-store` option, if you would rather store configuration and cached values separately. -If this argument is not provided, the proxy will look for `emailproxy.config` in the same directory as the script itself. - `--cache-store` is used to specify a separate location in which to cache authorised OAuth 2.0 tokens and associated metadata. -The value of this argument can either be the full path to a local file (which must be writable), or an identifier for an external store such as a secrets manager (see the [documentation below](https://github.com/simonrob/email-oauth2-proxy/#advanced-configuration)). +The value of this argument can either be the full path to a local file (which must be writable), or an identifier for an external store such as a secrets manager (see the [advanced configuration](#advanced-configuration) section). If this argument is not provided, credentials will be cached in the current configuration file. - `--log-file` allows you to specify the location of a file to send log output to (full path required). Log files are rotated at 32MB and 10 older log files are kept. -This option overrides the proxy's default behaviour, which varies by platform (see [below](https://github.com/simonrob/email-oauth2-proxy/#troubleshooting) for details). +This option overrides the proxy's default behaviour, which varies by platform (see [below](#troubleshooting) for details). -- `--debug` enables debug mode, printing more verbose output to the log as [discussed below](https://github.com/simonrob/email-oauth2-proxy/#troubleshooting). +- `--debug` enables debug mode, printing more verbose output to the log as [discussed below](#troubleshooting). This argument is identical to enabling debug mode from the proxy's menu bar icon. -### Advanced configuration +### Advanced configuration The [example configuration file](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) contains further documentation for various additional features of the proxy, including catch-all (wildcard) accounts, locally-encrypted connections and advanced Office 365 OAuth 2.0 flows. The proxy caches authenticated OAuth 2.0 tokens and associated metadata back to its own configuration file by default, but can alternatively be configured to use either a separate local file or a secrets manager service for this purpose. Currently only [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) is supported for remote token storage. -To use this feature, set the [`--cache-store`](https://github.com/simonrob/email-oauth2-proxy/#optional-arguments-and-configuration) parameter to either a full ARN or a secret name, prefixing the value with `aws:` to identify its type to the proxy. +To use this feature, set the [`--cache-store`](#optional-arguments-and-configuration) parameter to either a full ARN or a secret name, prefixing the value with `aws:` to identify its type to the proxy. You must also install the AWS SDK for Python: `python -m pip install boto3` and [set up authentication credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configuration) (including a region). The minimum required permissions for the associated AWS IAM user are `secretsmanager:GetSecretValue` and `secretsmanager:PutSecretValue`. If the named AWS Secret does not yet exist, the proxy will attempt to create it; here, the `secretsmanager:CreateSecret` permission is also required. If you are using the proxy in a non-GUI environment it is possible to skip installation of dependencies that apply only to the interactive version. -To do this, install the script via `python -m pip install emailproxy` (i.e., without the `[gui]` variant option), and pass the [`--no-gui`](https://github.com/simonrob/email-oauth2-proxy/#optional-arguments-and-configuration) argument when starting the proxy. +To do this, install via `python -m pip install emailproxy` (i.e., without the `[gui]` variant option), and pass the [`--no-gui`](#optional-arguments-and-configuration) argument when starting the proxy. Please note that the proxy was designed as a GUI-based tool from the outset due to the inherently interactive nature of OAuth 2.0 authorisation, and there are limits to its ability to support fully no-GUI operation. -See the [optional arguments and configuration](https://github.com/simonrob/email-oauth2-proxy/#optional-arguments-and-configuration) section of this file for further details. +See the [optional arguments and configuration](#optional-arguments-and-configuration) section of this file for further details. -If your network requires connections to use an existing proxy, you can instruct the script to use this by setting the [proxy handler](https://docs.python.org/3/library/urllib.request.html#urllib.request.ProxyHandler) environment variable `https_proxy` (and/or `http_proxy`) – for example, `https_proxy=localhost python emailproxy.py`. +If your network requires connections to use an existing proxy, you can instruct the script to use this by setting the [proxy handler](https://docs.python.org/3/library/urllib.request.html#urllib.request.ProxyHandler) environment variable `https_proxy` (and/or `http_proxy`) – for example, `https_proxy=localhost python -m emailproxy`. -After installing its requirements, the proxy can be packaged as a single self-contained executable using [pyinstaller](https://pyinstaller.org/) if desired: `pyinstaller --onefile emailproxy.py`. +After installing its requirements, the proxy script can be packaged as a single self-contained executable using [pyinstaller](https://pyinstaller.org/) if desired: `pyinstaller --onefile emailproxy.py`. If you are using the GUI version of the proxy, you may need to add `--hidden-import timeago.locales.en_short` until [this `timeago` issue](https://github.com/hustcc/timeago/issues/40) is resolved. Python 3.6 or later is required to run the proxy. -The [python2 branch](https://github.com/simonrob/email-oauth2-proxy/tree/python2) provides minimal compatibility with python 2.7, but with a limited feature set, and only very occasional maintenance. +The [python2 branch](https://github.com/simonrob/email-oauth2-proxy/tree/python2) provides minimal compatibility with python 2.7, but with a limited feature set, and no ongoing maintenance. See [issue 38](https://github.com/simonrob/email-oauth2-proxy/issues/38) for further discussion. -### Starting the proxy automatically +### Starting the proxy automatically In order for the proxy to authenticate background requests from your email client it needs to be kept running constantly. The easiest way to do this is to start the script automatically. In GUI mode the proxy has basic support for this built-in: click its menu bar icon and then select `Start at login`, which will stop the terminal instance and restart the script, configuring it to run each time you log in. -On macOS, if you are presented with a prompt about file access here, make sure you grant this so that python can run the script in the background. -For more advanced configurations, you may want to customise the startup behaviour and edit the script's parameters – see the sections below for further information on how to achieve this using macOS, Windows or Linux. +On macOS, if you are presented with a prompt about file access here, make sure you grant this so that python can run the proxy in the background. +For more advanced configurations, you may want to customise the startup behaviour and edit the script's parameters. +The method to achieve this differs depending on whether you are using macOS, Windows or Linux. On macOS, the file `~/Library/LaunchAgents/ac.robinson.email-oauth2-proxy.plist` is used to configure automatic starting of the proxy. If you stop the proxy's service (i.e., `Quit Email OAuth 2.0 Proxy` from the menu bar), you can restart it using `launchctl start ac.robinson.email-oauth2-proxy` from a terminal. You can stop, disable or remove the service from your startup items either via the menu bar icon option, or using `launchctl unload [plist path]`. If you edit the plist file manually, make sure you `unload` and then `load` it to update the system with your changes. -If the `Start at login` option appears not to be working for you on macOS, see the [known issues](https://github.com/simonrob/email-oauth2-proxy/#known-issues) section below for potential solutions. +If the `Start at login` option appears not to be working for you on macOS, see the [known issues section](#known-issues) for potential solutions. On Windows the auto-start functionality is achieved via a shortcut in your user account's startup folder. -Pressing `⊞ Win` + `r` and entering `shell:startup` (and then clicking OK) will open this folder – from here you can either double-click the `ac.robinson.email-oauth2-proxy.cmd` file to relaunch the proxy, edit it to configure, or delete this file (either manually or by deselecting the option in the proxy's menu) to remove the script from your startup items. +Pressing `⊞ Win` + `r` and entering `shell:startup` (and then clicking OK) will open this folder – from here you can either double-click the `ac.robinson.email-oauth2-proxy.cmd` file to relaunch the proxy, edit it to configure, or delete this file (either manually or by deselecting the option in the proxy's menu) to remove the proxy from your startup items. On Linux this feature assumes that your system supports XDG Autostart. A Desktop Entry file `ac.robinson.email-oauth2-proxy.desktop` will be created in `~/.config/autostart/`. @@ -163,12 +163,12 @@ Use the proxy's menu option (or manually remove this file) to prevent it startin It is also possible to run the proxy as a service (e.g., via `systemctl`) – see the `--no-gui` mode option above for more details. -## Troubleshooting -If you encounter problems using the proxy, enabling `Debug mode` from the menu or passing `--debug` as a script argument will print all client–proxy–server communication to your system log to help identify the issue. +## Troubleshooting +If you encounter problems using the proxy, enabling `Debug mode` from the menu or passing `--debug` as a command line argument will print all client–proxy–server communication to your system log to help identify the issue. This will include all commands to and responses from the server (and also as a result the content of your email messages). On macOS this can be viewed using Console.app or `log stream --predicate 'subsystem == "ac.robinson.email-oauth2-proxy"' --level=debug`. -On Windows a file `emailproxy.log` will be created in the same location as the proxy script (see also the `--log-file` option). +On Windows a file `emailproxy.log` will be created in the same location as the proxy (see also the `--log-file` option). On Linux you can use, for example, `tail -f /var/log/syslog | grep "Email OAuth 2.0 Proxy"`. Please note that debug mode may also result in your login credentials being printed to the log (though this is avoided where possible). @@ -178,11 +178,11 @@ Because of this, if you are concerned about debug mode and security you can use It is often helpful to be able to view the raw connection details when debugging (i.e., without using your email client). This can be achieved using `telnet`, [PuTTY](https://www.chiark.greenend.org.uk/~sgtatham/putty/) or similar. -For example, to test the Office 365 IMAP server from the [example configuration](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config), first open a connection using `telnet localhost 1993`, and then send a login command: `a1 login e@mail.com password`, replacing `e@mail.com` with your email address, and `password` with any value you like during testing (see above for why the password is irrelevant). +For example, to test the Office 365 IMAP server from the [example configuration](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config), first open a connection using `telnet 127.0.0.1 1993`, and then send a login command: `a1 login e@mail.com password`, replacing `e@mail.com` with your email address, and `password` with any value you like during testing (see above for why the password is irrelevant). If you have already authorised your account with the proxy you should see a response starting with `a1 OK`; if not, this command should trigger a notification from the proxy about authorising your account. If you are using a [secure local connection](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) the interaction with the remote email server is the same as above, but you will need to use a local debugging tool that supports encryption. -The easiest approach here is to use [OpenSSL](https://www.openssl.org/): `openssl s_client -crlf -connect localhost:1993`. +The easiest approach here is to use [OpenSSL](https://www.openssl.org/): `openssl s_client -crlf -connect 127.0.0.1:1993`. If you are having trouble actually connecting to the proxy, it is always worth double-checking the `local_address` values that you are using. The [sample configuration file](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) sets this parameter to `127.0.0.1` for all servers. @@ -190,8 +190,8 @@ If you remove this value and do not provide your own, the proxy defaults to `::` If you are unable to connect to the proxy from your client, it is always worth first specifying this value explicitly – see the [sample configuration file](https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) for further details about how to do this. Please try setting and connecting to both IPv4 (i.e., `127.0.0.1`) and IPv6 (i.e., `::1`) loopback addresses before reporting any connection issues with the proxy. -### Dependencies and setup -On macOS the setup and installation instructions above should automatically install all required dependencies. +### Dependencies and setup +On macOS the proxy's setup and installation instructions should automatically bundle or install all required dependencies. Any error messages you may encounter (for example, with your `pip` version and `cryptography`, or `pillow` and `imagingft` dependencies, or [macOS SSL failures](https://github.com/simonrob/email-oauth2-proxy/issues/14#issuecomment-1077379254)) normally give clear explanations of the issues and point to instructions for resolving these problems. Please [open an issue](https://github.com/simonrob/email-oauth2-proxy/issues) if you encounter any other problems here. @@ -203,14 +203,14 @@ A similar issue may occur on Windows with the [pythonnet](https://github.com/pyt If you are unable to resolve this by following the [pythonnet installation instructions](https://github.com/pythonnet/pythonnet/wiki/Installation), you may find that installing a [prebuilt wheel](https://www.lfd.uci.edu/~gohlke/pythonlibs/#pythonnet) helps fix the issue. Note that the public releases of pythonnet can take some time to be compatible with the latest major python release, so it can be worth using a slightly older version of python, or a pre-release version of pythonnet. -### Known issues +### Known issues - With some combinations of operating systems, web engines and virtual environments, keyboard control or input to the proxy's popup authorisation window may not always work. On Windows this is normally limited to keyboard shortcuts (i.e., copy/paste), but in some virtual environments on macOS the entire keyboard may not work. -As a workaround, the proxy will enable pywebview's debug mode when you run the proxy in debug mode, which should allow you to use the right-click context menu to copy/paste to enter text. +As a workaround, the proxy will enable pywebview's debug mode when you run the proxy itself in debug mode, which should allow you to use the right-click context menu to copy/paste to enter text. If you are unable to proceed with popup-based authentication even with this workaround, it is worth trying the proxy's `--external-auth` or `--local-server-auth` options. - On more recent macOS versions (10.14 and later), you may find that when first running the proxy as a service you need to manually load its launch agent in order to trigger a file access permission prompt. -You will know if intervention is necessary if the proxy exits (rather than restarts) the first time you click `Start at login` from its menu bar icon. +You will know intervention is necessary if the proxy exits (rather than restarts) the first time you click `Start at login` from its menu bar icon. To resolve this, exit the proxy and then run `launchctl load ~/Library/LaunchAgents/ac.robinson.email-oauth2-proxy.plist` from a terminal. A permission pop-up should appear requesting file access for python. Once this has been approved, the proxy's menu bar icon will appear as normal. @@ -220,27 +220,26 @@ If this still does not cause the prompt to appear, the only currently-known reso You may also need to edit the proxy's launch agent plist file, which is found at the location given in the command above, to set the path to your python executable – it must be the real path rather than a symlink (the `readlink` command can help here). Fortunately this is a one-time fix, and once the proxy loads successfully via this method you will not need to adjust its startup configuration again (except perhaps when upgrading to a newer major macOS version, in which case just repeat the procedure). -### Other problems +### Other problems Please feel free to [open an issue](https://github.com/simonrob/email-oauth2-proxy/issues) reporting any bugs you find, or [submit a pull request](https://github.com/simonrob/email-oauth2-proxy/pulls) to help improve this tool. -## Advanced / experimental features -The [plugins variant](https://github.com/simonrob/email-oauth2-proxy/tree/plugins) has a semi-experimental new feature that enables the use of separate scripts to modify IMAP/POP/SMTP commands when they are received from the client or server before passing through to the other side of the connection. +## Advanced features +The [plugins variant](https://github.com/simonrob/email-oauth2-proxy/tree/plugins) has an additional feature that enables the use of separate scripts to modify IMAP/POP/SMTP commands when they are received from the client or server before passing through to the other side of the connection. This allows a wide range of additional capabilities or triggers to be added the proxy. For example, the [IMAPIgnoreSentMessageUpload plugin](https://github.com/simonrob/email-oauth2-proxy/blob/plugins/plugins/IMAPIgnoreSentMessageUpload.py) intercepts any client commands to add emails to the IMAP sent messages mailbox, which resolves message duplication issues for servers that automatically do this when emails are received via SMTP (e.g., Office 365, Gmail, etc). -The [IMAPCleanO365ATPLinks plugin](https://github.com/simonrob/email-oauth2-proxy/blob/plugins/plugins/IMAPCleanO365ATPLinks.py) restores links modified by Office 365 Advanced Threat Protection to their original URLs. +The [IMAPCleanO365ATPLinks plugin](https://github.com/simonrob/email-oauth2-proxy/blob/plugins/plugins/IMAPCleanO365ATPLinks.py) restores "Safe Links" modified by Microsoft Defender for Office 365 to their original URLs. The [SMTPBlackHole plugin](https://github.com/simonrob/email-oauth2-proxy/blob/plugins/plugins/SMTPBlackHole.py) gives the impression emails are being sent but actually silently discards them, which is useful for testing email sending tools. See the [documentation and examples](https://github.com/simonrob/email-oauth2-proxy/tree/plugins/plugins) for further details, additional sample plugins and setup instructions. -## Potential improvements (pull requests welcome) -- Switch to asyncio? (asyncore is currently deprecated, but [PEP 594](https://peps.python.org/pep-0594/) will remove it completely in Python 3.12 – currently mitigated by the use of [pyasyncore](https://pypi.org/project/pyasyncore/)) +## Potential improvements (pull requests welcome) - Full feature parity on different platforms (e.g., live menu updating; monitoring network status; clickable notifications) +- Switch to asyncio? (with Python 3.12, [PEP 594](https://peps.python.org/pep-0594/) removed the asyncore package that the proxy is built upon – currently mitigated by the use of [pyasyncore](https://pypi.org/project/pyasyncore/)) - STARTTLS for IMAP/POP? -- Python 2 support? (see [discussion](https://github.com/simonrob/email-oauth2-proxy/issues/38)) -## Related projects and alternatives +## Related projects and alternatives Michael Stepner has created a [Terraform configuration](https://github.com/michaelstepner/email-oauth2-proxy-aws) that helps run this proxy on a lightweight cloud server (AWS EC2). Thiago Macieira has provided a [makefile and systemd configuration files](https://github.com/thiagomacieira/email-oauth2-proxy/tree/Add_a_Makefile_and_systemd_configuration_files_to_install_system_wide). For Docker, blacktirion has an [example configuration](https://github.com/blacktirion/email-oauth2-proxy-docker). @@ -254,5 +253,5 @@ That approach is very useful in situations where server-side IMAP/POP/SMTP is no This proxy was developed to work around these limitations for providers that do support IMAP/POP/SMTP natively. -## License +## License [Apache 2.0](https://github.com/simonrob/email-oauth2-proxy/blob/main/LICENSE) diff --git a/emailproxy.config b/emailproxy.config index 9909f1a..d4e4862 100644 --- a/emailproxy.config +++ b/emailproxy.config @@ -15,23 +15,23 @@ documentation = Local servers are specified as demonstrated below where, for exa Multiple accounts can share the same server, however. Each server section must specify the `server_address` and `server_port` of the remote server that it will be proxying - you can obtain these values from your email provider, or use the details below for Office 365 and/or Gmail. - + To allow the proxy to operate, your email client must be set up to use an unencrypted connection for IMAP/SMTP/POP (i.e., no STARTTLS or SSL/TLS, just plain login credentials). The proxy will create a secure connection on your behalf to the remote server from the outset by default (i.e., implicit SSL/TLS); see below if STARTTLS is required. - + Server customisation: - If your SMTP server uses the STARTTLS approach, add `starttls = True`, as shown in the [SMTP-1587] example below (assumed to be False otherwise). With this parameter set, STARTTLS negotiation will be handled by the proxy on your behalf (i.e., do not enable STARTTLS in your client). IMAP STARTTLS and POP STARTTLS are not currently supported. - + - The `local_address` property can be used to set an IP address or hostname for the proxy to listen on. Both IPv4 and IPv6 are supported. If not specified, this value is set to `::` (i.e., dual-stack IPv4/IPv6 on all interfaces). When a hostname is set the proxy will first resolve this to an IP address, preferring IPv6 over IPv4 if both are available. When running in an IPv6 environment with dual-stack support, the proxy will attempt to listen on both IPv4 and IPv6 hosts simultaneously. Note that tools such as `netstat` do not always accurately show dual-stack mode; if you are having trouble connecting to the proxy, it is worth actually testing both IPv4 and IPv6 connections. - + Advanced server configuration: - In the standard configuration the channel between your email client and the proxy is unencrypted. This is not normally of any concern since the proxy is typically a local-only service. However, if you prefer, you may provide @@ -77,7 +77,7 @@ documentation = Accounts are specified using your email address as the section h Each account section must provide values for `permission_url`, `token_url`, `oauth2_scope` and `redirect_uri`. If you are adding an account for a service other than the examples shown below then the provider's documentation should provide these details. - + You will also need to add your own `client_id` and `client_secret` values as indicated below. These can either be reused from an existing source (such as another email client that supports OAuth 2.0), or you can register and use your own desktop app API client credentials. See https://developers.google.com/identity/protocols/oauth2/native-app @@ -86,38 +86,38 @@ documentation = Accounts are specified using your email address as the section h `allow_catch_all_accounts` option. Note that while there are example account configurations for AOL and Yahoo Mail below, these services are not currently accepting new OAuth 2.0 client registrations with the mail access scope, so reusing existing credentials is the only option here. - + Once the proxy is correctly configured, after the first successful use of an account its access token details will be cached for future use, encrypted with the IMAP/POP/SMTP password you used in your email client. By default this configuration file is reused for caching (so it must be writable), but you can specify a different location or method using the proxy's `--cache-store` parameter. You should not add or edit cached values manually (i.e., `token_salt`, `access_token`, `access_token_expiry`, `refresh_token` and `last_activity`); the proxy handles this. - + The password used in your email client is not used for authentication with the actual email server (this is done via OAuth 2.0 in a web browser), so it can be different to your real account password, which is helpful for debugging. Please note, though, that all clients that use a particular account via the proxy should use the same IMAP/POP/SMTP password to avoid repeated re-authentication requests (which is the proxy's default behaviour when credential decryption fails). See the proxy's README.md file for more information and the end of this file for further options. - + Office 365 customisation: - Unlike other providers, Office 365 requires an OAuth 2.0 scope that explicitly specifies `offline_access` (shown in the example below) in order to allow the proxy to refresh its access token on your behalf. The proxy will still work if this parameter is not included, but you will need to re-authenticate extremely often (about once per hour). - - - By default, new Azure AD clients are accessible only within your own tenant. If you are registering a new client - to use with the proxy (and do not want to make it available outside your own organisation) you will need to replace - `common` with your tenant ID in the Office 365 `permission_url` and `token_url` values below. Alternatively, you can - reuse credentials from an existing client registration (see the proxy's README.md file), or configure your client - as a multi-tenant application. For more detail about this, and guides for setting up your desktop app client, see - the documentation at https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app. - + + - By default, new Entra (Azure AD) clients are accessible only within your own tenant. If you are registering a new + client to use with the proxy (and do not want to make it available outside your own organisation) you will need to + replace `common` with your tenant ID in the Office 365 `permission_url` and `token_url` values below. Alternatively, + you can reuse credentials from an existing client registration (see the proxy's README.md file), or configure your + client as a multi-tenant application. For more detail about this, and guides for setting up your desktop app client, + see the documentation at https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app. + - Office 365 shared mailboxes are supported: add an account entry here using the email address of the shared mailbox as the account name. When asked to authenticate, log in as the user that access has been delegated to. Note that Office 365 no-longer supports the `authorised.user@example.com/delegated.mailbox` username syntax here. - + - It is possible to create Office 365 clients that do not require a secret to be sent. If this is the case for your setup, delete the `client_secret` line from your account's configuration entry (do not leave the default value). - + - The proxy supports the client credentials grant (CCG) and resource owner password credentials grant (ROPCG) OAuth 2.0 flows (both currently only known to be available for Office 365). To use either of these flows, add an account entry as normal, but do not add a `permission_url` value (it does not apply, and its absence signals to the proxy to @@ -130,20 +130,20 @@ documentation = Accounts are specified using your email address as the section h of this file. In addition, if you are using the proxy in an environment where there is any possibility of malicious access attempts before the first valid login then pre-encrypting account entries is highly recommended - see the example setup script at https://github.com/simonrob/email-oauth2-proxy/issues/61#issuecomment-1259110336. - + Advanced account configuration: - For most configurations the default `redirect_uri` value of `http://localhost` is correct, unless you have explicitly set your OAuth 2.0 client to use a different address here (which you will need to manually redirect to the proxy). Please also note that this address is `http` and not `https` (which is not supported unless you provide a secure connection yourself outside of the proxy using an external tool). - + - When using the `--local-server-auth` option you will need to either run the proxy with additional privileges to use the implicit default port 80 (e.g., via sudo) or specify a different port (and/or a different host if needed) - for example, `redirect_uri = http://localhost:8080`. In addition, if you are using this proxy option in a setup that is not directly exposed (such as a container or private network) you may also need to map `redirect_uri` traffic via a private address. Once this is configured (via external tools/rules), instruct the proxy to listen here using the parameter `redirect_listen_address` - for example, `redirect_listen_address = http://10.0.0.0:8080`. - + - Please note that when using `--local-server-auth` the proxy will start/stop a new local server for each incoming authentication request, and does not explicitly handle repeated requests or multiple accounts authenticating using the same address. To avoid clashes, it is recommended that each account has a unique `redirect_uri` (or @@ -194,7 +194,7 @@ client_secret = *** your client secret here *** documentation = The parameters below control advanced options for the proxy. In most cases you will not need to modify the values in this section. If any of these values are not found, the proxy will assume the default value, which can be found in the example config file (https://github.com/simonrob/email-oauth2-proxy/blob/main/emailproxy.config) - + Parameter documentation: - delete_account_token_on_password_error (default = True): Once an account has been authenticated via the proxy, if there is a login attempt with a different (local) password, its default behaviour is to remove the saved OAuth 2.0 @@ -205,7 +205,7 @@ documentation = The parameters below control advanced options for the proxy. In malicious or not. It can also be the source of confusion if using a client (such as Firefox) that stores a separate password per protocol for each account, but does not make this clear when changing account passwords. Set this option to False and the proxy will instead return an error when an incorrect password is provided. - + - encrypt_client_secret_on_first_use (default = False): The proxy encrypts sensitive configuration values (e.g., cached access tokens) using the password that is given when accessing an account via IMAP/POP/SMTP. It does not do this for values that are not sensitive. In the most common operation mode (i.e., interactively authorising account @@ -217,7 +217,7 @@ documentation = The parameters below control advanced options for the proxy. In that this option is not compatible with `allow_catch_all_accounts` unless all accounts use the same login password. In addition, if you are using the proxy's `--cache-store` parameter you will need to manually remove unencrypted secrets from this configuration file after the encrypted secret has been created (i.e., it will not be automatic). - + - allow_catch_all_accounts (default = False): The default behaviour of the proxy is to require a full separate configuration file entry for each account. However, when proxying multiple accounts from the same domain it can be cumbersome to have to create multiple near-identical configuration profiles. To simplify this the proxy supports diff --git a/emailproxy.py b/emailproxy.py index 831d7d9..3e001b3 100644 --- a/emailproxy.py +++ b/emailproxy.py @@ -6,7 +6,7 @@ __author__ = 'Simon Robinson' __copyright__ = 'Copyright (c) 2023 Simon Robinson' __license__ = 'Apache 2.0' -__version__ = '2023-11-01' # ISO 8601 (YYYY-MM-DD) +__version__ = '2023-11-18' # ISO 8601 (YYYY-MM-DD) __package_version__ = '.'.join([str(int(i)) for i in __version__.split('-')]) # for pyproject.toml usage only import abc @@ -60,18 +60,25 @@ # where not having to install GUI-only requirements can be helpful - see the proxy's readme (the `--no-gui` option) MISSING_GUI_REQUIREMENTS = [] -try: - import pystray # the menu bar/taskbar GUI -except ImportError as gui_requirement_import_error: - MISSING_GUI_REQUIREMENTS.append(gui_requirement_import_error) - +no_gui_parser = argparse.ArgumentParser(add_help=False) +no_gui_parser.add_argument('--no-gui', action='store_false', dest='gui') +no_gui_args = no_gui_parser.parse_known_args()[0] +if no_gui_args.gui: + try: + import pystray # the menu bar/taskbar GUI + except Exception as gui_requirement_import_error: # see #204 - incomplete pystray installation can throw exceptions + MISSING_GUI_REQUIREMENTS.append(gui_requirement_import_error) + no_gui_args.gui = False # we need the dummy implementation +if not no_gui_args.gui: class DummyPystray: # dummy implementation allows initialisation to complete class Icon: pass pystray = DummyPystray # this is just to avoid unignorable IntelliJ warnings about naming and spacing +del no_gui_parser +del no_gui_args try: # noinspection PyUnresolvedReferences @@ -104,7 +111,7 @@ class Icon: try: # PyUnresolvedReferences; see: youtrack.jetbrains.com/issue/PY-11963 (same for others with this suppression) # noinspection PyPackageRequirements,PyUnresolvedReferences - import PyObjCTools # SIGTERM handling (only needed when in GUI mode; `signal` is sufficient otherwise) + import PyObjCTools.MachSignals # SIGTERM handling (only needed in GUI mode; `signal` is sufficient otherwise) except ImportError as gui_requirement_import_error: MISSING_GUI_REQUIREMENTS.append(gui_requirement_import_error) @@ -233,8 +240,9 @@ def initialise(log_file=None): Log._LOGGER = logging.getLogger(APP_NAME) if log_file or sys.platform == 'win32': handler = logging.handlers.RotatingFileHandler( - log_file or '%s/%s.log' % (os.path.dirname(sys.executable if getattr(sys, 'frozen', False) else - os.path.realpath(__file__)), APP_SHORT_NAME), + log_file or os.path.join(os.getcwd() if __package__ is not None else + os.path.dirname(sys.executable if getattr(sys, 'frozen', False) else + os.path.realpath(__file__)), '%s.log' % APP_SHORT_NAME), maxBytes=LOG_FILE_MAX_SIZE, backupCount=LOG_FILE_MAX_BACKUPS) handler.setFormatter(logging.Formatter('%(asctime)s: %(message)s')) @@ -2262,7 +2270,9 @@ class App: """Manage the menu bar icon, server loading, authorisation and notifications, and start the main proxy thread""" def __init__(self, args=None): - global CONFIG_FILE_PATH, CACHE_STORE + global CONFIG_FILE_PATH, CACHE_STORE, EXITING, prompt_toolkit + EXITING = False # needed to allow restarting when imported from parent scripts (or an interpreter) + parser = argparse.ArgumentParser(description='%s: transparently add OAuth 2.0 support to IMAP/POP/SMTP client ' 'applications, scripts or any other email use-cases that don\'t ' 'support this authentication method.' % APP_NAME, add_help=False, @@ -2806,7 +2816,10 @@ def get_script_start_command(self, quote_args=True): script_command = [python_command] if not getattr(sys, 'frozen', False): # no need for the script path if using pyinstaller - script_command.append(os.path.realpath(__file__)) + if __package__ is not None: + script_command.extend(['-m', APP_SHORT_NAME]) + else: + script_command.append(os.path.realpath(__file__)) # preserve any arguments - note that some are configurable in the GUI, so sys.argv may not be their actual state script_command.extend(arg for arg in sys.argv[1:] if arg not in ('--debug', '--external-auth')) @@ -3150,8 +3163,6 @@ def exit(self, icon, restart_callback=None): if sys.platform == 'darwin' and self.args.gui and self.macos_unload_plist_on_exit: self.macos_launchctl('unload') - EXITING = False # to allow restarting when imported from parent scripts (or an interpreter) - if __name__ == '__main__': App() diff --git a/pyproject.toml b/pyproject.toml index d0f1faf..9890b73 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -27,14 +27,16 @@ classifiers = [ ] dynamic = ["dependencies", "optional-dependencies", "version"] +[tool.setuptools] +py-modules = ["emailproxy"] # ignore other folders in the root directory (note that some (such as "tests") *cannot* be ignored in the .tar.gz sdist without a MANIFEST.in) + [project.scripts] emailproxy = "emailproxy:App" -[project.urls] -"Homepage" = "https://github.com/simonrob/email-oauth2-proxy" -"Changelog" = "https://github.com/simonrob/email-oauth2-proxy/releases" +[project.urls] # shown in alphabetical order on PyPI, and there is no way to put a "Homepage" link first as with the setup.py approach (but see https://github.com/pypi/warehouse/blob/main/warehouse/templates/packaging/detail.html) "Documentation" = "https://github.com/simonrob/email-oauth2-proxy#email-oauth-20-proxy" -"Bug Tracker" = "https://github.com/simonrob/email-oauth2-proxy/issues" +"Release Notes" = "https://github.com/simonrob/email-oauth2-proxy/releases" +"Report Issues" = "https://github.com/simonrob/email-oauth2-proxy/issues" "Source Code" = "https://github.com/simonrob/email-oauth2-proxy" [tool.setuptools.dynamic]