Update intranet to reflect upstream VPN changes

[nullnik-0]

Background

On Friday September 13th at 17h30 upstream IT sent and email to announce that, due to security concerns, they would be disabling the PolySSL VPN group for employees. This connection was disabled at 18h that same day (half an hour later.) Employees were told to instead use the PolyQuartz VPN, which uses an authentication flow that is not supported by openconnect.

Our current workflow at the lab encourages users to use openconnect because anyconnect is a proprietary software that requires root access to install and has many potentially very invasive endpoint monitoring and telemetry capabilities.

For those affected by the upstream change (as of writing, the full scope of this change is not clear), it means that we are now required use the cisco anyconnect client instead of openconnect

None of this is documented by our existing VPN docs. These docs need to be updated to relfect the new VPN requirements and how it will affect different users. However first we need to understand exactly who will be affected and how, and look at possible solutions that address user concerns about privacy.

Next steps

• Confirm whether or not upstream might be willing to be flexible about this. @jcohenadad has already contacted upstream about this.
• Determine who is affected and who will be affected in the future.
  • Right now we know for sure that employees (and people classed as employees for VPN purposes) are affected.
  • Anecdotally, it seems that students are not affected and this is reflected in the official Polytechique VPN docs
  • However, it is plausible that upstream intends to roll out these changes for students at some point in the future. We should try to confirm this if possible, so that we can have a better understanding of the potential scope of effects on the lab.
• Investigate whether it might be possible to reverse engineer the anyconnect auth flow and get it working with openconnect. (See this info and these discussions and projects for more context...
• Document new VPN workflows for different use cases and types of end users on the intranet.

Known issues and solutions

• I was concerned about the privacy implications of being forced to install anyconnect on my personal device (which is also my work device). I got anyconnect working properly in a vm, and tested with both NAT mode and bridge mode. This clearly not an ideal set up for many users.
• @mathieuboudreau had already installed anyconnect on a macos and found he needed to do the following to get anyconnect working with the PolyQuartz profile:
  1. Uninstall cisco anyconnect
  2. Open a terminal and run sudo rm -rf /opt/cisco
  3. Reinstall cisco anyconnect
  4. (annoyingly) add anyconnect to login items
  5. Type ssl.vpn.polymtl.ca in the anyconnect prompt
  6. Select PolyQuartz option
• @namgo is part of the PolyInvites group. This seems to have also been shut down by upstream, despite the fact that it was not mentioned in the original communiqué. As a contractor Nathan has not been included in the Okta transition and thus is not able to use the Okta-based auth flow under any circumstances. There is no know solution to this as of writing and Nathan has been functionally blocked from job-critical access to our infrastructure.
• It looks like interns are also affected by a similar issue as Nathan, and currently don't have a viable VPN option. Their existing VPN access has been revoked, but since they are not full employees, they have not been included in the Okta transition and thus cannot use PolyQuartz.

[namgo]

I'm in login group PolyInvites and am unable to log in, so I'm following this closely.

[nullnik-0]

I'm in login group PolyInvites and am unable to log in, so I'm following this closely.

Just added this under known issues. You should notify @jcohenadad and upstream about this problem asap.

[namgo]

you should notify Julien and upstream

I let Julien know, and I'm working through debugging what I can first. I might still have a way to sign up for Okta... or I might have been forgotten in that mix, we'll see.

[jcohenadad]

@namgo @nullnik-0 can you please document the suggestions that JS sent us after talking with Yves Simard? Thanks!

[nullnik-0]

@jcohenadad
Unfortunately the suggestion from JS and Yves Simard doesn't make a lot of sense. Their proposed solutions use a VPN protocol (OpenVPN) that is not compatible with the protocol they are using on their servers (Cisco AnyConnect). These protocols are not interoperable; by default a server configured for Cisco AnyConnect will not be able to support OpenVPN connections.

Unless upstream has their VPN servers configured to support two different VPN protocols (OpenVPN and Cisco AnyConnect)—which would technically be possible, but I think unlikely—then their suggested solution will not work for us.

I've drafted a message to upstream explaining this and asking for clarification. I believe @namgo was going to send it on my behalf tomorrow, but if you would like to forward please do!

Bonjour JS,

Les solutions qui nous sont proposées traitent toutes deux du protocole VPN OpenVPN et non des protocoles utilisés par Cisco AnyConnect, qui sont généralement incompatibles avec les connexions OpenVPN.

Pour clarifier, vos serveurs VPN sont-ils configurés pour prendre en charge les connexions OpenVPN ainsi que celles utilisées par Cisco AnyConnect ?

Sinon, les solutions utilisant des clients OpenVPN ne fonctionneront malheureusement pas.

Pour clarifier davantage, nous utilisons le client OpenConnect parce qu'il prend en charge spécifiquement les connexions Cisco AnyConnect.

Merci beaucoup,
Emma

[jcohenadad]

I've drafted a message to upstream explaining this and asking for clarification. I believe @namgo was going to send it on my behalf tomorrow, but if you would like to forward please do!

All good, I let you or Nathan send it. Just wanted to make sure we follow up with them. Thanks!

[namgo]

Sent! Alongside a brief reason for why I'm sending it and not Emma.

[nullnik-0]

Sent! Alongside a brief reason for why I'm sending it and not Emma.

Nathan got an answer back saying that OpenVPN is configured only for a special internal sysadmin group, and not for regular users. JS said it might be possible to add the two of us to that group, but unfortunately this means that OpenVPN is not going to be a viable solution for everyone else.

[nullnik-0]

Just added an update to the bottom of this issue about how it affects interns. NeuroPoly interns, I believe you are: @ArthurBoschet, @simonqueric, @KaterinaKrejci231054 , @CharlesPageot, and @abelsalm. Were you also in the PolyInvites group or had you been using PolySSL?

Also, if any of you get more information from upstream about the status of your VPN access, feel free to add an update here, so that we can share what we know more easily.

[nullnik-0]

Some Updates on various workarounds and solutions

Running AnyConnect client in VM and proxying traffic

Following an excellent suggestion from @namgo I set up the VM where I had already installed the AnyConnect VPN as an SSH server, and then configured it as a ProxyJump for my various connections to NeuroPoly resources.

So far I have successfully tested:

• regular ssh connections
• ssh connections from my ansible dev container
• forwarding RDP connections
• forwarding connections for the data web gui
• forwarding connection for duke and mounting on host machine
• remote git operations with data
• SOCKS proxy for other miscellaneous web connections

I have not tested forwarding connections for Poly's licensing server, but this should also work in theory...

Pros:

• Install AnyConnect in a isolated environment.
• Don't need to move your dev environment into a VM.
• Only send desired traffic through Poly VPN.

Cons:

• A moderate amount of work for initial set up.
• Still involves some manual steps to initiate (starting VM, starting VPN connection, initiating port forwarding for certain use cases).
• You still need to keep track of different port forwarding configurations for different kinds of connections and resources (using aliasing could make this easier).

Mathieu's manual workaround for OpenConnect

@mguaypaq has also come up with solution to get openconnect working with PolyQuartz. Right now it is a manual solution, but he points out that it could potentially automated with selenium

Mathieu's solution:

1. Visit https://ssl.vpn.polymtl.ca in a browser
2. Select PolyQuartz and log in with okta
3. In devtools, get the value of the webvpn cookie
4. Pass it to the following command on stdin (either by typing it, or piping to it):

sudo openconnect --protocol=anyconnect --authgroup=PolyQuartz --cookie-on-stdin https://ssl.vpn.polymtl.ca/

@joshuacwnewton and I have now both also tested this and confirm it works.

Pros:

• Integrates with openconnect, which we already use.
• One size fits all: no need for different port forwarding configurations for different resources.
• No need for virtualization or proxying (if not desired)
• Still compatible with above if desired (for set up similar to @namgo's previous work set up)
• Could be automated.
• If automated, probably much easier to implement for most lab members.

Cons:

• Right now it involves a lot of manual steps every time a VPN connection is initiated.

Questions

• Which solutions does it make sense to document for lab members?
• Should we automate @mguaypaq's solution and make that script available for lab members?
• Should we document this solution if we provide a script for automation?
• Should I document my solution using the official client, virtualization and a proxy jump? Would anyone actually want to reproduce this, or is too much work to implement?

[namgo]

Very well written! One thing I'd add is that you can use Linux Containers (lxc/lxd/incus and probably docker with added steps) which skips the need for hypervisor configuration. That's what I have been doing.

I was having a bit of trouble with this on my end due to running obscure non-systemd distros primarily (Void and Alpine), where anyconnect requires systemd hooks and Void doesn't play nice with systemd containers like the ubuntu base image.

[nullnik-0]

Very well written! One thing I'd add is that you can use Linux Containers (lxc/lxd/incus and probably docker with added steps) which skips the need for hypervisor configuration. That's what I have been doing.

Have you tested this with AnyConnect in particular though?

I opted for virtualization over containerization because I decided that it would be more straight-forward to deal with the graphical app (there's no fully-featured cli version of AnyConnect as far as I know?) and the web-based auth flow in a full Ubuntu desktop environment, instead of messing around with x11 forwarding to get both AnyConnect and a browser working in a headless container. Did you have a particular solution in mind for this?

[nullnik-0]

Just added an update to the bottom of this issue about how it affects interns. NeuroPoly interns, I believe you are: @ArthurBoschet, @simonqueric, @KaterinaKrejci231054 , @CharlesPageot, and @abelsalm. Were you also in the PolyInvites group or had you been using PolySSL?

JS has opened individual tickets for some of the affected interns. I have emailed him asking for more clarification about the "plusieurs configurations VPN en fonction de différents paramètres" he mentions, asking him to clarify which configuration is used for particular classes of NeuroPoly members (permanent staff, interns, contractors etc.)

We will need this information to be able to accurately document the new VPN procedures and constraints across various use cases.

[nullnik-0]

Bumping this for myself since I still really need to finish this. Will prioritize it first after dealing with urgent issues and setting up my new work computer. Since I opened this ticket there have been some new developments:

• I got a bit more info back from JS, which I will incorporate into the new docs
• There is now a new VPN group which some users (@abelsalm) have been placed in: PolyPhoton. I'll try to note this as well.
• @SomeoneInParticular has been working on automating @mguaypaq's work around. If this is ready to be shared before I finish the new docs, I'll include it. If not it can be added later.

[SomeoneInParticular]

Regarding @nullnik-0's remark, I have a working script up and ready (though only tested w/ Selenium's Firefox driver, and only on my Linux-like system).

To that ends does anyone have preference on whether the script should be hosted as a personal repository or on NeuroPoly (so it can be more easily maintained if/when I leave the lab)?

[mguaypaq]

I think it's fine to host it in a personal repository, since it's very unofficial. It's unlikely that anyone else will maintain the script, but if they do it's easy to fork the repository at that point.

[SomeoneInParticular]

The code has been uploaded and is available here. Feel free to test and push any feature requests/issues!