DNS not resolved in Fedora

[anat0lius]

I have checked /etc/resolv.conf and I do have the name servers added like this:

nameserver x.x.x.x
nameserver y.y.y.y
search company.lan

I don't have anything in /etc/systemd/resolved.conf.

I'm able to connect via IP.

VPN configuration looks like:

host = x.comp-any.com
port = nnnn
username = uuuuu 
password = xxxxx 
trusted-cert = tttttt

I have tried systemd-resolve subdomain.domain.com and fails. But systemd-resolve domain.com does work.

Could you help me to find out where is the issue?

[fran-pastor]

it's not a fix, but my workaround for Fedora is.. . . disable systemd-resolved
procedure:

1. systemctl disable systemd-resolved
2. rm -vf /etc/resolv.conf
3. touch /etc/resolv.conf
4. reboot
   Now, /etc/resolv.conf are controlled by NetworkManager, you can retry to connect to VPN and test if you get the correct DNSs behaviour
   regards!

BONUS TRACK: same behaviour for Ubuntu and we apply the same workaround. To disable systemd-resolved in Ubuntu follow https://gist.github.com/zoilomora/f7d264cefbb589f3f1b1fc2cea2c844c

[DimitriPapadopoulos]

I think it's best to keep systemd-resolved and try to use resolveconf instead:

• Make sure openfortivpn is built with resolveconf support.
• Make sure resolveconf is available in the PATH.

[madrisan]

I have the same problem on Fedora 39 and noticed that resolvectl status does not show any DNS servers for the domain ppp0.
As a workaround I run (once the VPN tunnel is up and running):

sudo resolvectl dns ppp0 <ip-corp-dns-1> <ip-corp-dns-2>

Note that I also had to upgrade openfortivpn to the version 1.21.0 and add the option --pppd-accept-remote=1.
The version shipped by Fedora 39 is broken (because of the package ppp version 2.5.0, see PR#1148)

[DimitriPapadopoulos]

Note that I also had to upgrade openfortivpn to the version 1.21.0 and add the option --pppd-accept-remote=1.
The version shipped by Fedora 39 is broken (because of the package ppp version 2.5.0, see PR#1148)

You should not need both openfortivpn 1.21 and the option --pppd-accept-remote=1 as the ipcp-accept-remote pppd option is built in openfortivpn 1.21.

However, you do need support for systemd-resolved, which should be provided by resolvconf support. Try building openfortivpn with resolvconf support:

• either by making openresolv available at build-time and run-time,
• or by building using configure option --enable-resolvconf --with-resolvconf=/usr/sbin/resolvconf and making openresolv available at run-time.

@adrienverge Would it be possible to generate RPMs for Fedora and EPEL with openfortivpn 1.21?

• Fedora ≥ 39 ship pppd 2.5.0 and do not need configure option --enable-legacy-pppd.
• Fedora ≤ 38 ship pppd 2.4.9 and do require configure option --enable-legacy-pppd.
• For the time being, all EPEL packages shipd pppd < 2.4.9 and do require configure option --enable-legacy-pppd.

Also, all Fedora version require built-in resolvconf support:

• either by making openresolv a build-time requirement,
• or by building using configure option --enable-resolvconf --with-resolvconf=/usr/sbin/resolvconf.

[madrisan]

I've rebuilt the rpm package with the options --enable-resolvconf --with-resolvconf=/usr/sbin/resolvconf.
If I run the command

sudo openfortivpn -c /etc/openfortivpn/config --otp=123456

the tunnel is created successfully but I still need to run

sudo resolvectl dns ppp0 <ip-corp-dns-1> <ip-corp-dns-2>

for DNS to work.
resolvconf is installed:

$ ls -l /usr/sbin/resolvconf /etc/alternatives/resolvconf
lrwxrwxrwx. 1 root root 31 Sep 21 18:58 /etc/alternatives/resolvconf -> /usr/sbin/resolvconf.openresolv
lrwxrwxrwx. 1 root root 28 Nov  9 22:49 /usr/sbin/resolvconf -> /etc/alternatives/resolvconf

[DimitriPapadopoulos]

Ah, that's interesting. Right now, I don't know whether it means that openfortivpn does not call resolvconf correctly, or that resolvconf doesn't do the right thing on Fedora with systemd-resolved. Help debugging this would be welcome, as would be help replacing the current built-in routing and DNS code with calls to vpnc-script.

[madrisan]

Specifically on Fedora 39.
It worked flawlessly before I upgraded to this version a couple of days ago (using NetworkManager-fortisslvpn).
I'll be glad to help you debugging.

[madrisan]

It worked with Fedora 38 fully updated

$ rpm -q openfortivpn ppp NetworkManager-fortisslvpn
openfortivpn-1.19.0-1.fc38.x86_64
ppp-2.4.9-9.fc38.x86_64
NetworkManager-fortisslvpn-1.4.0-3.fc38.x86_64

and it's not working with Fedora 39 where:

$ rpm -q openfortivpn ppp NetworkManager-fortisslvpn
openfortivpn-1.19.0-2.fc39.x86_64
ppp-2.5.0-3.fc39.x86_64
NetworkManager-fortisslvpn-1.4.0-5.fc39.x86_64

Just rebuilding openfortivpn using the Fedora .spec file but with the version 1.21.0 vanilla, fixes all the issues but the DNS resolution.

[DimitriPapadopoulos]

Yes, I did expect 1.21 to fix some if not all issues, because it adds the ipcp-accept-remote pppd option that is required by pppd ≥ 2.5.0 but breaks pppd < 2.5.0.

The remaining issue is the DNS one. I need to know whether you can reproduce the DNS issue when running openfortivpn from the command line (without messing with NetworkManager-fortisslvpn).

[madrisan]

I switched to openfortivpn because NetworkManager-fortisslvpn does not work anymore.
So yes I can reproduce the bug using openfortivpn (1.21).

[DimitriPapadopoulos]

I see. There are two ways openfortivpn can modify DNS settings:

1. The traditional ugly way, by directly modifying /etc/resolv.conf, still works more or less on many Linux platforms but probably not recent Fedora distributions relying on systemd-resolved.
2. The alternative way is to entrust resolveconf to do the right thing.

I am not surprised 1 does not work any more on recent Fedora releases, but I was hoping 2 would work better. The openfortivpn very verbose logs (redacted if needed) will at least show what openfortivpn does, hopefully in enough detail to start understanding the issue at hand.

[madrisan]

[...]
DEBUG:  Retrieving configuration
DEBUG:  Found dns server 10.10.20.1 in xml config
DEBUG:  Found dns server 10.10.20.2 in xml config
DEBUG:  Establishing the tunnel
DEBUG:  ppp_path: /usr/sbin/pppd
DEBUG:  Switch to tunneling mode
DEBUG:  Starting IO through the tunnel
DEBUG:  pppd_read thread
DEBUG:  if_config thread
DEBUG:  ssl_read thread
DEBUG:  ssl_write thread
Using interface ppp0
Connect: ppp0 <--> /dev/pts/3
[...]
DEBUG:  pppd ---> gateway (12 bytes)
DEBUG:  gateway ---> pppd (12 bytes)
INFO:   Got addresses: [10.100.18.2], ns [10.10.20.1, 10.10.20.2]
INFO:   Negotiation complete.
DEBUG:  Got Address: 10.100.18.2
DEBUG:  if_config: not ready yet...
local  IP address 10.100.18.2
remote IP address 88.200.100.100
DEBUG:  pppd ---> gateway (42 bytes)
DEBUG:  pppd ---> gateway (124 bytes)
[...]
INFO:   Interface ppp0 is UP.
INFO:   Setting new routes...
[...]
INFO:   Adding VPN nameservers...
DEBUG:  Attempting to run /usr/sbin/resolvconf.
DEBUG:  resolvconf_call: /usr/sbin/resolvconf -a "ppp0.openfortivpn"
INFO:   Tunnel is up and running.

but the two DNS servers are not set

$ resolvectl status ppp0
Link 19 (ppp0)
    Current Scopes: LLMNR/IPv4
         Protocols: -DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported

[adrienverge]

@adrienverge Would it be possible to generate RPMs for Fedora and EPEL with openfortivpn 1.21?

Sure, I just created updates for Fedora 39 (new configure options for built-in resolvconf support), Fedora 38 (with options for built-in resolvconf support and --enable-legacy-pppd) and EPEL 9 (with option --enable-legacy-pppd). These updates are going to be pushed to testing repos in the next hours, and will be installable using:

sudo dnf install --refresh --enablerepo=updates-testing openfortivpn

[DimitriPapadopoulos]

@madrisan It might be that resolvconf does not work as expected, or that we don't pass proper arguments to it. In the first case, there is not much we can do but move to a different mechanism to handle DNS. I notice that openfortivpn runs /usr/sbin/resolvconf -a "ppp0.openfortivpn" while your run resolvectl status ppp0. So My first reaction is that the network interfaces names are different in either case.

From the man page of resolvctl:

[...]
resolvconf | [-m metric] [-p] [-x] -a interface[.protocol] <file
[...]

[madrisan]

I've tried with the patch

--- openfortivpn-1.21.0/src/ipv4.c      2023-11-08 07:42:39.000000000 +0100
+++ openfortivpn-1.21.0-ppp0/src/ipv4.c 2023-11-13 14:15:58.560815378 +0100
@@ -1105,7 +1105,7 @@
                }
 
                snprintf(resolvconf_call, resolvconf_call_len,
-                        "%s -a \"%s.openfortivpn\"",
+                        "%s -a \"%s\"",
                         RESOLVCONF_PATH,
                         tunnel->ppp_iface);

but the two DNS are still not configured. The log of openfortivpn is now showing as expected:

INFO:   Adding VPN nameservers...
DEBUG:  Attempting to run /usr/sbin/resolvconf.
DEBUG:  resolvconf_call: /usr/sbin/resolvconf -a "ppp0"
INFO:   Tunnel is up and running.

[scaronni]

NetworkManager-fortisslvpn is not working on Fedora 39. I have to use openfortivpn to connect to VPN now.

Yes, being tracked here: https://bugzilla.redhat.com/show_bug.cgi?id=2246228

Trying to figure out what needs to be changed to NetworkManager-fortisslvpn to make it work. By luck, I have admin rights to the package. As written in the ticket, openfortivpn works fine on its own, but it's called with --noroutes --no-dns --pppd-use-peerdns=1 to let routes and dns settings be managed by NetworkManager.

When called like this, NetworkManager does not set the routes properly and the VPN does not work.

In my case, I had to delete these two:

<vpn_endpoint> dev ppp0 proto kernel scope link src <ip_of_ppp0> 
<vpn_endpoint> dev ppp0 proto kernel scope link src <ip_of_ppp0> metric 50

After deleting these two, it works.

[scaronni]

Funny thing is that on a system with openfortivpn 1.19.0 (Fedora 38), the two routes are there with the automatic ip address:

169.254.2.1 dev ppp0 proto kernel scope link src <ip_of_ppp0> 
169.254.2.1 dev ppp0 proto kernel scope link src <ip_of_ppp0> metric 50

But everything works fine.

[scaronni]

I guess with 1.19.0 it worked because the route 169.254.0.0/24 is never hit. Let me explain. With 1.19.0 (where it works):

$ ip route
default dev ppp0 proto static scope link metric 50 
default via 192.168.1.1 dev wlp61s0 proto dhcp src 192.168.1.33 metric 600 
82.197.169.3 via 192.168.1.1 dev wlp61s0 proto static metric 50 
169.254.2.1 dev ppp0 proto kernel scope link src 10.161.232.33 
169.254.2.1 dev ppp0 proto kernel scope link src 10.161.232.33 metric 50 
192.168.1.0/24 dev wlp61s0 proto kernel scope link src 192.168.1.33 metric 600 
192.168.1.1 dev wlp61s0 proto static scope link metric 50 

The two routes with 169.254.2.1 are never hit, and the VPN gateway itself is found with the third rule (82.197.169.3 via 192.168.1.1) which bypasses the default dev ppp0 at the top.

With 1.21:

$ ip route
default dev ppp0 proto static scope link metric 50 
default via 192.168.1.1 dev wlp61s0 proto dhcp src 192.168.1.33 metric 600 
82.197.169.3 via 192.168.1.1 dev wlp61s0 proto static metric 50
82.197.169.3 dev ppp0 proto kernel scope link src 10.161.232.33
82.197.169.3 dev ppp0 proto kernel scope link src 10.161.232.33 metric 50 
192.168.1.0/24 dev wlp61s0 proto kernel scope link src 192.168.1.33 metric 600 
192.168.1.1 dev wlp61s0 proto static scope link metric 50

As you can see the two (useless) routes with 169.254.2.1 are replaced with the VPN gateway. The first of the 3 is correct (82.197.169.3 via 192.168.1.1), while the other two prevent it from working, as it tells the kernel to find the VPN endpoint at the end of ppp0, instead of through the local gateway.

[scaronni]

This seems to be due to this change: #1120 (comment)

Is there any reason why the two routes are being added? They were superfluous but harmless with 169.254.2.1, but they are now an issue with the VPN gateway address instead of 169.254.2.1.

[DimitriPapadopoulos]

What I know is that pppd might fail at run-time if:

• --pppd-accept-remote=1 and pppd < 2.5.0,
• --pppd-accept-remote=0 and pppd ≥ 2.5.0.

By failure, I mean a hard failure, where pppd, and subsequently openfortivpn, exit with a relevant error message. That's why it's best to make --pppd-accept-remote=1 the default at build-time when building for a target with pppd ≥ 2.5.0, and to make --pppd-accept-remote=0 the default at build-time when building for a target with pppd < 2.5.0 (using configure option --enable-legacy-pppd). You can check your situation by comparing what happens without --pppd-accept-remote, with --pppd-accept-remote=0 and with --pppd-accept-remote=1.

Next, I need to understand whether you have a routing problem (typically you cannot ping both IP addresses and DNS names) or a DNS problem (typically you cannot ping DNS names but can ping the associated IP addresses).

Indeed, specifically in the case of DNS problems, there is a second issue to explore, depending on which exact program attempts to change DNS parameters:

• with --set-dns=0 (same as --no-dns) and --pppd-use-peerdns=1, handling DNS parameters is delegated to pppd, and that's what NetworkManager-fortisslvpn does as far as I know,
• with --set-dns=1 and --pppd-use-peerdns=0, handling DNS is done directly by openfortivpn, either by modifying directly /etc/resolv.conf or by delegating to resolvectl,
• with --set-dns=1 and --pppd-use-peerdns=1, both might attempt to modify DNS parameters, and I don't know what exactly happens under the hood in that case, I just know it usually works.

Now, it might be that in the case where pppd handles DNS parameters, which is the NetworkManager-fortisslvpn case as far as I know, pppd is not able to handle and modify DNS parameters properly, or at least as we expect it to, with --pppd-accept-remote=1. At the same time, on platforms with pppd ≥ 2.5.0, openfortivpn --pppd-accept-remote=0 exits, often if not always, before even getting the chance to handle DNS parameters, so we cannot get back to --pppd-accept-remote=0.

Where I am getting to is;

1. Please only refer to the command line openfortivpn (using the NetworkManager option if you want to).
2. Decide if it looks like a routing problem or a DNS problem. Back it up with examples such as IP address pings and DNS names are not resolved.
3. It might be useful to test all (or almost all or only necessary) combinations of --set-dns=..., --pppd-use-peerdns=..., --pppd-accept-remote=... to get a clearer view of the problem at hand.
4. Providing logs will help, but as attached files and not copied/pasted in GitHub issues - otherwise I won't be able to manage the issue because of its size.

This issue has already been polluted by reports of possibly different issues, that do not seem identical to the initial issue. It has grown almost intractable using the time I can spend on it.

[DimitriPapadopoulos]

Perhaps we should discuss the specific case of NetworkManager-fortisslvpn failing, while openfortivpn works a standalone program from the command line, in a dedicated issue. Unless this is what is being discussed in this issue?

@anat0lius @madrisan At this point, I'm not sure what we are discussing and whether this is a single issue or multiple issues.

[DimitriPapadopoulos]

@Predrag No, I don't think you are doing anything wrong. If you can ping the IP addresses, but not the DNS names, then openfortivpn does not set DNS parameters properly. Traditionally, openfortivpn attempts to modify /etc/resolv.conf, which is of course a bad idea on newer Linux distributions. Optionally, openfortivpn may let resolvconf handle DNS parameters.

Is openresolv installed on your machine? If so, openfortivpn should use it. If you attach a (redacted) log, we should be able to see whether it does use resolvconf or not.

[madrisan]

@Predrag You can read my previous comment for a workaround: #1141 (comment)

[Predrag]

I have installed openresolv and also tried to connect with sudo openfortivpn host:port --pppd-accept-remote=1 --cookie="VALUE"

[madrisan]

@Predrag And now the DNS resolution is working ?
What is the output of the command readlink -f /usr/sbin/resolvconf ?
In my Fedora installation is /usr/sbin/resolvconf.openresolv but I'm curious to know the value on a fresh installed one (I upgraded several times: F37 >> F38 >> F39).

[Predrag]

It is still not working. I have the same output as you

[DimitriPapadopoulos]

I see. This probably means resolvconf does not do the job. It looks like we will have to call systemd directly.

[tallandtree]

I also had an issue with the latest openfortivpn 1.21.0 on Fedora 38. Note that I had to make a change to nsswitch.conf to get the nameservers working correctly before with versions prior to 1.21.0:

hosts:      files myhostname mdns4_minimal [NOTFOUND=return] dns

With version 1.21.0, name resolving didn't work anymore: nslookup on IP address did work, but nslookup on DNS name did not work. For now, I've rolled back to 1.19.0. But let me know if you want me to test anything.

[DimitriPapadopoulos]

@tallandtree Are you positive it works with openfortivpn 1.19.0, but not openfortivpn 1.21.0 on the same machine?

There are few differences between the two versions, mostly --pppd-accept-remote=1 being the default in 1.21.0, unless built and packaged otherwise. Does --pppd-accept-remote=0 help?

[Predrag]

With --pppd-accept-remote=1 or --pppd-accept-remote=0 I can not resolve dns name servers

[madrisan]

It works for me after setting set-dns = 1 in /etc/openfortivpn/config.

[DimitriPapadopoulos]

@Predrag That's quite disturbing as pppd-accept-remote is the only relevant difference I can find between the two versions.

@madrisan Note that set-dns = 1 is the default. You don't need to set it.

[madrisan]

@DimitriPapadopoulos You're right.
I added this setting today and noticed that the DNS for ppp0 was set correctly. But actually even without the option, DNS is set now! So it seems that in the last few days a Fedora update fixed the issue.

The packages openfortivpn and openresolv have not been updated

$ rpm -q --last openfortivpn openresolv
openfortivpn-1.21.0-2.fc39.x86_64             Wed 15 Nov 2023 11:26:25 CET
openresolv-3.13.2-2.fc39.noarch               Thu 09 Nov 2023 22:49:59 CET

but systemd and glibc were.

[tallandtree]

@tallandtree Are you positive it works with openfortivpn 1.19.0, but not openfortivpn 1.21.0 on the same machine?

There are few differences between the two versions, mostly --pppd-accept-remote=1 being the default in 1.21.0, unless built and packaged otherwise. Does --pppd-accept-remote=0 help?

Yes, if I use the option --pppd-accept-remote=0 it works with 1.21.0.

[tallandtree]

@DimitriPapadopoulos unfortunately, it doesn't work always with the --pppd-accept-remote=0. Now I've got the issue again. Can resolve ip addresses, but not the DNS names.

[tallandtree]

I've rolled back to 1.19 where it does work consistently.

[madrisan]

I tested openfortivpn with a fully updated Fedora 39

$ rpm -q --last openfortivpn openresolv systemd
systemd-254.7-1.fc39.x86_64                   Fri 01 Dec 2023 23:56:20 CET
openfortivpn-1.21.0-2.fc39.x86_64             Wed 15 Nov 2023 11:26:25 CET
openresolv-3.13.2-2.fc39.noarch               Thu 09 Nov 2023 22:49:59 CET

with and without my company DNS entries added to /etc/ppp/resolv.conf.
It works in both cases (tunnel setup and DNS resolution).
My /etc/openfortivpn/config just contains the values for

host
port
username
password
trusted-cert

and at command-line I just add an extra option: --otp.
resolvconf is a symlink to resolvectl (provided by the package systemd).

$ readlink -f /usr/sbin/resolvconf
/usr/bin/resolvectl

I hope it helps.

[DimitriPapadopoulos]

@madrisan If you have package openresolv installed, it should provide /usr/sbin/resolvconf. I'm surprised it is a symlink to resolvectl, but perhaps that's expected on a system based on systemd-resolved. What is the output of:

rpm -qf /usr/sbin/resolvconf

[madrisan]

$ rpm -qf /usr/sbin/resolvconf
openresolv-3.13.2-2.fc39.noarch
systemd-resolved-254.7-1.fc39.x86_64

[DimitriPapadopoulos]

I see. So both packages provide the file, I don't know where the logic for choosing is hidden.

Any way, package openresolv should not be needed on Fedora. The /usr/sbin/resolvconf symlink provided by systemd ought to be enough. Installing package openresolv is mostly needed on systems without a working systemd-resolved.

[madrisan]

It is probably due to the successive upgrades I have made from fedora 37 to 38 and finally to 39.

[DimitriPapadopoulos]

Does it still work for you if you uninstall package openresolv?

[madrisan]

Yes. Without the package openresolv

$ rpm -q openresolv
package openresolv is not installed
$ rpm -qf /usr/sbin/resolvconf
systemd-resolved-254.7-1.fc39.x86_64
$ readlink -f /usr/sbin/resolvconf
/usr/bin/resolvectl

The tunnel is successfully created as well as the DNS configuration for the interface ppp0.

[blauigris]

I have also tested with openfortivpn from the command line and it works like a charm. Curiously, doing the same with NetworkManager does not work, but I imagine that this is another problem. Thanks a lot for your efforts, very appreciated ! :)

[tallandtree]

@DimitriPapadopoulos Just to confirm: I've upgraded to fedora 40 and the latest version of openfortivpn 1.21.0 works well. I only needed to make one change after I setup the vpn, and that is to add another local search domain:

sudo resolvectl domain ppp0 <main.localdomain.local> <second.localdomain.eu>

It seems that with the older version (1.19.0) the dns search always went via ppp0 when VPN was up, but now only for the 'search' domain it goes via ppp0 and my guess is that our vpn service does not configure the search domains correctly.

[Predrag]

On Fedora 40 after I set up vpn I need to run this two commands:
sudo resolvectl domain ppp0 ~. and sudo resolvectl dns ppp0 <main.localdomain.local> <second.localdomain.eu>
as suggested in this https://community.fortinet.com/t5/Support-Forum/DNS-settings-on-Ubuntu-22-04-and-FortiClient-VPN-7-0-0-0018/m-p/279051/highlight/true#M222399

[Predrag]

Just question is it possible to run this commands after Tunel is up and running ? (sudo resolvectl domain ppp0 ~. and sudo resolvectl dns ppp0 <main.localdomain.local> <second.localdomain.eu>)