Solving 90% of GNS3 issues on Windows (x-post from GNS3 Community)

[eantowne]

Cross post from GNS3 Community Post

I am going to describe a solution for solving 90% of user experienced issues when using GNS3 on Windows. The method proposed will probably be an unpopular choice. I invite discussion regarding the pros and cons of this method.

Background:
90% of the problems that user experience when using GNS3 on Windows comes down to one thing: the GNS3 "Local" server. This is a GNS3 server running in Windows. It provides a couple of very useful capabilities, such as:

• Starting the GNS3 VM (VMware WS/Virtualbox) automatically after starting the GNS3 desktop client
• Integration of VMware WS/Virtualbox VM's into GNS3 as available appliances

The majority of issues seem to be:

• Desktop client unable to connect to GNS3 VM after being auto-started by the client
• Network connectivity issues between integrated VMware WS/Virtualbox VM's and other nodes within a project
• Network connectivity issues with Cloud/NAT (running on "local" GNS3 server) and the host OS and nodes within the project (regardless of which source they are running on)
• Network connectivity issues between nodes running on "local" server and GNS3 VM

GNS3 3.0 will remove the Windows GNS3 server as an option, however, development on 3.0 is still ongoing and a release date has not been determined.

Solution:
Stop using the "local" server, for anything. In fact do not even configure the desktop client to start/use the "local" server. When installing the GNS3 client or running the setup wizard, select "Remote" and enter the IP address of the GNS3 VM. Majority of problems are now solved.

Discussion:
Ok, so how do I run my Windows or some other VM that I run in VMware on the GNS3 VM? Easy, create a custom QEMU appliance in GNS3 and use the .vmdk drives, because GNS3 can use those. Ok, but what about UEFI booting? Once again, GNS3 provides a solution, the OVMF2016.fd (or something like that) is a file that can be added to a QEMU image to boot Tianocore "BIOS", which is a UEFI compatible loader. Finally, it looks like there is even a solution for TPM (I have not tested it though).

So how do I start the GNS3 VM? Open your virtualization software and start the VM. Not sure why that seems to be such an issue. I am sure that a batch file or PowerShell script could be written to start very easily.

This method works really well for the simple fact that you are removing the majority of back and forth interactions and complexity between Windows, the virtualization software and GNS3 server(s).

Discussion question:
For those Windows users out there, my question is this, what benefit does the "local" server provide you that you cannot live with out? If there was a viable alternative, would you try it?

I can think of one possible use.

You only run dynamics based images and have limited system resources, so you do not run the GNS3 VM. Fair enough, but then again, this particular use case eliminates 75% of the issues anyway as it only uses dynamics appliances. If you don't have the resources to run the GNS3 VM, you are most likely also not integrating VM's from VMware WS/Virtualbox either. It does still leave possible issues with NAT/Cloud host interactions.

[josephmhiggins]

Yep.
90% of the problems that user experience when using GNS3 on Windows comes down to one thing: the GNS3 "Local" server.

Discussion question:
For those Windows users out there, my question is this, what benefit does the "local" server provide you that you cannot live with out?

I have not run anything in LocalServer on Windows in over 5 years, maybe 7 years.

The local server is a nuisance.
I have absolute extreme confidence gns3 version 3.0 will be awesome and fix these problems that you describe.
I can not figure out anyway that the gns3 developers can NOT fix all these problems.
The gns3 developers would have to come up with a completely screw design to NOT fix all these problems.

Hold on to your hat.

My script is running in 0 second mode now.
0 second mode means 30 seconds after tcp connection is established, I stop the gns3 vm and stop the vmware process
This script will complete in 2 hours.
I ran the script in 30 second mode previously; it took 4 hours.
30 second mode means 30 seconds after tcp connection is established, I stop the gns3 vm and stop the vmware process

In this script run, I am not starting gns3.
Consequently, I am not waiting until the gns3 vm icon turns green.

I have a 4 hour csv file - which I can turn into a graph in 15 seconds.

My Start-Gns3.ps1 is very powerful and all encompassing; but it is initially created because of the atrocity which is Windows AppData.

My main problem is I am have not learned exceptions (how to handle errors) - it is hard to do that like the gns3 developers do it.
The gns3 developers do a beautiful job with gns3_server.log

And my coding skill is low, but getting better, so have never made a Class and I use Script Blocks.
Powershell Script Blocks are extremely ugly.

I do not think I want to "fix" my script until April.
The problem with my script is there is windows powershell 5.1 and there is powershell core 7.2 now AND
you can not learn powershell on Windows with Visual Studio Code. There is a problem with
visual studio code and Windows which seems to be associated with conpty or winpty.

My code, in my HUMBLE opinion, obliterates the gns3 developers code.
I love my code.

But my scriptblock $GetVmrunResultSB
is as ugly as it gets.

start-gns3vm.ps1 is called from start-mygns3vm.ps1 and start-mygns3vm.ps is building a csv file for how long it takes to start the gns3 vm. the number of loops is 240 because of how microsoft excel works.

I have no traditional comments yet in my start-gns3vm.ps1
because I just removed all the code in start-gns3vm.ps1 from start-gns3.ps1 yesterday, err last year.
Happy New Year!.

<#
#>
[CmdletBinding()]
[OutputType([System.Int32])]
param(
[Parameter(Mandatory=$true,
ValueFromPipeline=$false,
Position=0)]
[ValidateScript({
# Validate the -Type parameter passed is valid and vmrun exists.
$fpdir = 'C:\Program Files (x86)\VMware'
if ($_ -eq 'ws') {
$fp = $fpdir + '\VMware Workstation\vmrun.exe'
} else {
$fp = $fpdir + '\VMware Player\vmrun.exe'
}
if (Test-Path -Path $fp -PathType Leaf) {
$true
} else {
$msg = $fp + ' does not exist.'
Write-Verbose -Message $msg -Verbose
# TODO: does throw work in powershell 7???
throw $msg
}
})]
[ValidateSet('ws', 'player')]
[ValidateNotNullOrEmpty()]
[String]$Type
)

function Start-Gns3VM {
[CmdletBinding()]
[OutputType([System.Int32])]
param()

begin {
    $startgns3vmtimelimit = 420
    $tcpconngns3vmtimelimit = 300

    # Part 1: Initialize vmrun
    $vmwaredir = 'C:\Program Files (x86)\VMware'
    if ($Type -eq 'ws') {
        $vmrun = $vmwaredir + '\VMware Workstation\vmrun.exe'
    } else {
        $vmrun = $vmwaredir + '\VMware Player\vmrun.exe'
    }

    # vmxunq is an unquoted string
    $vmxunq = $HOME + '\Documents\Virtual Machines\GNS3 VM\GNS3 VM.vmx'
    # vmx must be put inside quotes because $vmxunq has spaces in it.
    $vmx        = "`"$vmxunq`""
    $vmrunstate = "-T " + $Type + " checkToolsState " + $vmx
    $vmrunstateargs = "-T","ws","checkToolsState",$vmx
    $vmrunstop = "-T " + $Type + " stop " + $vmx
    $vmrunstart = "-T " + $Type + " start " + $vmx
    $vmrunip    = "getGuestIPAddress " + $vmx + " -wait"


    # Initialize vmrun.exe parameters with a ordered dictionary
    $VmrunStartInfoParams = [ordered]@{
        TypeName    = 'System.Diagnostics.ProcessStartInfo'
        Property    =  [ordered]@{
            Filename                = $vmrun
            Arguments               = 'ToBeDeterminedInProcessBlock'
            RedirectStandardOutput  = $true
            RedirectStandardError   = $true
            UseShellExecute         = $false
            CreateNoWindow          = $true
        }
    }

    if ($DebugPreference -eq 'Continue') {
        $orddict = 'System.Collections.Specialized.OrderedDictionary'

        Write-Debug -Message $startgns3vmtimelimit
        Write-Debug -Message $vmrun
        Write-Debug -Message $vmx
        Write-Debug -Message $vmrunstate
        Write-Debug -Message $vmrunstart
        Write-Debug -Message $vmrunip
        foreach($key in $VmrunStartInfoParams.keys) {
            $val = $VmrunStartInfoParams[$key]
            if ($orddict -ne  $val.GetType()) {
                $msg = "VmrunStartInfoParams    $key    $val"
                Write-Debug -Message $msg
            } else {
                $preface = "VmrunStartInfoParams    $key    "
                $msg = $preface + $val
                Write-Debug -Message $msg
                foreach($subkey in $val.keys) {
                    $subval = $val[$subkey]
                    $submsg = $preface + $subkey.PadRight(28," ") + $subval
                    Write-Debug -Message $submsg
                }
            }
        }
    }

} #end begin

process {

    $StartTime = Get-Date
    $GetVmrunResultSB = {
        # A scriptblock to start vmrun.exe with the argument cmd
        param ($cmd)

        # replace the ToBeDeterminedInProcessBlock with $cmd
        $VmrunStartInfoParams.Property['Arguments'] = $cmd
        # create VmrunStartInfo object with the splatted hashtable
        $VmrunStartInfo = New-Object @VmrunStartInfoParams
        # create an empty process object
        $VmrunProcess = New-Object System.Diagnostics.Process
        # Apply the StartInfo after the VmrunProcess is created.
        $VmrunProcess.StartInfo = $VmrunStartInfo

        # Start vmrunprocess.
        try {
            $VmrunProcess.Start() | Out-Null
        } catch {
            Throw $_
        }
        $vmout = $VmrunProcess.StandardOutput
        $vmerr = $VmrunProcess.StandardError
        # 30000 is in milliseconds
        $VmrunProcess.WaitForExit(30000) | Out-Null

        # VMware appears to not write to StandardError
        $vmerrstr = $vmerr.ReadToEnd()
        if (($null -ne $vmerrstr) -and ($vmerrstr.Length -gt 0))  {
            $msg = ('Unknown error: ' + $vmerrstr)
            Write-verbose -Message $msg #-Verbose
            Throw $msg
        }

        $vmoutstr = $vmout.ReadToEnd()
        if (($null -eq $vmoutstr) -or ($vmoutstr.Length -lt 3)) {
            # Vmnrun -start the GNS3 VM does not write to StandardOutput
            $null
        } else {
            $result = $vmoutstr.Substring(0,$vmoutstr.Length - 2)
            $msg = "The argument: $cmd result is: $result"

            if ($result.Length -lt 6) {
                # maybe getguestipaddress could return .1 as an ip???
                Write-Verbose -Message $msg                    
                $result
            } elseif ('Error:' -eq $result.Substring(0,6)) {

                Write-Verbose -Message $msg #-Verbose
                throw $result
            } else {
                Write-Verbose -Message $msg
                $result
            }
        }
    } #end $GetVmrunResultSB


    $VmrunStartGns3VmSB = {
        # Start the GNS3 VM in this Script Block

        # Determine how long it takes to start the GNS3 VM
        $RunningTimeOut = New-TimeSpan -Seconds $startgns3vmtimelimit
        $StopWatch = [Diagnostics.Stopwatch]::StartNew()
        while ($StopWatch.Elapsed -lt $RunningTimeOut) {

            & $GetVmrunResultSB -cmd $vmrunstart | Out-Null

            $VmrunStartResult = & $GetVmrunResultSB -cmd $vmrunstate

            if ('running' -eq $VmrunStartResult) {
                $ElapsedTime = $StopWatch.Elapsed.TotalSeconds
                # 'n1' represents one decimal point
                $secs = $ElapsedTime.tostring('n1')
                $msg = "The GNS3 VM started in $secs seconds."
                Write-Verbose -Message $msg #-Verbose
                break
            } #end if
        } #end while

        # Check if the Timelimit was reached.
        if ('running' -ne $VmrunStartResult) {
            $msg = "vmrun.exe did not report the GNS3 VM as " +
                "running after $startgns3vmtimelimit seconds."
            Throw $msg
        } #end if
    } #end $VmrunStartGns3VmSB

    # Check the status of the GNS3 VM
    $VmrunResult = & $GetVmrunResultSB -cmd $vmrunstate

    switch ($VmrunResult) {
        'running' {
            $msg = 'The GNS3 VM is already running.'
            Write-Verbose -Message $msg # -Verbose
        } #end 'running'
        'installed' {
            Write-Verbose -Message 'Starting the GNS3 VM.' # -Verbose
            $msg = 'This can take ' +
            'a few minutes if the GNS3 VM was not shutdown properly.'
            Write-Verbose -Message $msg #-Verbose                
            & $VmrunStartGns3VmSB
        } #end 'installed'
        default {
            $msg = "Received a result of neither installed " +
                   "nor running from: $vmrunstate"
            Write-Verbose -Message $msg -Verbose
            $errmsg = 'Unknown error ' + $VmrunResult
            Throw $errmsg
        } #end 'default'
    }

    # The GNS3 VM has been started. Get the ip address of the GNS3 VM
    $Gns3VMIP = & $GetVmrunResultSB -cmd $vmrunip
    $msg = "The GNS3 VM ip address is $Gns3VMIP"
    Write-Verbose -Message $msg # -Verbose

    $Vmnet1Params = @{
        InterfaceAlias = 'VMware Network Adapter VMnet1'
        AddressFamily  = 'IPv4'
    }
    $Vmnet1 = Get-NetIPAddress @Vmnet1Params
    $Vmnet1Address = $Vmnet1.IPAddress
    $msg = "Attempting TCP Connect from $Vmnet1Address to " +
           "remote $Gns3VMIP and remote port 80."
    # FROM VMNET1 INTERFACE
    Write-Verbose -Message $msg # -Verbose

    # Put the TcpClient in a while loop with a maxtime of 300s
    $tcpresultparams = @{
        TypeName    = 'System.Net.Sockets.TcpClient'
        Argument    = $Gns3VMIP, 80
    }

    $TcpConnTimeOut = New-TimeSpan -Seconds $tcpconngns3vmtimelimit
    $StopWatch = [Diagnostics.Stopwatch]::StartNew()
    while ($StopWatch.Elapsed -lt $TcpConnTimeOut) {
        try {
            $tcpresult = New-Object @tcpresultparams
            # If New-Object threw no error, then it succeeded.
            break
        } catch {
            # New-Object can throw a terminating error, catch it.
            $attempts += 1
        } # end try catch
    } #end while

    # Unknown at this point if TcpClient Connected successfully
    # because TcpClient could have timed out after $Timeout, test it.
    if ((-not($tcpresult)) -or ($tcpresult.Connect -eq $false)) {
        $msg = "TCP timer expired after $TcpConnTimeOut seconds " +
               "and $attempts attempts."
        Write-Verbose -Message $msg # -Verbose
        throw $msg
    } else {
        $FinishTime = Get-Date
        $ElapsedTime = $FinishTime - $StartTime
        $Duration = $ElapsedTime.totalseconds.tostring('n1')
        $msg = "Total time to establish TCP connection " +
               "to the GNS3 VM was $Duration seconds."
        Write-Verbose -Message $msg # -Verbose
    }

    $TotalGns3VmStartTime = [math]::Round($ElapsedTime.totalseconds)
    $TotalGns3VmStartTime
} #end process

} #end function

$Gns3VmStartTime = Start-Gns3VM
$Gns3VmStartTime

Edit: oh yeah, and yesterday was the first time I removed a function from a script, so this script will not have a function.
That is a very confusing aspect of powershell - the .ps1 file is the "function" because all the powershell books want you to make modules and I do not know how to do that.

[josephmhiggins]

The anomalies in the graph could be caused by anything in Windows.
Primarily, I am focusing on when the GNS3 VM reaches a steady state - which I have arbitrarily defined as
/var/log/syslog
repeating over and over again:
Dec 30 10:38:48 gns3vm multipathd[591]: sdb: add missing path
Dec 30 10:38:48 gns3vm multipathd[591]: sdb: failed to get udev uid: Invalid argument
Dec 30 10:38:48 gns3vm multipathd[591]: sdb: failed to get sysfs uid: Invalid argument
Dec 30 10:38:48 gns3vm multipathd[591]: sdb: failed to get sgio uid: No such file or directory

Edit: AND btw, I like a safety factor of 3-4. I read the python code and the code is 120 seconds before it throws an error - a possible FALSE error. A possible false is EXTREMELY damaging to new gns3 users because they go in an change stuff; that should be expected from new gns3 users. My code is looking for a factor of 4 to protectnewgns3 users.

[josephmhiggins]

I see no rational reason why the gns3 developers should associate with powershell.
i have the utmost confidence that the gns3 developers, in gns3 v3.0, will guarantee that the gns3vm on windows is at a state of
"Ready"
I do not know what a state of "Ready" is. It is up to the gns3 developers to figure that out on their own.

[grossmj]

@eantowne thanks for this post, totally agree 👍

[josephmhiggins]

New windows gns3 users can not handle troubleshooting the local server option because they do not know if it is windows, the gui, or the server. They go to the internet and start causing trouble by turning off their firewall or try to run gns3 in admin mode or maybe start messing with their vmware vmnets.

My start-gns3.ps1 starts vmware first, the gns3 vm, gns3_server.exe and then the gns3.exe "after the gns3 vm is green".
There is no signal "initialization complete signal" from the gns3 vm or gns3_server.exe and I am not filtering wireshark captures to determine when the gns3 vm and gns3_server.exe are fully booted - way too much work.
I just sleep powershell for 5 seconds after gns3_server.exe starts; that is good enough for me.

The remote server option for windows, to the best of my recollection, prevents vmware vms to work - but I do not support them anyway.

[josephmhiggins]

Two major weaknesses in the gns3 code:

1. the disable/enable vmware bug wherein gns3 users have to disable and then enable their vmnets via Windows to reinitialize the vmnets. There was a vmware KB about this problem; I read it. This appears to be confined to somewhere around vmware 15.5. If it is even possible for me to investigate what causes this, it might take me 6 months to learn how to do this. However, the problem could be internal to vmware software so that I might not be able to detect the failure in the .NET framework.
2. errr, I forgot while I was writing this post...

My start-gns3.ps1 is a vastly superior design because I have all my log files going to syslog at their creation.
I know how long everything takes to boot and I can graph everything.

There is some kind of problem with windows powershell in that it requires to be started in admin mode to put gns3_server.exe in debug mode, but visual studio code 7.2 can do it effortlessly. But windows powershell is no longer maintained and there is seemingly universal agreement that it should not be used - which is very confusing to me.

2. I remember. The vm ip address of the GNS3 VM is drawn in the vmware rest api after the vm is started. I would have to look back at vmware_vm.py, but the vmrest api has some kind of logic to say what the ip address of the vm is. What gns3 users appear to be doing is changing vmnet1 from host-only to some other vmnet type. This incorrect setting should be able to detected in Cmd on Windows with some combination of vmrun.exe

Edit: The gns3 developer message after 120 seconds of "check your firewall settings" is devastating to gns3 windows users and is outdated by 5-7 years. And 120 seconds offers no safety factor because it is normal for the gns3 vm to take longer than 60 seconds. My code changes the timeout to 300 seconds.

Until the web-ui is fully formed, the wireshark problem with the web-ui is crushing. Wireshark is mission critical and the GUI is still superior for this reason and this reason alone.

[josephmhiggins]

Oh, the "initialization complete signal" is highly desirable.
But...I have to wait for gns3 3.0
I would be able to very easily detect when a new file named:
gns3server_gns3vm.transaction
or something similar is created inside
%appdata%\gns3

It is vastly superior methodology not to try to load a project until the gns3_server.exe and gns3 vm are known to be communicating; the icon for the gns3 vm is green in the gui.

[josephmhiggins]

the gns3 developers have to come to the realization that there design of starting the gns3 client before the gns3 server/gns3 vm is an incorrect design

there is no way around that.

Ean's point of "For those Windows users out there, my question is this,"...new windows gns3 users can not handle that question.

Linux users have KVM, their hypervisor, up and running - normally - already before they start gns3.
Windows users that use VMware do not, except for me, have their hypervisor up and running before they start gns3.
I do not like the idea of shifting the blame, or the onus of responsibility of new Windows Gns3 users, on Windows Gns3 users.
I blame the gns3 developers for their convoluted design.

The best analogy I came up with in the last 3 weeks is that if I spend an hour driving to a store and I find the store is closed, then I am not going to break into the store - which is what the gns3 users are metaphorically trying to do.

With gns3, if the gns3 client and the gns3 vm/gns3 server are not communicating, then

1. I am not turning off my Windows firewall
2. I am not trying to restart my gns3 in admin mode
3. I am not going to "re-ip" my home network (change my vmnet settings)

The hypervisor must be started first.
The server must be started second.
The gui, or web-ui, must be started last.

This is a gns3 developer problem and not a gns3 windows user problem.

[eantowne]

the gns3 developers have to come to the realization that there design of starting the gns3 client before the gns3 server/gns3 vm is an incorrect design

To a certain extent, I agree, kind of. In it's current state, yes, it does cause issues. However, if the Client had all "normal operations" menu items and such disabled until the connection to the Controller was established and the Controller validates that all Computes are connected, then that would solve the particular issue you are getting at.

"For those Windows users out there, my question is this,"...new windows gns3 users can not handle that question.

Not really sure that "New" Windows, or any other operating system, users are really the demographic here on this GitHub repository.

Along with that, previously you stated (Dec 31st, as the first response to me):

I have not run anything in Local Server on Windows in over 5 years, maybe 7 years.

So, would this be your answer, "There is no valid use case for the "local" server on Windows?

The hypervisor must be started first.
The server must be started second.
The gui, or web-ui, must be started last.

I honestly am not sure that I agree with this sequence. I understand the logic of it, however my first comment points to a direction that could make swapping the first two viable.

I blame the gns3 developers for their convoluted design.

Nope, sorry, I cannot get behind you on this one. Incoming Rant....

Look at the history of GNS3 and what it originally supported and the environment it ran in. GNS3 has been evolving to accommodate new technologies in the appliances, the virtualization of them, as well as the underlying Operating Systems. While there are improvements that can be made, I am NOT going to lay it at the feet of the Dev Team and walk away.

You want to know where the blame starts? Head out to Redmond, WA and ask the folks in MS why their operating system is so unbelievably dysfunctional. Ask the folks and Apple why they sandbox and break everything, or why they have not turned on the nested virtualization in their Operating System for the CPU's they make that have the capability baked into them?

Nothing is stopping you from switching to Linux...

[josephmhiggins]

i recommend this thread be closed

[eantowne]

Out of curiosity, why? You made 4 responses to a post that had been dead for 3 months after Jeremy provided a 1 line response. I then responded to your four additional responses, and now you want the thread closed?

The entire purpose here is to generate a discussion. Just because I do not necessarily agree with everything you say, does not mean the conversation needs to end. Your opinions are just as valid as mine, we just have different perspectives. It is up to Jeremy to decide what to do with the information generated from the discussion. The fact that you and I do not agree, is actually a good thing, it provides Jeremy with different perspectives.

My comment about Linux was my standard jab at Windows and nothing towards you. I like Mac, but for anything beyond basic user computing, it has serious issues.