"sfctl cluster upgrade" fails with (FABRIC_E_FABRIC_ALREADY_IN_TARGET_VERSION) Fabric is already in this version

[mrkarMSFT]

sfctl Version:

Name: sfctl
Version: 9.0.0

Issue:

In an azure hosted SF cluster running 7.0.470.9590 version, while trying to run the below command it is failing with "(FABRIC_E_FABRIC_ALREADY_IN_TARGET_VERSION) Fabric is already in this version" error.

---

Command ran:

sfctl cluster upgrade --code-version 7.0.470.9590 --unhealthy-nodes 50 --rolling-upgrade-mode Monitored

Command arguments: ['cluster', 'upgrade', '--code-version', '7.0.470.9590', '--unhealthy-nodes', '50', '--rolling-upgrade-mode', 'Monitored', '--debug']
Event: Cli.PreExecute []
Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fb2b2d70048>, <function OutputProducer.on_global_arguments at 0x7fb2b28baa60>, <function CLIQuery.on_global_arguments at 0x7fb2b28d8b70>]
Event: CommandInvoker.OnPreCommandTableCreate []
Event: CommandLoader.OnLoadArguments []
Event: CommandInvoker.OnPostCommandTableCreate []
Event: CommandInvoker.OnCommandTableLoaded []
Event: CommandInvoker.OnPreParseArgs []
Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fb2b28baae8>, <function CLIQuery.handle_query_parameter at 0x7fb2b28d8bf8>]
msrest.universal_http.requests : Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90
msrest.service_client : Accept header absent and forced to application/json
msrest.universal_http : Configuring redirects: allow=True, max=30
msrest.universal_http : Configuring request: timeout=100, verify=True, cert=None
msrest.universal_http : Configuring proxies: ''
msrest.universal_http : Evaluate proxies against ENV settings: True
urllib3.connectionpool : Starting new HTTPS connection (1): mrkarcluster.westus.cloudapp.azure.com:19080
/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'mrkarcluster.westus.cloudapp.azure.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
urllib3.connectionpool : https://mrkarcluster.westus.cloudapp.azure.com:19080 "POST /$/Upgrade?timeout=60&api-version=6.0 HTTP/1.1" 409 108
msrest.exceptions : (FABRIC_E_FABRIC_ALREADY_IN_TARGET_VERSION) Fabric is already in this version
(FABRIC_E_FABRIC_ALREADY_IN_TARGET_VERSION) Fabric is already in this version
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/knack/cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/local/lib/python3.5/dist-packages/sfctl/entry.py", line 81, in execute
return super(SFInvoker, self).execute(args)
File "/usr/local/lib/python3.5/dist-packages/knack/invocation.py", line 208, in execute
cmd_result = parsed_args.func(params)
File "/usr/local/lib/python3.5/dist-packages/knack/commands.py", line 139, in call
return self.handler(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/knack/commands.py", line 246, in _command_handler
result = op(client, **command_args) if client else op(**command_args)
File "/usr/local/lib/python3.5/dist-packages/sfctl/custom_cluster_upgrade.py", line 132, in upgrade
client.start_cluster_upgrade(upgrade_desc, timeout=timeout)
File "/usr/local/lib/python3.5/dist-packages/azure/servicefabric/operations/_service_fabric_client_ap_is_operations.py", line 1417, in start_cluster_upgrade
raise models.FabricErrorException(self._deserialize, response)
azure.servicefabric.models._models_py3.FabricErrorException: (FABRIC_E_FABRIC_ALREADY_IN_TARGET_VERSION) Fabric is already in this version
msrest.universal_http.requests : Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90
Performing cluster version check
msrest.universal_http : Configuring redirects: allow=True, max=30
msrest.universal_http : Configuring request: timeout=100, verify=True, cert=None
msrest.universal_http : Configuring proxies: ''
msrest.universal_http : Evaluate proxies against ENV settings: True
urllib3.connectionpool : Starting new HTTPS connection (1): mrkarcluster.westus.cloudapp.azure.com:19080
/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'mrkarcluster.westus.cloudapp.azure.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
urllib3.connectionpool : https://mrkarcluster.westus.cloudapp.azure.com:19080 "GET /$/GetClusterVersion?timeout=60&api-version=6.4 HTTP/1.1" 200 26
Check cluster version failed due to error: <class 'sfctl.custom_exceptions.SFCTLInternalException'>

If I change the code version to something else, then the command works. Without the code version, the command fails with,

Command arguments: ['cluster', 'upgrade', '--unhealthy-nodes', '50', '--rolling-upgrade-mode', 'Monitored', '--debug']
Event: Cli.PreExecute []
Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f9e235b8048>, <function OutputProducer.on_global_arguments at 0x7f9e23102a60>, <function CLIQuery.on_global_arguments at 0x7f9e23120b70>]
Event: CommandInvoker.OnPreCommandTableCreate []
Event: CommandLoader.OnLoadArguments []
Event: CommandInvoker.OnPostCommandTableCreate []
Event: CommandInvoker.OnCommandTableLoaded []
Event: CommandInvoker.OnPreParseArgs []
Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f9e23102ae8>, <function CLIQuery.handle_query_parameter at 0x7f9e23120bf8>]
msrest.universal_http.requests : Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90
msrest.service_client : Accept header absent and forced to application/json
msrest.universal_http : Configuring redirects: allow=True, max=30
msrest.universal_http : Configuring request: timeout=100, verify=True, cert=None
msrest.universal_http : Configuring proxies: ''
msrest.universal_http : Evaluate proxies against ENV settings: True
urllib3.connectionpool : Starting new HTTPS connection (1): mrkarcluster.westus.cloudapp.azure.com:19080
/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'mrkarcluster.westus.cloudapp.azure.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
urllib3.connectionpool : https://mrkarcluster.westus.cloudapp.azure.com:19080 "POST /$/Upgrade?api-version=6.0&timeout=60 HTTP/1.1" 400 148
msrest.exceptions : (E_INVALIDARG) Must specify either a valid target code or config version for Fabric upgrade. Requested value: ('','')
(E_INVALIDARG) Must specify either a valid target code or config version for Fabric upgrade. Requested value: ('','')
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/knack/cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/local/lib/python3.5/dist-packages/sfctl/entry.py", line 81, in execute
return super(SFInvoker, self).execute(args)
File "/usr/local/lib/python3.5/dist-packages/knack/invocation.py", line 208, in execute
cmd_result = parsed_args.func(params)
File "/usr/local/lib/python3.5/dist-packages/knack/commands.py", line 139, in call
return self.handler(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/knack/commands.py", line 246, in _command_handler
result = op(client, **command_args) if client else op(**command_args)
File "/usr/local/lib/python3.5/dist-packages/sfctl/custom_cluster_upgrade.py", line 132, in upgrade
client.start_cluster_upgrade(upgrade_desc, timeout=timeout)
File "/usr/local/lib/python3.5/dist-packages/azure/servicefabric/operations/_service_fabric_client_ap_is_operations.py", line 1417, in start_cluster_upgrade
raise models.FabricErrorException(self._deserialize, response)
azure.servicefabric.models._models_py3.FabricErrorException: (E_INVALIDARG) Must specify either a valid target code or config version for Fabric upgrade. Requested value: ('','')
msrest.universal_http.requests : Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90
Performing cluster version check
msrest.universal_http : Configuring redirects: allow=True, max=30
msrest.universal_http : Configuring request: timeout=100, verify=True, cert=None
msrest.universal_http : Configuring proxies: ''
msrest.universal_http : Evaluate proxies against ENV settings: True
urllib3.connectionpool : Starting new HTTPS connection (1): mrkarcluster.westus.cloudapp.azure.com:19080
/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'mrkarcluster.westus.cloudapp.azure.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
urllib3.connectionpool : https://mrkarcluster.westus.cloudapp.azure.com:19080 "GET /$/GetClusterVersion?api-version=6.4&timeout=60 HTTP/1.1" 200 26
Check cluster version failed due to error: <class 'sfctl.custom_exceptions.SFCTLInternalException'>

This seems like a bug as to why do we have to pass in a different code version for the command to succeed?

[jeffj6123]

When running the command, what was the intent?
It looks like from the command you were attempting to upgrade to the version the cluster currently is?
The error returned says FABRIC_E_FABRIC_ALREADY_IN_TARGET_VERSION, which would be the expected behavior.

[mrkarMSFT]

@jeffj6123 The intent is to set "--unhealthy-nodes" value. As you can see above, without the code version, it doesn't allow the update. - "(E_INVALIDARG) Must specify either a valid target code or config version for Fabric upgrade. Requested value: ('','')"

Upon providing the code version, another issue comes up.

[jeffj6123]

Is this for a cluster in azure or on prem? Because there are a few steps that need to be taken to do a full upgrade through the CLI and depending on where the cluster is, its an easier process.

[mrkarMSFT]

@jeffj6123 it is azure hosted.

[jeffj6123]

@mrkarMSFT for a better and simpler experience you should follow this guide.
https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-config-upgrade-azure

[mrkarMSFT]

@jeffj6123 this works. But I want to use sfctl on azure hosted clusters. Is it not possible? Do I have to use ARM template or PowerShell? We have customers looking for sfctl. Please share the steps on how to update using sfctl.

[jeffj6123]

Hey, @mrkarMSFT so I did check about azure hosted clusters and you should always perform these upgrades using arm or something that goes through arm. There is another CLI option that will allow you to do this easily https://docs.microsoft.com/en-us/cli/azure/sf/cluster/setting?view=azure-cli-latest#az-sf-cluster-setting-set .