Scaling not working, external scaler throws exception

[rohannevrikar]

I want to use KEDA for durable functions. I have been following Getting Started doc of this repo to test scaling of durable functions, but can't get to make it work. I have a simple durable function orchestration like this one. Also tried with QueueTriggered orchestration (actually this is what I to implement in my project), but doesn't work for that too.

Steps which I followed:

1. Deployed KEDA to keda namespace.
2. Deployed secret (create_secrets.sh) and external scaler to keda namespace.
3. Deployed the function app to default namespace along with secrets converted to base64.
4. Pods are running fine. I can start the orchestration by hitting the URL (http://localhost/api/Function1_HttpStart).
5. Started load testing. Number of pods didn't scale up as expected.

Logs of events and external scaler pod:

kubectl get events

FailedGetExternalMetric horizontalpodautoscaler/keda-hpa-durable-keda unable to get external metric default/ScaleRecommendation/&LabelSelector{MatchLabels:map[string]string{deploymentName: durable-keda,},MatchExpressions:[],}: unable to fetch metrics from external metrics API: No matching metrics found for scalerecommendation FailedComputeMetricsReplicas horizontalpodautoscaler/keda-hpa-durable-keda invalid metrics (1 invalid out of 1), first error is: failed to get ScaleRecommendation external metric: unable to get external metric default/ScaleRecommendation/&LabelSelector{MatchLabels:map[string]string{deploymentName: durable-keda,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: No matching metrics found for scalerecommendation

kubectl logs keda-durable-external-scaler-86c6f85bcc-s77qz -n keda -f

Request starting HTTP/2 POST http://durable-external-scaler-service.keda.svc.cluster.local:5000/externalscaler.ExternalScaler/GetMetrics application/grpc info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0] Executing endpoint 'gRPC - /externalscaler.ExternalScaler/GetMetrics' dbug: Grpc.AspNetCore.Server.ServerCallHandler[1] Reading message. info: Keda.Durable.Scaler.Server.Services.ExternalScalerService[0] Namespace: default DeploymentName: durable-keda GetMetrics() called. fail: Grpc.AspNetCore.Server.ServerCallHandler[3] Error when executing service method 'GetMetrics'. System.NullReferenceException: Object reference not set to an instance of an object. at Keda.Durable.Scaler.Server.Services.ExternalScalerService.GetMetrics(GetMetricsRequest request, ServerCallContext context) in /src/src/Keda.Durable.Scaler.Server/Services/ExternalScalerService.cs:line 112 at Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext) at Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext) at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase3.<HandleCallAsync>g__AwaitHandleCall|17_0(HttpContextServerCallContext serverCallContext, Method2 method, Task handleCall)

Specifications: Docker Desktop for Windows (2.4.0.0), Kubernetes (1.18.8)

Please let me know if more information is needed, I'd be happy to dig out as much as I can. I'm still new to Kubernetes so not sure whether this is a bug or I'm just missing something. Thanks!

[tomkerkhove]

Which version of KEDA are you running?

[rohannevrikar]

I'm using 2.0 stable version of KEDA, which I installed using helm: helm install kedacore/keda --namespace keda --name keda.
I also tried using the release candidate version helm install keda kedacore/keda --version 2.0.0-rc --namespace keda, but it shows this error while deploying the function app:
error: unable to recognize "deployone.yml": no matches for kind "ScaledObject" in version "keda.k8s.io/v1alpha1"
Deployment and service are created, but fails for ScaledObject.

[tomkerkhove]

We have changed the name of the scaled object type, you can find it in our migration docs.

However, I think this scaler has not ben migrated to 2.0 as well, right @TsuyoshiUshio ?

Function tooling is being updated to support 2.0 and should land soon.

[rohannevrikar]

Thank you for your guidance @tomkerkhove, I changed values of scaled object after referring to migration docs, and I'm able to run 2.0.0-rc version. However, I'm getting the same error.

Will wait for @TsuyoshiUshio 's inputs regarding migrating the scaler to 2.0. Please let me know if I could be of any help. Thanks!

[TsuyoshiUshio]

@rohannevrikar
Sorry for being late. Yes. We need to migrate to 2.0. I need to work on it.

[samfisher07]

Hi.
Has there been any movement on this issue?
I'm experiencing the same errors as @rohannevrikar after following the steps in the guide for this external scaler.
When inspecting the logs from the durable scaler pod I see this exception repeated:

fail: Grpc.AspNetCore.Server.ServerCallHandler[3] Error when executing service method 'GetMetrics'. System.NullReferenceException: Object reference not set to an instance of an object. at Keda.Durable.Scaler.Server.Services.ExternalScalerService.GetMetrics(GetMetricsRequest request, ServerCallContext context) in /src/src/Keda.Durable.Scaler.Server/Services/ExternalScalerService.cs:line 112 at Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext) at Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext) at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase3.<HandleCallAsync>g__AwaitHandleCall|17_0(HttpContextServerCallContext serverCallContext, Method2 method, Task handleCall)

Also, when running kubectl get events I see this warning repeated:

Warning FailedGetExternalMetric horizontalpodautoscaler/keda-hpa-durable-keda unable to get external metric default/ScaleRecommendation/&LabelSelector{MatchLabels:map[string]string{scaledObjectName: durable-keda,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: No matching metrics found for scalerecommendation 56m Warning FailedComputeMetricsReplicas horizontalpodautoscaler/keda-hpa-durable-keda invalid metrics (1 invalid out of 1), first error is: failed to get ScaleRecommendation external metric: unable to get external metric default/ScaleRecommendation/&LabelSelector{MatchLabels:map[string]string{scaledObjectName: durable-keda,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: No matching metrics found for scalerecommendation

I am using KEDA v2.2.0:
kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.2.0/keda-2.2.0.yaml
I've used this documentation (https://keda.sh/docs/2.0/migration/) for the scaled object.

Like @rohannevrikar, my pod is running fine but there is no scaling.

Any help would be greatly appreciated. Thanks!

[TsuyoshiUshio]

Hi @samfisher07
Sorry for the inconvenience. I'd like to introduce, one workaround.

You can consider SQL backend for the Durable. @cgillum develops SQL backend and keda support it. SQL Provider is developed for k8s usage since on-prem scenario sometimes, they can't access to the Storage Account. If you are interested, please have a look!

https://techcommunity.microsoft.com/t5/apps-on-azure/new-storage-providers-for-azure-durable-functions/ba-p/2382044
MSSQL scaler
https://keda.sh/docs/2.2/scalers/mssql/

Durable Task SQL Provider
https://microsoft.github.io/durabletask-mssql/#/taskhubs

[samfisher07]

Hi @TsuyoshiUshio,

Thanks for the workaround, it works very well.

Not sure if this belongs here, but I had to modify the "GetScaleRecommendation" query to return a minimum value of 1 for the mssql scaler to work correctly with durable functions.

Cheers,
Sam

[cgillum]

@samfisher07 I’m glad to hear the SQL provider worked well for you!

Would you be willing to share more details of your feedback on dt.GetScaleRecommendation here? https://github.com/microsoft/durabletask-mssql/issues?