grpc_max_message_size.md

GRPC Max Message Size

GRPC has a default max message size of 4MB. If you need to run models whose input features or output predictions are larger than this you can congifure Seldon Core to run with gRPC server/clients that handle a larger mesage size with annotations.

Add the annotation seldon.io/grpc-max-message-size with the number of bytes of the largest expected message. For example the SeldonDeployment resource below sets this to 10MB:

{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "seldon-deployment-example"
    },
    "spec": {
        "annotations": {
            "project_name": "FX Market Prediction",
            "deployment_version": "v1",
	    "seldon.io/grpc-max-message-size":"10485760"
        },
        "name": "test-deployment",
        "oauth_key": "oauth-key",
        "oauth_secret": "oauth-secret",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier_grpc:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 20
                    }
                }],
                "graph": {
                    "children": [],
                    "name": "classifier",
                    "endpoint": {
			"type" : "GRPC"
		    },
                    "type": "MODEL"
                },
                "name": "fx-market-predictor",
                "replicas": 1,
		"annotations": {
		    "predictor_version" : "v1"
		}
            }
        ]
    }
}

API OAUTH Gateway

If you are using the default API OAUTH Gateway you will also need to update your Helm or Ksonnet install:

For Helm add to your Helm values, for example:

apife:
  annotations:
      seldon.io/grpc-max-message-size: "10485760"

For Ksonnet set the parameters grpcMaxMessageSize:

ks param set seldon-core grpcMaxMessageSize '10485760' --as-string

Example

To see a worked example, run the Jupyter notebook here.