MINOR: Add protocol documentation about ApiVersions Fallback behaviour (

#17967)

Documents current ApiVersions fallback mechanism

The broker behaviour when it receives an ApiVersionsRequest ahead of it's supported versions is useful to know as an implementor/manipulator of the protocol.

The behaviour contradicts a point documented in the protocol guide. In the fallback case the protocol version of the response is different from the protocol version of the request.

Also spotted a couple of dead links in the table of contents

Signed-off-by: Robert Young <[email protected]>

Reviewers: Luke Chen <[email protected]>

docs/protocol.html

@@ -45,11 +45,6 @@ <h1>Kafka protocol guide</h1>
             <li><a href="#protocol_recordbatch">Record Batch</a>
         </ul>
     </li>
-    <li><a href="#protocol_evolution">Evolving the Protocol</a>
-        <ul>
-            <li><a href="#protocol_versioning">The Request Header</a>
-            <li><a href="#protocol_versioning">Versioning</a>
-        </ul>
     <li><a href="#protocol_constants">Constants</a>
         <ul>
             <li><a href="#protocol_error_codes">Error Codes</a>
@@ -125,7 +120,7 @@ <h5 class="anchor-heading"><a id="protocol_compatibility" class="anchor-link"></
 
 <p>The intention is that clients will support a range of API versions. When communicating with a particular broker, a given client should use the highest API version supported by both and indicate this version in their requests.</p>
 
-<p>The server will reject requests with a version it does not support, and will always respond to the client with exactly the protocol format it expects based on the version it included in its request. The intended upgrade path is that new features would first be rolled out on the server (with the older clients not making use of them) and then as newer clients are deployed these new features would gradually be taken advantage of.</p>
+<p>The server will reject requests with a version it does not support, and will always respond to the client with exactly the protocol format it expects based on the version it included in its request. The intended upgrade path is that new features would first be rolled out on the server (with the older clients not making use of them) and then as newer clients are deployed these new features would gradually be taken advantage of. Note there is an exceptional case while <a href="#api_versions">retrieving supported API versions</a> where the server can respond with a different version.</p>
 
 <p>Note that <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields">KIP-482 tagged fields</a> can be added to a request without incrementing the version number.  This offers an additional way of evolving the message schema without breaking compatibility.  Tagged fields do not take up any space when the field is not set.  Therefore, if a field is rarely used, it is more efficient to make it a tagged field than to put it in the mandatory schema.  However, tagged fields are ignored by recipients that don't know about them, which could pose a challenge if this is not the behavior that the sender wants.  In such cases, a version bump may be more appropriate.
 
@@ -143,7 +138,13 @@ <h5 class="anchor-heading"><a id="api_versions" class="anchor-link"></a><a href=
         Kafka protocol requests may take place on an SSL listener before the SSL handshake is finished). If this is considered to
         leak information about the broker version a workaround is to use SSL with client authentication which is performed at an
         earlier stage of the connection where the <code>ApiVersionRequest</code> is not available. Also, note that broker versions older
-        than 0.10.0.0 do not support this API and will either ignore the request or close connection in response to the request.</li>
+        than 0.10.0.0 do not support this API and will either ignore the request or close connection in response to the request. Also
+        note that if the client <code>ApiVersionsRequest</code> version is unsupported by the broker (client is ahead), and the broker
+        version is 2.4.0 or greater, then the broker will respond with a version 0 <a href="#The_Messages_ApiVersions">ApiVersionsResponse</a>
+        with the <a href="#protocol_error_codes">error code</a> set to <code>UNSUPPORTED_VERSION</code> and the <code>api_versions</code>
+        field populated with the supported version of the <code>ApiVersionsRequest</code>. It is then up to the client to retry, making
+        another <code>ApiVersionsRequest</code> using the highest version supported by the client and broker.
+        See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-511%3A+Collect+and+Expose+Client%27s+Name+and+Version+in+the+Brokers">KIP-511: Collect and Expose Client's Name and Version in the Brokers</a></li>
     <li>If multiple versions of an API are supported by broker and client, clients are recommended to use the latest version supported
         by the broker and itself.</li>
     <li>Deprecation of a protocol version is done by marking an API version as deprecated in the protocol documentation.</li>