go-client fails on reading large responses

[yuliy]

I try running this simple code:

package main

import "fmt"
import "github.com/manticoresoftware/go-sdk/manticore"

func main() {
    cl := manticore.NewClient()
    cl.SetServer("127.0.0.1", 3003)
    res, err := cl.Open()
    fmt.Println("Open result: ", res, err)

    r3, e3 := cl.Sphinxql(`SELECT ...some query...;`)
    fmt.Println("Query result: ", r3, e3)
}

It works fine on small responses. But when response is pretty large (say 100K+) the client returns error:

failed to read searchd response (status=0, ver=256, len=112460, read=24608)

If I reduce response size (by lowering the LIMIT or number of fields returned) it works.

Via Wireshark I see that manticore returns whole response. But client interrupts the connection.

I looked through the client code and found out that reading from socket is implemented with single call to Conn.Read(). It returns data read. If response is large it reads part of the response and returns size of that part. Further in code there is a comparison of data expected and data read. They're not equal. Hence, client returns error. link to code

Googling over the internet I found that it's not the right way to read from socket. E.g. look through this article.

I tried to implement reading by chunks and it worked:

diff --git a/manticore/client.go b/manticore/client.go
index bf9f3e6..315579e 100644
--- a/manticore/client.go
+++ b/manticore/client.go
@@ -135,6 +135,34 @@ func (cl *Client) connect() error {
 	return cl.failclose(err)
 }
 
+/// read raw answer (with fixed size) to buf
+func (cl *Client) readRawAnswer(buf []byte, size int) (int, error) {
+	const MAX_CHUNK_SIZE = 16 * 1024
+	nbytes := 0
+	for {
+		chunkSize := MAX_CHUNK_SIZE
+		bytesRemaining := size - nbytes
+		if bytesRemaining < chunkSize {
+			chunkSize = bytesRemaining
+		}
+		n, e := cl.conn.Read(buf[nbytes:nbytes+chunkSize])
+		if e != nil {
+			return n, e
+		}
+		nbytes += n
+		if (nbytes < size) {
+			continue
+		}
+		break
+	}
+
+	if (nbytes > size) {
+		return nbytes, errors.New("Logical error in Client.read()!")
+	}
+
+	return nbytes, nil
+}
+
 /// get and check response packet from searchd server
 func (cl *Client) getResponse(client_ver uCommandVersion) (apibuf, error) {
 	rawrecv := cl.getByteBuf(8)
@@ -151,7 +179,7 @@ func (cl *Client) getResponse(client_ver uCommandVersion) (apibuf, error) {
 	iReplySize := rawrecv.getInt()
 
 	rawanswer := cl.getByteBuf(iReplySize)
-	nbytes, err = cl.conn.Read(*rawanswer)
+	nbytes, err = cl.readRawAnswer(*rawanswer, iReplySize)
 	if err != nil {
 		return nil, err
 	}

I'm not sure that it is correct go-idiomatic solution because I'm not a go-developer. But I hope it helps to make a proper fix.

[sanikolaev]

Thank you. Please create a pull request.