Skip to content

Commit

Permalink
support/historyarchive: Fix error reporting (#1648)
Browse files Browse the repository at this point in the history
After deploying `release-horizon-v0.20.1` branch to staging server it
stopped processing ledger entries after around 300k entries with an
error:
```
Read wrong number of bytes from XDR
```
returned from `XdrStream.ReadOne`. First, I changed the code to return
`err` values before proceeding to the next checks. This reveal the real
error:
```
stream error: stream ID 311; PROTOCOL_ERROR
```
I couldn't find any information online about this but I suspect this is
happening because of the idle timeout. In `SingleLedgerStateReader` we
preload entries for a better performance (smaller round-trip time) when
getting temporary data from `TempSet`. When entries are being processed
the connection stays idle. The second fix (actually: quickfix) was
decreasing `preloadedEntries` to `20000` what makes a delay between
streaming batches shorter. However it's not a permanent fix. The issue
is tracked in #1649.
  • Loading branch information
bartekn authored Aug 26, 2019
1 parent b28bcc9 commit 47ad3b5
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 4 deletions.
2 changes: 1 addition & 1 deletion exp/ingest/io/single_ledger_state_reader.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ type TempSet interface {
// preloadedEntries defines a number of bucket entries to preload from a
// bucket in a single run. This is done to allow preloading keys from
// temp set.
const preloadedEntries = 50000
const preloadedEntries = 20000

// MakeSingleLedgerStateReader is a factory method for SingleLedgerStateReader
func MakeSingleLedgerStateReader(
Expand Down
10 changes: 7 additions & 3 deletions support/historyarchive/xdrstream.go
Original file line number Diff line number Diff line change
Expand Up @@ -123,21 +123,25 @@ func (x *XdrStream) ReadOne(in interface{}) error {
}
x.buf.Grow(int(nbytes))
read, err := x.buf.ReadFrom(io.LimitReader(x.rdr, int64(nbytes)))
if err != nil {
x.rdr.Close()
return err
}
if read != int64(nbytes) {
x.rdr.Close()
return errors.New("Read wrong number of bytes from XDR")
}

readi, err := xdr.Unmarshal(&x.buf, in)
if err != nil {
x.rdr.Close()
return err
}

readi, err := xdr.Unmarshal(&x.buf, in)
if int64(readi) != int64(nbytes) {
return fmt.Errorf("Unmarshalled %d bytes from XDR, expected %d)",
readi, nbytes)
}
return err
return nil
}

func WriteFramedXdr(out io.Writer, in interface{}) error {
Expand Down

0 comments on commit 47ad3b5

Please sign in to comment.