panic in test: Unclosed iterator at time of Txn.Discard

[tgross]

Received a one-off panic during one of the unit tests of the store package. This doesn't look like a vault issue at first glance and it looks like a pretty rare failure (I've only see it once after running these tests a few hundred times by now). I'm opening this now so that I don't misplace the stack trace info and because I don't want to rabbit-hole on this right now. Will come back to it sometime next week most likely.

Stack trace:

panic: runtime error: invalid memory address or nil pointer dereference
        panic: Unclosed iterator at time of Txn.Discard.
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x7ea55f]

goroutine 185 [running]:
github.com/dgraph-io/badger/v2.(*Txn).Discard(0xc00aaa4750)
        /home/tim/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/txn.go:517 +0xc5
panic(0xa26c60, 0x1052d10)
        /usr/local/go/src/runtime/panic.go:969 +0x175
github.com/dgraph-io/badger/v2/skl.(*Skiplist).IncrRef(...)
        /home/tim/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/skl/skl.go:86
github.com/dgraph-io/badger/v2.(*DB).getMemTables(0xc00a982800, 0x0, 0x0, 0x0, 0x0)
        /home/tim/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:629 +0xdf
github.com/dgraph-io/badger/v2.(*Txn).NewIterator(0xc00aaa4750, 0x1, 0xa, 0x0, 0xc00aa340c0, 0x21, 0x21, 0x0, 0x0)
        /home/tim/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/iterator.go:471 +0x93
github.com/plan-systems/plan-vault-libp2p/store.(*subscriber).read.func1.1(0xc00aaa4750, 0x0, 0x0)
        /home/tim/go/src/github.com/plan-systems/plan-vault-libp2p/store/store.go:479 +0x165
github.com/dgraph-io/badger/v2.(*DB).View(0xc00a982800, 0xc0001f3df0, 0x0, 0x0)
        /home/tim/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/txn.go:802 +0x95
github.com/plan-systems/plan-vault-libp2p/store.(*subscriber).read.func1(0x0, 0x0)
        /home/tim/go/src/github.com/plan-systems/plan-vault-libp2p/store/store.go:474 +0xab
github.com/plan-systems/plan-vault-libp2p/store.(*subscriber).read(0xc0002d0aa0, 0xc00aa18330, 0x0, 0x0)
        /home/tim/go/src/github.com/plan-systems/plan-vault-libp2p/store/store.go:557 +0x230
created by github.com/plan-systems/plan-vault-libp2p/store.(*Channel).Subscribe
        /home/tim/go/src/github.com/plan-systems/plan-vault-libp2p/store/store.go:377 +0x337
FAIL    github.com/plan-systems/plan-vault-libp2p/store 0.040s
FAIL

The last frame we own is on line 479 here:

plan-vault-libp2p/store/store.go

Lines 479 to 480 in 7617f31

	it := txn.NewIterator(iOpts)
	defer it.Close()

We set up the defer to close the iterator in the next line, which is the earliest we could possibly do so. If the txn being discarded can cause a panic, I'm not sure why it would be allowed to do so during the iterator constructor. It could be that one of our tests doesn't bother to finish iterating, so it's calling iterate() just as a test tear-down is destroying the DB... I'll need to check each of the store_test.go tests to make sure it isn't something dumb like that.

If not, I'll follow-up with upstream. Unfortunately the dgraph folks don't have a real public issue tracker, just a forum. https://discuss.dgraph.io/search?q=unclosed%20iterator doesn't show anything current for this kind of issue.