Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] tstore: Run updates in a transaction. #1406

Closed
wants to merge 144 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
144 commits
Select commit Hold shift + click to select a range
e5da7c0
tstorebe: Run all updates in a tx.
lukebp Apr 2, 2021
e946347
store/mysql: Implement Tx interface.
lukebp May 5, 2021
86b8461
store/localdb: Implement Tx interface.
lukebp May 9, 2021
36a0238
store/localdb: Finish Tx interface.
lukebp May 10, 2021
ff68f70
Cleanup tx implementations.
lukebp May 10, 2021
b2befe3
tstore: Implement record tx methods.
lukebp May 10, 2021
1b471b7
tstorebe: Remove record mutexes.
lukebp May 14, 2021
4c1b492
tstorebe: Use tx for record writes.
lukebp May 18, 2021
878465e
tstorebe: Setup blobkv cache.
lukebp May 18, 2021
14e9a38
Merge branch 'master' into kvtx
lukebp May 24, 2021
8a1341d
Move Getter to store package.
lukebp May 26, 2021
d85ec15
Start implementing inv package.
lukebp May 26, 2021
91d7f71
Finish implementing inv package.
lukebp May 27, 2021
59ab5d3
Fix localdb tx implementation.
lukebp May 27, 2021
aba8273
Bug fixes.
lukebp May 27, 2021
398f049
tstorebe: Start implementing recordinv.
lukebp May 28, 2021
14500da
tstorebe: Finish implementing recordinv.
lukebp Jun 3, 2021
71187f6
tstorbe: Fix RecordSetStatus.
lukebp Jun 6, 2021
877edcb
tstore: Add RecordTx
lukebp Jun 6, 2021
6dcc5b6
Linter fix.
lukebp Jun 6, 2021
e9857ab
inv: Add GetAll.
lukebp Jun 7, 2021
51a8b44
tstorebe: Add separate read and write plugin cmds.
lukebp Jun 8, 2021
e46b950
Start moving plugins over to tx updates.
lukebp Jun 8, 2021
c2b9c1f
Merge branch 'master' into kvtx
lukebp Jun 10, 2021
bbe3103
politeiad/comments: Refactor plugin.
lukebp Jun 10, 2021
bea3d39
tstorebe/comments: Get plugin to build.
lukebp Jun 11, 2021
4e6dee1
tstorebe: Get all plugins to build.
lukebp Jun 11, 2021
20d22ef
Merge branch 'master' into kvtx
lukebp Jun 13, 2021
3b9fde7
tstore: Finish client implmn.
lukebp Jun 14, 2021
5078412
tstorebe: Update plugin hooks.
lukebp Jun 16, 2021
32c86f9
tstorebe: Finish updating hooks.
lukebp Jun 29, 2021
ddc158f
Merge branch 'master' into kvtx
lukebp Jun 29, 2021
64b7f8d
Merge branch 'master' into kvtx
lukebp Jul 13, 2021
411d488
Merge branch 'master' into kvtx
lukebp Jul 13, 2021
18f61fa
Get pi plugin to build.
lukebp Jul 13, 2021
7410fb2
Merge branch 'master' into kvtx
lukebp Jul 14, 2021
80ed209
Get dcrdata plugin to build.
lukebp Jul 14, 2021
73fc182
usermd: Update hook naming.
lukebp Jul 14, 2021
fe839aa
usermd: Remove tstore dep.
lukebp Jul 14, 2021
80d192f
plugins: Add InvClient interface.
lukebp Jul 15, 2021
d80991d
usermd: Refactor user cache.
lukebp Jul 15, 2021
a737ee7
tstore: Del legacy client.
lukebp Jul 15, 2021
deb3396
tstorebe: Rename kv to cache.
lukebp Jul 15, 2021
f98e321
ticketvote: Rename hooks.
lukebp Jul 15, 2021
a0324ee
politeiad: Improve various logging.
lukebp Jul 2, 2021
cdaacb3
ticketvote: Refactor cmdAuthorize.
lukebp Jul 16, 2021
e3359bb
politeiad: Add BackendSettings.
lukebp Jul 16, 2021
0b5eff3
Merge branch 'master' into kvtx
lukebp Jul 23, 2021
641bf60
Merge commit '01ea3adc' into kvtx
lukebp Aug 4, 2021
7787fcb
Merge commit '2eef6113' into kvtx
lukebp Aug 4, 2021
a644a18
Merge commit '024b2e9a' into kvtx
lukebp Aug 4, 2021
4fb41fc
Merge commit '503b1e5d' into kvtx
lukebp Aug 4, 2021
f52ad14
Merge commit '8c95e641' into kvtx
lukebp Aug 4, 2021
1449544
inv: Add extra data field.
lukebp Aug 4, 2021
8794491
localdb: Add tx documentation.
lukebp Aug 9, 2021
02b791b
Merge branch 'master' into kvtx
lukebp Aug 9, 2021
de2fde8
tstorebe: Update inv calls.
lukebp Aug 19, 2021
2ada161
tstore: Remove data dir.
lukebp Aug 19, 2021
a566a85
tstorebe: Del data dir.
lukebp Aug 19, 2021
a3f4d7b
tstore: Use errors Errorf.
lukebp Aug 19, 2021
d5d37ee
store: Add Decode function.
lukebp Aug 22, 2021
de8712d
tstore: Make anchoring concurrency safe (partial).
lukebp Aug 22, 2021
9b5128f
tstore: Implmn droppingAnchorStart.
lukebp Aug 24, 2021
4822354
tstore: Implmn setupAnchorProcess.
lukebp Aug 24, 2021
cbe76fe
tstore: Finish anchor refactor.
lukebp Aug 24, 2021
45a26af
tstore: Update anchor naming convention.
lukebp Aug 26, 2021
7e5cb30
tstore: Remove short token mem cache.
lukebp Aug 26, 2021
84c14a9
tstore: Clean up extra data struct.
lukebp Aug 27, 2021
67a8bc9
tstore: Fix naming.
lukebp Aug 27, 2021
248a055
tstore: Replace tlog client errors.
lukebp Aug 27, 2021
11989ef
politeiad: Unpack error stack trace.
lukebp Aug 28, 2021
e196c3a
tstore: Improve extra data docs.
lukebp Aug 28, 2021
3652803
tstore: Simplify extra data key prefix.
lukebp Aug 28, 2021
6027bd6
tstorebe: Cleanup.
lukebp Aug 28, 2021
4f677ef
tstore: Refactor record index methods.
lukebp Aug 28, 2021
64e5f33
tstorebe: Refactor blob entry init.
lukebp Aug 28, 2021
32e87de
tstore: Cleanup.
lukebp Aug 28, 2021
4f66f3d
tstore: Use stack trace errors.
lukebp Aug 28, 2021
1a3bad4
Cleanup
lukebp Aug 28, 2021
19c9666
comments: Cleanup.
lukebp Aug 28, 2021
fd382fc
tstore: Add InvClient.
lukebp Aug 30, 2021
ece7a6f
tstore: Improve naming and docs.
lukebp Aug 30, 2021
905ee00
tstore: Cleanup and docs.
lukebp Aug 31, 2021
09b254b
store: Use stack trace errors.
lukebp Aug 31, 2021
20e32c4
store: MySQL Insert, Update, Get, GetBatch.
lukebp Aug 31, 2021
fe259cc
tstorebe: Verify mysql ops.
lukebp Sep 1, 2021
86246b4
store: Make errors more granular.
lukebp Sep 1, 2021
8e08621
mysql: Batch del query.
lukebp Sep 1, 2021
202c7aa
mysql: Get single op tests working.
lukebp Sep 2, 2021
ae64471
tstorebe: Remove Put kv method.
lukebp Sep 2, 2021
2d4d1f8
tstorebe: Clean setup process.
lukebp Sep 2, 2021
e2cc8e1
tstore: Get localdb working.
lukebp Sep 2, 2021
bcbfc0c
d/plugins: Update CachClient.
lukebp Sep 2, 2021
48f8f32
store: Run op tests on leveldb.
lukebp Sep 2, 2021
27e3972
tstore: Add CacheClient.
lukebp Sep 3, 2021
c18f9e5
plugins: Update to use CacheClient.
lukebp Sep 3, 2021
399b1fb
Cleanup.
lukebp Sep 6, 2021
fc1a931
usermd: Fix func name bug.
lukebp Sep 7, 2021
a4051ee
inv: Rename to Client.
lukebp Sep 7, 2021
0e7ba00
ticketvote: Start implementing inv.
lukebp Sep 7, 2021
f39fbe1
ticketvote: Add settings struct.
lukebp Sep 7, 2021
5a61b58
ticketvote: Add startStandard.
lukebp Sep 7, 2021
2ad456d
ticketvote: Refactor setting parsing.
lukebp Sep 7, 2021
fca3807
ticketvote: Add started inv call.
lukebp Sep 7, 2021
1cd35f7
ticketvote: Add runoffdetails.
lukebp Sep 7, 2021
f61f91c
ticketvote: Implmn cmdStartRunoffSub.
lukebp Sep 7, 2021
07cd709
Cleanup.
lukebp Sep 7, 2021
213016b
ticketvote: Get all write commands working.
lukebp Sep 8, 2021
3e23a0d
comments: Version record index cache.
lukebp Sep 8, 2021
d7b2275
ticketvote: Add runoffSubs cache.
lukebp Sep 8, 2021
7ecd7bb
store: Rename DataDescriptor.
lukebp Sep 9, 2021
bd58599
ticketvote: Add CmdDetails and CmdResults.
lukebp Sep 9, 2021
62b9e9a
tstorebe: Rename DataDescriptor.
lukebp Sep 9, 2021
3480551
ticketvote: Add local authDetails struct.
lukebp Sep 9, 2021
b1e7002
ticketvote: Add local vote details struct.
lukebp Sep 10, 2021
1d67a70
ticketvote: Add local castVoteDetails.
lukebp Sep 10, 2021
94ee5df
ticketvote: Remove dead code.
lukebp Sep 10, 2021
4239e52
ticketvote: Start implementing summaries cache.
lukebp Sep 10, 2021
402b374
ticketvote: Working on cmdCastBallot.
lukebp Sep 11, 2021
ea1db6e
ticketvote: Refactor commitment addresses.
lukebp Sep 21, 2021
ed2c402
Merge commit '75181166' into kvtx
lukebp Sep 21, 2021
ec349fc
pi: Fix merge issues.
lukebp Sep 21, 2021
2cbb4f9
ticketvote: Get to build temporarily.
lukebp Sep 21, 2021
cb6b1b9
Merge commit '325f107a10dfbeb285b79681b950021213c94dcd' into kvtx
lukebp Sep 21, 2021
af52d26
Merge commit '014cd1a' into kvtx
lukebp Sep 21, 2021
5c35641
Merge commit 'b909d7c7' into kvtx
lukebp Sep 21, 2021
3051452
Merge commit '5bc5b002' into kvtx
lukebp Sep 21, 2021
a04620a
Merge commit 'e7c6c65c' into kvtx
lukebp Sep 21, 2021
45bf43d
Merge commit 'f15b25e7' into kvtx
lukebp Sep 21, 2021
2cd53f0
Merge commit 'b43eebde' into kvtx
lukebp Sep 21, 2021
56d10b0
Merge commit '9fc474f6' into kvtx
lukebp Sep 21, 2021
b7aec3c
Merge commit '9b30978f' into kvtx
lukebp Sep 21, 2021
42a60b0
Merge commit '0dd068e3' into kvtx
lukebp Sep 21, 2021
be42d34
Merge commit '8f780d0b' into kvtx
lukebp Sep 21, 2021
f0e4d2c
Merge commit 'c9a52d6a' into kvtx
lukebp Sep 21, 2021
e92f431
Merge commit '7c2d2b7f' into kvtx
lukebp Sep 21, 2021
47636f2
Merge commit '1100a3bb' into kvtx
lukebp Sep 21, 2021
4d1dd54
Merge commit '91859262' into kvtx
lukebp Sep 21, 2021
4a76321
Merge commit '8c944a7a' into kvtx
lukebp Sep 21, 2021
ac56f5e
Merge commit 'a926339b' into kvtx
lukebp Sep 21, 2021
43c042d
Merge commit '8627759e' into kvtx
lukebp Sep 21, 2021
217db7e
Merge commit '44ee3108' into kvtx
lukebp Sep 21, 2021
0921122
Fix linter errors.
lukebp Sep 21, 2021
4779918
Merge branch 'master' into kvtx
lukebp Sep 22, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions politeiad/backendv2/backendv2.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"errors"
"fmt"

"github.com/decred/dcrd/chaincfg/v3"
"github.com/decred/politeia/politeiad/api/v1/identity"
)

Expand Down Expand Up @@ -256,6 +257,18 @@ type Inventory struct {
Vetted map[StatusT][]string
}

// BackendSettings are the backend settings that are available to plugins.
type BackendSettings struct {
// Identity contains the full identity that the plugin uses to
// create receipts, i.e. signatures of user provided data that
// prove the backend received and processed a plugin command.
Identity identity.FullIdentity

// Net contains the Decred network and its parameters that the
// backend is running on.
Net chaincfg.Params
}

// PluginSetting represents a configurable plugin setting.
//
// The value can either contain a single value or multiple values. Multiple
Expand All @@ -269,11 +282,6 @@ type PluginSetting struct {
type Plugin struct {
ID string
Settings []PluginSetting

// Identity contains the full identity that the plugin uses to
// create receipts, i.e. signatures of user provided data that
// prove the backend received and processed a plugin command.
Identity *identity.FullIdentity
}

// PluginError represents an error that occurred during plugin execution that
Expand Down
67 changes: 67 additions & 0 deletions politeiad/backendv2/tstorebe/blobkv.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
// Copyright (c) 2020-2021 The Decred developers
// Use of this source code is governed by an ISC
// license that can be found in the LICENSE file.

package tstorebe

import (
"fmt"

"github.com/decred/politeia/politeiad/backendv2/tstorebe/store"
"github.com/decred/politeia/politeiad/backendv2/tstorebe/store/localdb"
"github.com/decred/politeia/politeiad/backendv2/tstorebe/store/mysql"
"github.com/pkg/errors"
)

const (
// DBTypeLevelDB is the config option that sets the backing
// key-value store to a leveldb instance.
DBTypeLevelDB = "leveldb"

// DBTypeMySQL is a config option that sets the backing key-value
// store to a MySQL instance.
DBTypeMySQL = "mysql"

// dbUser contains the username that will be used to connect to
// the MySQL instance.
dbUser = "politeiad"
)

// blobKVOpts contains configuration options for a blob key-value store.
type blobKVOpts struct {
Type string // Database type

// LevelDB options
AppDir string // App home directory
DataDir string // App data directory

// MySQL options
Host string // MySQL host
Password string // Password for politeiad user
Net string // Active net name
}

// newBlobKV initializes and returns a new BlobKV.
func newBlobKV(opts blobKVOpts) (store.BlobKV, error) {
var (
kv store.BlobKV
err error
)
switch opts.Type {
case DBTypeLevelDB:
kv, err = localdb.New(opts.AppDir, opts.DataDir)
if err != nil {
return nil, err
}
case DBTypeMySQL:
dbName := fmt.Sprintf("%v_kv", opts.Net)
kv, err = mysql.New(opts.Host, dbUser, opts.Password, dbName)
if err != nil {
return nil, err
}
default:
return nil, errors.Errorf("invalid db type '%v'", opts.Type)
}

return kv, nil
}
Loading