Introduce tablets to SM #3753

Michal-Leszczynski · 2024-03-14T16:38:36Z

Testing against

scylla-nightly:5.5.0-dev-0.20240314.ce1784186042
with / without raft topology
with / without tablets

There was a little mix up with different PRs being opened and closed regarding tablet related functionalities. This one is supposed to prepare backup and repair services for tablet tables and test them.

Fixes #3759
Fixes #3704
Fixes #3703
Fixes #3713

Ref #16509 (in scylladb)

Vnode keyspaces have per keyspace ring, and tablet keyspaces have per table ring.

This allows for checking which keyspaces have vnodes/tablets replication.

Michal-Leszczynski · 2024-03-14T16:45:51Z

Used scylla nightly version has all options used by SM repair enabled.

Investigated issues:
Scenario: cluster 2 dcs 3 nodes each, tablet keyspace with replication {'dc2': 2}, creating table, writing data, flushing table on all hosts, waiting 30s.

getting table size (/column_family/metrics/total_disk_space_used/{name}) sometimes returns 404.
SM error: Repair failed: create repair plan: calculate tables size: 192.168.200.23: giving up after 2 attempts: agent [HTTP 404] Not found
scheduling repair (/storage_service/repair_async/{keyspace}) sometimes returns 404.
SM error: Repair failed {"error": "master 192.168.200.22 keyspace test_repair_rf_2 table test_table_2 command 0: schedule repair: giving up after 3 attempts: agent [HTTP 404] Not found
scheduled repair failed on Scylla side.
Scylla error: WARN 2024-03-14 17:35:23,726 [shard 0:strm] repair - repair[cd35f88b-e481-4105-ba82-998c48179cc8]: Repair tablet for keyspace=test_repair_rf_2 status=failed: std::runtime_error (repair[cd35f88b-e481-4105-ba82-998c48179cc8]: 0 out of 1 ranges failed, keyspace=test_repair_rf_2, tables={test_table_2}, repair_reason=repair, nodes_down_during_repair={}, aborted_by_user=false, failed_because=seastar::rpc::closed_error (connection is closed))

Note: all issues mentioned above are really flaky (meaning that the test passes sometimes - especially when ignoring erros from querying table size). After setting up the scenario and running repair multiple times, it is possible that it fails, succeeds and then fails again.

Other:

new system keyspace introduced: system_auth_v2. What about its repair order?

Michal-Leszczynski · 2024-03-14T16:50:46Z

@asias @karol-kokoszka FYI this is the PR preparing SM for tablets - tests are running and I'm investigating some encountered issues described above.

Michal-Leszczynski · 2024-03-14T18:13:58Z

Tests against Scylla nightly with TABLETS DISABLED passed for all services (link here).

Michal-Leszczynski · 2024-03-14T18:22:28Z

Tests against Scylla nightly with TABLETS:

backup - passed - but there is no certainty since restore tests fail
repair - failed - see reasons mentioned above
restore tables - failed - it goes through repair and fails when comparing row count in source and destination cluster

Michal-Leszczynski · 2024-03-14T19:13:30Z

From the logs in restore tables tests we can see that tablet table is repaired:

20:07:56.783	INFO	repair.restoretest_tables_smoke.big_table	Start table generator
20:07:56.787	INFO	repair.worker 1	Repairing	{"keyspace": "restoretest_tables_smoke", "table": "big_table", "master": "192.168.200.11", "hosts": ["192.168.200.11", "192.168.200.12"], "ranges": [{"StartToken":-9223372036854775808,"EndToken":-6917529027641081857}], "job_id": 7}
20:07:56.828	INFO	repair.worker 1	Repair done	{"job_id": 7}
20:07:56.835	INFO	repair.worker 1	Repairing	{"keyspace": "restoretest_tables_smoke", "table": "big_table", "master": "192.168.200.11", "hosts": ["192.168.200.11", "192.168.200.12"], "ranges": [{"StartToken":-6917529027641081857,"EndToken":-4611686018427387905}], "job_id": 8}
20:07:56.835	INFO	repair.worker 1	Repair done	{"job_id": 8}
20:07:56.842	INFO	repair.worker 1	Repairing	{"keyspace": "restoretest_tables_smoke", "table": "big_table", "master": "192.168.200.12", "hosts": ["192.168.200.12", "192.168.200.13"], "ranges": [{"StartToken":-4611686018427387905,"EndToken":-2305843009213693953}], "job_id": 5}
20:07:56.842	INFO	repair.worker 1	Repair done	{"job_id": 5}
20:07:56.848	INFO	repair.worker 1	Repairing	{"keyspace": "restoretest_tables_smoke", "table": "big_table", "master": "192.168.200.12", "hosts": ["192.168.200.12", "192.168.200.13"], "ranges": [{"StartToken":-2305843009213693953,"EndToken":-1}], "job_id": 6}
20:07:56.849	INFO	repair.worker 1	Repair done	{"job_id": 6}
20:07:56.855	INFO	repair.worker 1	Repairing	{"keyspace": "restoretest_tables_smoke", "table": "big_table", "master": "192.168.200.12", "hosts": ["192.168.200.12", "192.168.200.13"], "ranges": [{"StartToken":4611686018427387903,"EndToken":6917529027641081855}], "job_id": 7}
20:07:56.855	INFO	repair.worker 1	Repair done	{"job_id": 7}
20:07:56.862	INFO	repair.worker 1	Repairing	{"keyspace": "restoretest_tables_smoke", "table": "big_table", "master": "192.168.200.11", "hosts": ["192.168.200.11", "192.168.200.13"], "ranges": [{"StartToken":-1,"EndToken":2305843009213693951}], "job_id": 9}
20:07:56.905	INFO	repair.worker 1	Repair done	{"job_id": 9}
20:07:56.911	INFO	repair.worker 1	Repairing	{"keyspace": "restoretest_tables_smoke", "table": "big_table", "master": "192.168.200.11", "hosts": ["192.168.200.11", "192.168.200.13"], "ranges": [{"StartToken":2305843009213693951,"EndToken":4611686018427387903}], "job_id": 10}
20:07:56.912	INFO	repair.worker 1	Repair done	{"job_id": 10}
20:07:56.918	INFO	repair.worker 1	Repairing	{"keyspace": "restoretest_tables_smoke", "table": "big_table", "master": "192.168.200.11", "hosts": ["192.168.200.11", "192.168.200.13"], "ranges": [{"StartToken":6917529027641081855,"EndToken":9223372036854775807}], "job_id": 11}
20:07:56.918	INFO	repair.worker 1	Repair done	{"job_id": 11}
20:07:56.922	INFO	repair.restoretest_tables_smoke.big_table	Close table generator

yet it doesn't contain all required rows afterwards:

{restoretest_tables_smoke big_table id}, srcCount = 6400, dstCount = 1591

It's worth to note that schema is recreated via CQL on the restore destination cluster and that both cluster support tablets.

Michal-Leszczynski · 2024-03-15T13:04:35Z

I also run the tests with just raft-topology enabled and they passed - meaning that enabling tablets messes up the tests.

Michal-Leszczynski · 2024-03-15T13:08:09Z

FYI @dani-tweig @karol-kokoszka @asias

karol-kokoszka · 2024-03-15T18:04:18Z

@bhalevy this PR may be interesting for you.
@Michal-Leszczynski executed SM tests against the tablets with Scylla taken from the nightly build #3753 (comment)

dani-tweig · 2024-03-17T13:49:24Z

@Michal-Leszczynski

Why is this PR in 'draft' mode?
What is the issue it refers to?

asias · 2024-03-18T06:14:48Z

Used scylla nightly version has all options used by SM repair enabled.

Investigated issues: Scenario: cluster 2 dcs 3 nodes each, tablet keyspace with replication {'dc2': 2}, creating table, writing data, flushing table on all hosts, waiting 30s.
* getting table size (`/column_family/metrics/total_disk_space_used/{name}`) sometimes returns 404.
  SM error: `Repair failed: create repair plan: calculate tables size: 192.168.200.23: giving up after 2 attempts: agent [HTTP 404] Not found`

* scheduling repair (`/storage_service/repair_async/{keyspace}`) sometimes returns 404.
  SM error: `Repair failed	{"error": "master 192.168.200.22 keyspace test_repair_rf_2 table test_table_2 command 0: schedule repair: giving up after 3 attempts: agent [HTTP 404] Not found`

* scheduled repair failed on Scylla side.
  Scylla error: `WARN  2024-03-14 17:35:23,726 [shard 0:strm] repair - repair[cd35f88b-e481-4105-ba82-998c48179cc8]: Repair tablet for keyspace=test_repair_rf_2 status=failed: std::runtime_error (repair[cd35f88b-e481-4105-ba82-998c48179cc8]: 0 out of 1 ranges failed, keyspace=test_repair_rf_2, tables={test_table_2}, repair_reason=repair, nodes_down_during_repair={}, aborted_by_user=false, failed_because=seastar::rpc::closed_error (connection is closed))`
Note: all issues mentioned above are really flaky (meaning that the test passes sometimes - especially when ignoring erros from querying table size). After setting up the scenario and running repair multiple times, it is possible that it fails, succeeds and then fails again.

Other:
* new system keyspace introduced: `system_auth_v2`. What about its repair order?

SM is supposed to ignore system_auth_v2. It is a raft managed table which does not need repair.

asias · 2024-03-18T06:22:13Z

Used scylla nightly version has all options used by SM repair enabled.

Investigated issues: Scenario: cluster 2 dcs 3 nodes each, tablet keyspace with replication {'dc2': 2}, creating table, writing data, flushing table on all hosts, waiting 30s.

* getting table size (`/column_family/metrics/total_disk_space_used/{name}`) sometimes returns 404.
  SM error: `Repair failed: create repair plan: calculate tables size: 192.168.200.23: giving up after 2 attempts: agent [HTTP 404] Not found`

* scheduling repair (`/storage_service/repair_async/{keyspace}`) sometimes returns 404.
  SM error: `Repair failed	{"error": "master 192.168.200.22 keyspace test_repair_rf_2 table test_table_2 command 0: schedule repair: giving up after 3 attempts: agent [HTTP 404] Not found`

* scheduled repair failed on Scylla side.
  Scylla error: `WARN  2024-03-14 17:35:23,726 [shard 0:strm] repair - repair[cd35f88b-e481-4105-ba82-998c48179cc8]: Repair tablet for keyspace=test_repair_rf_2 status=failed: std::runtime_error (repair[cd35f88b-e481-4105-ba82-998c48179cc8]: 0 out of 1 ranges failed, keyspace=test_repair_rf_2, tables={test_table_2}, repair_reason=repair, nodes_down_during_repair={}, aborted_by_user=false, failed_because=seastar::rpc::closed_error (connection is closed))`

This sounds like a node was down during repair, e.g., a node is restarted. Did you try some basic tests like starting a cluster manually, insert data, repair with SM. Did you see similar repair errors?

Note: all issues mentioned above are really flaky (meaning that the test passes sometimes - especially when ignoring erros from querying table size). After setting up the scenario and running repair multiple times, it is possible that it fails, succeeds and then fails again.

Other:
* new system keyspace introduced: `system_auth_v2`. What about its repair order?

Michal-Leszczynski · 2024-03-18T10:19:19Z

@asias you are right, it looks like some node is getting segfault and then restarts. This are example logs:

INFO  2024-03-18 09:50:49,948 [shard 0:strm] schema_tables - Creating keyspace test_repair_rf_3
INFO  2024-03-18 09:50:49,956 [shard 0:strm] migration_manager - Gossiping my schema version 00668f4a-e50d-11ee-ee93-e05f713fb39f
INFO  2024-03-18 09:50:49,956 [shard 0:strm] schema_tables - Schema version changed to 00668f4a-e50d-11ee-ee93-e05f713fb39f
INFO  2024-03-18 09:50:52,114 [shard 0:strm] schema_tables - Creating test_repair_rf_3.test_table_1 id=01b3f310-e50d-11ee-ab86-1e408f0c9f18 version=01b3f311-e50d-11ee-ab86-1e408f0c9f18
INFO  2024-03-18 09:50:52,114 [shard 0:strm] schema_tables - Tablet metadata changed
INFO  2024-03-18 09:50:52,148 [shard 0:strm] migration_manager - Gossiping my schema version 01b3f3ec-e50d-11ee-b922-1270073624e7
INFO  2024-03-18 09:50:52,148 [shard 0:strm] schema_tables - Schema version changed to 01b3f3ec-e50d-11ee-b922-1270073624e7
INFO  2024-03-18 09:50:52,207 [shard 0:comp] compaction - [Compact system.peers 01c4bbf0-e50d-11ee-8d0a-6ee4d5c18d0b] Compacting [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/me-3gej_0r9a_24y28257pj6szorsi3-big-Data.db:level=0:origin=compaction,/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/me-3gej_0rcs_0ofmo257pj6szorsi3-big-Data.db:level=0:origin=memtable]
INFO  2024-03-18 09:50:52,324 [shard 0:comp] compaction - [Compact system.peers 01c4bbf0-e50d-11ee-8d0a-6ee4d5c18d0b] Compacted 2 sstables to [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/me-3gej_0rcs_18d81257pj6szorsi3-big-Data.db:level=0]. 66kB to 36kB (~55% of original) in 100ms = 663kB/s. ~256 total partitions merged to 5.
INFO  2024-03-18 09:50:52,324 [shard 0:comp] compaction - [Compact system.local 01d69640-e50d-11ee-8d0a-6ee4d5c18d0b] Compacting [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/me-3gej_0rcs_0w5eo257pj6szorsi3-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/me-3gej_0rcs_0qksg257pj6szorsi3-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/me-3gej_0r9a_15kwx257pj6szorsi3-big-Data.db:level=0:origin=compaction]
INFO  2024-03-18 09:50:52,415 [shard 0:comp] compaction - [Compact system.local 01d69640-e50d-11ee-8d0a-6ee4d5c18d0b] Compacted 3 sstables to [/var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/me-3gej_0rcs_1xnps257pj6szorsi3-big-Data.db:level=0]. 30kB to 12kB (~42% of original) in 78ms = 391kB/s. ~384 total partitions merged to 1.
INFO  2024-03-18 09:50:55,529 [shard 0:strm] schema_tables - Creating test_repair_rf_3.test_table_2 id=03bc9450-e50d-11ee-b2de-98b886307c8c version=03bc9451-e50d-11ee-b2de-98b886307c8c
INFO  2024-03-18 09:50:55,529 [shard 0:strm] schema_tables - Tablet metadata changed
INFO  2024-03-18 09:50:55,559 [shard 0:strm] migration_manager - Gossiping my schema version 03bca8b4-e50d-11ee-a396-da13942235e0
INFO  2024-03-18 09:50:55,559 [shard 0:strm] schema_tables - Schema version changed to 03bca8b4-e50d-11ee-a396-da13942235e0
{"L":"DEBUG","T":"2024-03-18T09:50:58.630Z","N":"http","M":"GET /snitch/datacenter?host=192.168.200.13","from":"192.168.200.1:54180","status":200,"bytes":5,"duration":"0ms"}
{"L":"DEBUG","T":"2024-03-18T09:50:58.633Z","N":"http","M":"GET /gossiper/endpoint/live/","from":"192.168.200.1:54180","status":200,"bytes":103,"duration":"0ms"}
{"L":"DEBUG","T":"2024-03-18T09:50:58.636Z","N":"http","M":"GET /column_family/name","from":"192.168.200.1:54180","status":200,"bytes":2041,"duration":"0ms"}
{"L":"DEBUG","T":"2024-03-18T09:50:58.639Z","N":"http","M":"GET /storage_service/host_id","from":"192.168.200.1:54180","status":200,"bytes":451,"duration":"0ms"}
{"L":"DEBUG","T":"2024-03-18T09:50:58.651Z","N":"http","M":"GET /storage_service/describe_ring/system","from":"192.168.200.1:54180","status":200,"bytes":360580,"duration":"10ms"}
{"L":"DEBUG","T":"2024-03-18T09:50:58.695Z","N":"http","M":"GET /storage_service/describe_ring/system_auth","from":"192.168.200.1:54180","status":200,"bytes":1120900,"duration":"29ms"}
{"L":"DEBUG","T":"2024-03-18T09:50:58.753Z","N":"http","M":"GET /storage_service/describe_ring/system_auth_v2","from":"192.168.200.1:54180","status":200,"bytes":360580,"duration":"5ms"}
{"L":"DEBUG","T":"2024-03-18T09:50:58.781Z","N":"http","M":"GET /storage_service/describe_ring/system_distributed","from":"192.168.200.1:54180","status":200,"bytes":664708,"duration":"10ms"}
{"L":"DEBUG","T":"2024-03-18T09:50:58.847Z","N":"http","M":"GET /storage_service/describe_ring/system_distributed_everywhere","from":"192.168.200.1:54180","status":200,"bytes":1120900,"duration":"34ms"}
{"L":"DEBUG","T":"2024-03-18T09:50:58.914Z","N":"http","M":"GET /storage_service/describe_ring/system_schema","from":"192.168.200.1:54180","status":200,"bytes":360580,"duration":"9ms"}
Segmentation fault on shard 1.
Backtrace:
  0x5dbc458
  0x5df2c05
  /opt/scylladb/libreloc/libc.so.6+0x3dbaf
  0x486abdf
  0x4113970
  0x3ee790d
  0x405cc1c
  0x4b190ab
  0x4b180f5
  0x61e7513
  0x61e7476
  0x61e73fb
  0x61e684c
  0x61ecd97
  0x61c7fb4
  0x61cec54
  0x61ce061
  0x61cd202
  0x61d00e1
  0x5dce08f
  0x5dcf377
  0x5df32a0
  0x5d8e9ca
  /opt/scylladb/libreloc/libc.so.6+0x8c946
  /opt/scylladb/libreloc/libc.so.6+0x11296f
2024-03-18 09:51:01,325 INFO exited: scylla (terminated by SIGSEGV (core dumped); not expected)
{"L":"INFO","T":"2024-03-18T09:51:01.324Z","M":"http: proxy error: read tcp 127.0.0.1:54208->127.0.0.1:10000: read: connection reset by peer"}
{"L":"ERROR","T":"2024-03-18T09:51:01.324Z","N":"http","M":"GET /storage_service/describe_ring/test_repair_rf_3?table=test_table_1","from":"192.168.200.1:54180","status":502,"bytes":0,"duration":"2383ms","S":"github.com/scylladb/go-log.Logger.log\n\tgithub.com/scylladb/[email protected]/logger.go:101\ngithub.com/scylladb/go-log.Logger.Error\n\tgithub.com/scylladb/[email protected]/logger.go:84\nmain.(*logEntry).Write\n\tgithub.com/scylladb/scylla-manager/v3/pkg/cmd/agent/log.go:53\nmain.newRouter.RequestLogger.RequestLogger.func5.1.1\n\tgithub.com/go-chi/chi/[email protected]/middleware/logger.go:54\nmain.newRouter.RequestLogger.RequestLogger.func5.1\n\tgithub.com/go-chi/chi/[email protected]/middleware/logger.go:58\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/go-chi/chi/v5.(*Mux).ServeHTTP\n\tgithub.com/go-chi/chi/[email protected]/mux.go:87\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2938\nnet/http.(*conn).serve\n\tnet/http/server.go:2009"}
2024-03-18 09:51:01,328 INFO spawned: 'scylla' with pid 1175

Michal-Leszczynski · 2024-03-18T10:24:17Z

It looks like it happens after querying DescribeRing of table created for test purposes.
The test is just creating a single tablet table ({'dc1: 3, 'dc2':3'}), writing 5MB to it and then trying to repair it, but it fails at calculating table size, so no repair is scheduled.

Michal-Leszczynski · 2024-03-18T10:25:49Z

SM is supposed to ignore system_auth_v2. It is a raft managed table which does not need repair.

Is there a way to know which tables are raft based and should be skipped by SM repair?

Michal-Leszczynski · 2024-03-18T10:41:34Z

Here are logs from mentioned test scenario:
logs.zip

Michal-Leszczynski · 2024-03-18T11:16:57Z

Why is this PR in 'draft' mode?

Changed to ready for review.

What is the issue it refers to?

It is unsure if the problem lies in SM test env setup or something on Scylla side or in some misunderstanding of tablet features.

bhalevy · 2024-03-18T13:37:56Z

Decoded Segmentation fault:

[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:68
 (inlined by) seastar::backtrace_buffer::append_backtrace() at ./build/release/seastar/./seastar/src/core/reactor.cc:825
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:855
seastar::print_with_backtrace(char const*, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:867
 (inlined by) seastar::sigsegv_action() at ./build/release/seastar/./seastar/src/core/reactor.cc:4066
 (inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4047
 (inlined by) __invoke at ./build/release/seastar/./seastar/src/core/reactor.cc:4043
/data/scylla-s3-reloc.cache/by-build-id/88e75a691d931975c5c6ee7cc071e4cbaf87257c/extracted/scylla/libreloc/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=70e92bb237883be3065a6afc9f0696aef2d068bf, for GNU/Linux 3.2.0, not stripped

__GI___sigaction at :?
service::wait_until_every_peer_joined_group0(db::system_keyspace&, service::group0_members const&, seastar::abort_source&) at raft_group0.cc:?
service::storage_service::host2ip(utils::tagged_uuid<locator::host_id_tag>) const at ./service/storage_service.cc:5361
 (inlined by) operator() at ./service/storage_service.cc:4802
 (inlined by) seastar::noncopyable_function<seastar::future<void> (locator::tablet_id, locator::tablet_info const&)>::direct_vtable_for<service::storage_service::describe_ring_for_table(seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&) const::$_0>::call(seastar::noncopyable_function<seastar::future<void> (locator::tablet_id, locator::tablet_info const&)> const*, locator::tablet_id, locator::tablet_info const&) at ././seastar/include/seastar/util/noncopyable_function.hh:129
seastar::noncopyable_function<seastar::future<void> (locator::tablet_id, locator::tablet_info const&)>::operator()(locator::tablet_id, locator::tablet_info const&) const at ././seastar/include/seastar/util/noncopyable_function.hh:215
 (inlined by) locator::tablet_map::for_each_tablet(seastar::noncopyable_function<seastar::future<void> (locator::tablet_id, locator::tablet_info const&)>) const at ./locator/tablets.cc:255
service::storage_service::describe_ring_for_table(seastar::basic_sstring<char, unsigned int, 15u, true> const&, seastar::basic_sstring<char, unsigned int, 15u, true> const&) const at ./service/storage_service.cc:4789
api::describe_ring_as_json_for_table(seastar::sharded<service::storage_service> const&, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>) at ./api/storage_service.cc:579
operator() at ./api/storage_service.cc:719
 (inlined by) seastar::future<seastar::json::json_return_type> std::__invoke_impl<seastar::future<seastar::json::json_return_type>, api::set_storage_service(api::http_context&, seastar::httpd::routes&, seastar::sharded<service::storage_service>&, service::raft_group0_client&)::$_10&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >(std::__invoke_other, api::set_storage_service(api::http_context&, seastar::httpd::routes&, seastar::sharded<service::storage_service>&, service::raft_group0_client&)::$_10&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
 (inlined by) std::enable_if<is_invocable_r_v<seastar::future<seastar::json::json_return_type>, api::set_storage_service(api::http_context&, seastar::httpd::routes&, seastar::sharded<service::storage_service>&, service::raft_group0_client&)::$_10&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >, seastar::future<seastar::json::json_return_type> >::type std::__invoke_r<seastar::future<seastar::json::json_return_type>, api::set_storage_service(api::http_context&, seastar::httpd::routes&, seastar::sharded<service::storage_service>&, service::raft_group0_client&)::$_10&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >(api::set_storage_service(api::http_context&, seastar::httpd::routes&, seastar::sharded<service::storage_service>&, service::raft_group0_client&)::$_10&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:114
 (inlined by) std::_Function_handler<seastar::future<seastar::json::json_return_type> (std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >), api::set_storage_service(api::http_context&, seastar::httpd::routes&, seastar::sharded<service::storage_service>&, service::raft_group0_client&)::$_10>::_M_invoke(std::_Any_data const&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/std_function.h:290
std::function<seastar::future<seastar::json::json_return_type> (std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)>::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/std_function.h:591
 (inlined by) operator() at ./build/release/seastar/./seastar/include/seastar/http/function_handlers.hh:102
seastar::future<std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> > > std::__invoke_impl<seastar::future<std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> > >, seastar::httpd::function_handler::function_handler(std::function<seastar::future<seastar::json::json_return_type> (std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)> const&)::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >)#1}&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> > >(std::__invoke_other, seastar::httpd::function_handler::function_handler(std::function<seastar::future<seastar::json::json_return_type> (std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)> const&)::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >)#1}&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
std::enable_if<is_invocable_r_v<seastar::future<std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> > >, seastar::httpd::function_handler::function_handler(std::function<seastar::future<seastar::json::json_return_type> (std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)> const&)::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >)#1}&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> > >, std::enable_if>::type std::__invoke_r<seastar::future<std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> > >, seastar::httpd::function_handler::function_handler(std::function<seastar::future<seastar::json::json_return_type> (std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)> const&)::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >)#1}&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> > >(seastar::future<std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> > >&&, (seastar::httpd::function_handler::function_handler(std::function<seastar::future<seastar::json::json_return_type> (std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)> const&)::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >)#1}&)...) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:114
 (inlined by) std::_Function_handler<seastar::future<std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> > > (std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >), seastar::httpd::function_handler::function_handler(std::function<seastar::future<seastar::json::json_return_type> (std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)> const&)::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >)#1}>::_M_invoke(std::_Any_data const&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/std_function.h:290
std::function<seastar::future<std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> > > (std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >)>::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/std_function.h:591
 (inlined by) seastar::httpd::function_handler::handle(seastar::basic_sstring<char, unsigned int, 15u, true> const&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >) at ./build/release/seastar/./seastar/include/seastar/http/function_handlers.hh:118
seastar::httpd::routes::handle(seastar::basic_sstring<char, unsigned int, 15u, true> const&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::reply, std::default_delete<seastar::http::reply> >) at ./build/release/seastar/./seastar/src/http/routes.cc:95
seastar::httpd::connection::generate_reply(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) at ./build/release/seastar/./seastar/src/http/httpd.cc:363
operator() at ./build/release/seastar/./seastar/src/http/httpd.cc:275
 (inlined by) seastar::future<bool> seastar::futurize<seastar::future<bool> >::invoke<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda()#1}>(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda()#1}&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2022
 (inlined by) seastar::future<bool> seastar::futurize<seastar::future<bool> >::invoke<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda()#1}>(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda()#1}&&, seastar::internal::monostate) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1866
 (inlined by) seastar::future<bool> seastar::future<void>::then_impl<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda()#1}, seastar::future<bool> >(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda()#1}&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1490
 (inlined by) seastar::future<bool> seastar::future<void>::then<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda()#1}, seastar::future<bool> >(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda()#1}&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1416
 (inlined by) operator() at ./build/release/seastar/./seastar/src/http/httpd.cc:274
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}&&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2022
 (inlined by) seastar::future<void> seastar::future<std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >::then_impl<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}, seastar::future<void> >(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1490
 (inlined by) seastar::future<void> seastar::future<std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >::then<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}, seastar::future<void> >(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}::operator()(seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&) const::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1416
 (inlined by) operator() at ./build/release/seastar/./seastar/src/http/httpd.cc:273
 (inlined by) seastar::future<void> std::__invoke_impl<seastar::future<void>, seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&, seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&>(std::__invoke_other, seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&, seastar::input_stream<char>&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, seastar::basic_sstring<char, unsigned int, 15u, true>&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
 (inlined by) std::__invoke_result<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&, seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&>::type std::__invoke<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&, seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>&>(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&, (std::__invoke_result&&)...) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:96
 (inlined by) decltype(auto) std::__apply_impl<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&, std::tuple<seastar::input_stream, seastar::basic_sstring, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >&, 0ul, 1ul, 2ul>(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&, std::tuple<seastar::input_stream, seastar::basic_sstring, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/tuple:2288
 (inlined by) decltype(auto) std::apply<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&, std::tuple<seastar::input_stream, seastar::basic_sstring, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >&>(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&, std::tuple<seastar::input_stream, seastar::basic_sstring, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/tuple:2299
 (inlined by) auto seastar::internal::do_with_impl<seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}>(seastar::input_stream<char>&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&, seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&&) at ./build/release/seastar/./seastar/include/seastar/core/do_with.hh:99
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::future<void> (*&)(seastar::input_stream<char>&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&, seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&&), seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&>(seastar::future<void> (*&)(seastar::input_stream<char>&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&, seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&&), seastar::input_stream<char>&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2022
 (inlined by) auto seastar::futurize_invoke<seastar::future<void> (*&)(seastar::input_stream<char>&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&, seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&&), seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&>(seastar::future<void> (*&)(seastar::input_stream<char>&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&, seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&&), seastar::input_stream<char>&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2053
 (inlined by) auto seastar::do_with<seastar::input_stream<char>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >, seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}>(seastar::input_stream<char>&&, seastar::basic_sstring<char, unsigned int, 15u, true>&&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&, seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(seastar::input_stream<char>&, seastar::basic_sstring<char, unsigned int, 15u, true>&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&)#1}&&) at ./build/release/seastar/./seastar/include/seastar/core/do_with.hh:137
 (inlined by) operator() at ./build/release/seastar/./seastar/src/http/httpd.cc:272
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}&&, std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2022
 (inlined by) seastar::future<void> seastar::future<std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >::then_impl<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}, seastar::future<void> >(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1490
 (inlined by) seastar::future<void> seastar::future<std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> > >::then<seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}, seastar::future<void> >(seastar::httpd::connection::read_one()::$_0::operator()()::{lambda(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >)#1}&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1416
 (inlined by) operator() at ./build/release/seastar/./seastar/src/http/httpd.cc:271
seastar::future<void> std::__invoke_impl<seastar::future<void>, seastar::httpd::connection::read_one()::$_0&>(std::__invoke_other, seastar::httpd::connection::read_one()::$_0&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
 (inlined by) std::__invoke_result<seastar::httpd::connection::read_one()::$_0&>::type std::__invoke<seastar::httpd::connection::read_one()::$_0&>(seastar::httpd::connection::read_one()::$_0&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:96
 (inlined by) std::invoke_result<seastar::httpd::connection::read_one()::$_0&>::type std::invoke<seastar::httpd::connection::read_one()::$_0&>(seastar::httpd::connection::read_one()::$_0&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/functional:113
 (inlined by) auto seastar::internal::future_invoke<seastar::httpd::connection::read_one()::$_0&, seastar::internal::monostate>(seastar::httpd::connection::read_one()::$_0&, seastar::internal::monostate&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1172
 (inlined by) operator() at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1475
 (inlined by) void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<void>::then_impl_nrvo<seastar::httpd::connection::read_one()::$_0, seastar::future<void> >(seastar::httpd::connection::read_one()::$_0&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::httpd::connection::read_one()::$_0&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::httpd::connection::read_one()::$_0&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, seastar::httpd::connection::read_one()::$_0&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2007
 (inlined by) operator() at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1471
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::httpd::connection::read_one()::$_0, seastar::future<void>::then_impl_nrvo<seastar::httpd::connection::read_one()::$_0, seastar::future<void> >(seastar::httpd::connection::read_one()::$_0&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::httpd::connection::read_one()::$_0&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose() at ./build/release/seastar/./seastar/include/seastar/core/future.hh:748

It allows for easy and efficient ring describing which works with older Scylla versions and tablet and vnode tables.

Michal-Leszczynski · 2024-03-22T14:16:38Z

@dani-tweig @karol-kokoszka good news! It looks like after the segfault fix the tests are mostly passing.
Failures (all in repair tests) look like connected to the fact that repair happens too quickly - mainly because with tablets we have 8 ranges to repair instead of thousands, so we save a lot of time on API calls. I will update the tests so that they work with tablets as well.

dani-tweig · 2024-03-24T11:40:43Z

It looks like after the segfault fix the tests are mostly passing.

@Michal-Leszczynski
This is great news.
Thank you for the update.

@bhalevy , FYI

This is required for tablet tables which have per table ring description.

…oint

Fixes #3759

Some tests assumed that SM is going to send thousands of ranges to be repaired and that it is easy to interrupt this process in the middle without any synchronization. Tablet tables have far fewer ranges (e.g. 8), so they require dedicated synchronization for those cases.

Michal-Leszczynski · 2024-03-27T09:56:06Z

@karol-kokoszka could you take a look at this PR?

karol-kokoszka · 2024-03-27T10:49:23Z

I like the change with introducing nightly build.
Overall the PR is good.

karol-kokoszka

👍

Michal-Leszczynski · 2024-04-03T08:32:33Z

Even though there were some problems regarding dtests that doesn't look connected to tablets, I'm going to merge it so that the master dtests and sct will have a chance of passing.

Michal-Leszczynski added 3 commits March 13, 2024 17:20

feat(swagger): add parameters needed to repair tablet tables

5a387cf

Ref #16509 (in scylladb)

feat(scyllaclient): decouple vnode and tablet ring description

066ff54

Vnode keyspaces have per keyspace ring, and tablet keyspaces have per table ring.

feat(scyllaclient): add ReplicationKeyspaces

08c0a9d

This allows for checking which keyspaces have vnodes/tablets replication.

Michal-Leszczynski mentioned this pull request Mar 14, 2024

Tablet backup and restore #3748

Closed

Michal-Leszczynski force-pushed the ml/tablet-repair-cont branch 2 times, most recently from 8525fa6 to 1a5fa11 Compare March 14, 2024 17:10

dani-tweig added the tablets label Mar 17, 2024

asias mentioned this pull request Mar 18, 2024

RFC: backward-compatible implement repair for Tablets scylladb/scylladb#16101

Closed

Michal-Leszczynski marked this pull request as ready for review March 18, 2024 10:41

Michal-Leszczynski requested a review from karol-kokoszka as a code owner March 18, 2024 10:41

bhalevy mentioned this pull request Mar 18, 2024

Scylla segfaults from the api::describe_ring path with manager repair scylladb/scylladb#17871

Closed

feat(scyllaclient): add RingDescriber

d1be94c

It allows for easy and efficient ring describing which works with older Scylla versions and tablet and vnode tables.

Michal-Leszczynski force-pushed the ml/tablet-repair-cont branch from d657e08 to 367fcd3 Compare March 20, 2024 16:30

Michal-Leszczynski added 5 commits March 24, 2024 14:29

feat(makefile): add TABLETS to make start-dev-env

8921402

feat(repair): use RingDescriber per table basis

142d53e

This is required for tablet tables which have per table ring description.

feat(backup): use RingDescriber per table basis

1803fe5

This is required for tablet tables which have per table ring description.

feat(swagger): scylla_v1, add /storage_service/tablets/balancing endp…

a4a2549

…oint

feat(backup): disable tablet migration on snapshot

04298e3

Fixes #3759

Michal-Leszczynski force-pushed the ml/tablet-repair-cont branch 4 times, most recently from 6a3b386 to 27fa6fb Compare March 25, 2024 08:54

Michal-Leszczynski force-pushed the ml/tablet-repair-cont branch from 27fa6fb to e3e0de6 Compare March 25, 2024 09:33

karol-kokoszka approved these changes Mar 27, 2024

View reviewed changes

Michal-Leszczynski mentioned this pull request Mar 29, 2024

[Placeholder] Support repair on tablets enabled environment. #3704

Closed

Michal-Leszczynski merged commit e042a33 into master Apr 3, 2024
51 of 56 checks passed

Michal-Leszczynski deleted the ml/tablet-repair-cont branch April 3, 2024 08:32

karol-kokoszka mentioned this pull request May 9, 2024

Release 3.2.8 #3842

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce tablets to SM #3753

Introduce tablets to SM #3753

Michal-Leszczynski commented Mar 14, 2024 •

edited

Loading

Michal-Leszczynski commented Mar 14, 2024 •

edited

Loading

Michal-Leszczynski commented Mar 14, 2024

Michal-Leszczynski commented Mar 14, 2024

Michal-Leszczynski commented Mar 14, 2024 •

edited

Loading

Michal-Leszczynski commented Mar 14, 2024 •

edited

Loading

Michal-Leszczynski commented Mar 15, 2024

Michal-Leszczynski commented Mar 15, 2024

karol-kokoszka commented Mar 15, 2024

dani-tweig commented Mar 17, 2024

asias commented Mar 18, 2024

asias commented Mar 18, 2024

Michal-Leszczynski commented Mar 18, 2024

Michal-Leszczynski commented Mar 18, 2024

Michal-Leszczynski commented Mar 18, 2024

Michal-Leszczynski commented Mar 18, 2024

Michal-Leszczynski commented Mar 18, 2024

bhalevy commented Mar 18, 2024

Michal-Leszczynski commented Mar 22, 2024

dani-tweig commented Mar 24, 2024

Michal-Leszczynski commented Mar 27, 2024

karol-kokoszka commented Mar 27, 2024

karol-kokoszka left a comment

Michal-Leszczynski commented Apr 3, 2024

Introduce tablets to SM #3753

Introduce tablets to SM #3753

Conversation

Michal-Leszczynski commented Mar 14, 2024 • edited Loading

Michal-Leszczynski commented Mar 14, 2024 • edited Loading

Michal-Leszczynski commented Mar 14, 2024

Michal-Leszczynski commented Mar 14, 2024

Michal-Leszczynski commented Mar 14, 2024 • edited Loading

Michal-Leszczynski commented Mar 14, 2024 • edited Loading

Michal-Leszczynski commented Mar 15, 2024

Michal-Leszczynski commented Mar 15, 2024

karol-kokoszka commented Mar 15, 2024

dani-tweig commented Mar 17, 2024

asias commented Mar 18, 2024

asias commented Mar 18, 2024

Michal-Leszczynski commented Mar 18, 2024

Michal-Leszczynski commented Mar 18, 2024

Michal-Leszczynski commented Mar 18, 2024

Michal-Leszczynski commented Mar 18, 2024

Michal-Leszczynski commented Mar 18, 2024

bhalevy commented Mar 18, 2024

Michal-Leszczynski commented Mar 22, 2024

dani-tweig commented Mar 24, 2024

Michal-Leszczynski commented Mar 27, 2024

karol-kokoszka commented Mar 27, 2024

karol-kokoszka left a comment

Choose a reason for hiding this comment

Michal-Leszczynski commented Apr 3, 2024

Michal-Leszczynski commented Mar 14, 2024 •

edited

Loading

Michal-Leszczynski commented Mar 14, 2024 •

edited

Loading

Michal-Leszczynski commented Mar 14, 2024 •

edited

Loading

Michal-Leszczynski commented Mar 14, 2024 •

edited

Loading