Release v0.7.0

What's Changed

Highlights

• Metric Engine: handles a large number of small tables, making it particularly suitable for cloud-native monitoring.
• Region Migration: enhances the user experience and simplifies region migrations with straightforward SQL commands.
• Inverted Index: dramatically improves the efficiency of locating data segments relevant to user queries, significantly reducing the IO operations needed for scanning data files and thus accelerating the query process.

Features:

• feat: add modulo function by @etolbakov in #3147
• feat: support HTTP&gRPC&pg set timezone by @Taylor-lagrange in #3125
• feat(mito): enable inverted index by @zhongzc in #3158
• feat: upgrade pgwire to 0.19 by @sunng87 in #3157
• feat: let tables API return a stream by @fengjiachun in #3170
• feat: add tests-fuzz crate by @WenyXu in #3173
• feat: tables stream with CatalogManager by @fengjiachun in #3180
• feat: adds date_format function by @killme2008 in #3167
• feat: auto config cache size according to memory size by @QuenKar in #3165
• feat: precise filter for mito parquet reader by @waynexia in #3178
• feat: adds parse options for SQL parser by @killme2008 in #3193
• feat(tests-fuzz): add CreateTableExprGenerator & AlterTableExprGenerator by @WenyXu in #3182
• feat: make procedure able to return output by @WenyXu in #3201
• feat: copy database from by @v0y4g3r in #3164
• feat: introduce information schema provider cache by @WenyXu in #3208
• feat: lazy initialize vector builder on write by @waynexia in #3210
• feat: make query be aware of timezone setting by @killme2008 in #3175
• feat: add create alter table expr translator by @WenyXu in #3203
• feat: read column and region info from state cache by @waynexia in #3222
• feat: enable concurrent write by @WenyXu in #3214
• feat: add Arrow IPC output format for http rest api by @sunng87 in #3177
• feat: change Range Query’s default align behavior aware of timezone by @killme2008 in #3219
• feat: update dashboard to v0.4.7 by @ZonaHex in #3229
• feat: http sql api return schema on empty resultset by @sunng87 in #3237
• feat: add pg create alter table expr translator by @WenyXu in #3206
• feat: read metadata from write cache by @QuenKar in #3224
• feat: batch create ddl by @fengjiachun in #3194
• feat: add insert/select generator & translator by @WenyXu in #3240
• feat: return request outdated error on handling alter by @waynexia in #3239
• feat: don't map semantic type in metric engine by @waynexia in #3243
• feat: create tables in batch on prom write by @fengjiachun in #3246
• feat: Only allow inserts and deletes operations to be executed in parallel by @fengjiachun in #3257
• feat: basic types for Dataflow Framework by @discord9 in #3186
• feat: initial configuration for grafana dashboard by @waynexia in #3263
• feat: support fraction part in timestamp by @discord9 in #3272
• feat: use simple filter to prune memtable by @waynexia in #3269
• feat: Basic Definitions for Expression&Functions for Dataflow by @discord9 in #3267
• feat: support cache for batch_get in CachedMetaKvBackend by @fengys1996 in #3277
• feat: put all filter exprs in a filter plan separately by @waynexia in #3288
• feat: administration functions by @killme2008 in #3236
• feat(mito): adjust seg size of inverted index to finer granularity instead of row group level by @zhongzc in #3289
• feat(mito): add options to ignore building index for specific column ids by @zhongzc in #3295
• feat: organize tracing on query path by @waynexia in #3310
• feat: impl partitions and region_peers information schema by @killme2008 in #3278
• feat: batch get physical table routes by @fengjiachun in #3319
• feat: Defines structs in the merge tree memtable by @evenyag in #3326
• feat(metric-engine): set index options for data region by @zhongzc in #3330
• feat: data buffer and related structs by @v0y4g3r in #3329
• feat: Implement KeyDictBuilder for the merge tree memtable by @evenyag in #3334
• feat: replace pk index with pk_weight during freeze by @v0y4g3r in #3343
• feat: merge tree data parts by @v0y4g3r in #3346
• feat: Defines more structs and methods for a partitioned merge tree by @evenyag in #3348
• feat(flow): impl ScalarExpr&Scalar Function by @discord9 in #3283
• feat: impl migrate_region and procedure_state SQL function by @killme2008 in #3325
• feat: add isnull function by @KKould in #3360
• feat: skip filling NULL for put and delete requests by @waynexia in #3364
• feat: impl merge reader for DataParts by @v0y4g3r in #3361
• feat: Implement write and fork for the new memtable by @evenyag in #3357
• feat: distinguish between different read paths by @v0y4g3r in #3369
• feat: Add freeze and fork method to the memtable by @evenyag in #3374
• feat: merge tree dedup reader by @v0y4g3r in #3375
• feat: Implement iter for the new memtable by @evenyag in #3373
• feat: Implement dedup for the new memtable and expose the config by @evenyag in #3377
• feat: change how region id maps to region worker by @waynexia in #3384
• feat: make tls certificates/keys reloadable (part 1) by @sunng87 in #3335
• feat(grafana): enable shared tooltip, add raft engine throughput by @waynexia in #3387
• feat: Implement partition eviction and only add value size to write buffer size by @evenyag in #3393
• feat: enable zstd compression and encodings in merge tree data part by @v0y4g3r in #3380
• feat(flow): impl for MapFilterProject by @discord9 in #3359
• feat: flush or compact table and region functions by @killme2008 in #3363
• feat: Use a partition level map to look up pk index by @evenyag in #3400
• feat(index): measure memory usage in global instead of single-column and add metrics by @zhongzc in #3383
• feat: enable ArrowFlight compression by @tisonkun in #3403
• feat: zero copy on split rows by @fengjiachun in #3407
• feat: Support automatic DNS lookup for kafka bootstrap servers by @J0HN50N133 in #3379
• feat: add configuration for tls watch option by @sunng87 in #3395
• feat: employ sparse key encoding for shard lookup by @v0y4g3r in #3410
• feat: support Create Table ... Like by @KKould in #3372
• feat: tableref cache by @fengjiachun in #3420
• feat: add verbose support for tql explain/analyze by @etolbakov in #3390
• feat: reduce a clone of string by @fengjiachun in #3422
• feat: Correct server metrics and add more metrics for scan by @evenyag in #3426
• feat: support tracing rule sampler by @Taylor-lagrange in #3405
• feat: decode prom requests to grpc by @v0y4g3r in #3425
• feat: implement multi-dim partition rule by @waynexia in #3409

Breaking Changes

• feat!: switch prom remote write to metric engine by @waynexia in #3198
• refactor!: rename initialize_region_in_background to init_regions_in_background by @WenyXu in #3216
• feat!: new partition grammar - parser part by @waynexia in #3347

Fixes

• fix: change back GREPTIME_DB_HEADER_NAME header key name by @shuiyisong in #3184
• fix(index): S3 EntityTooSmall error by @zhongzc in #3192
• fix: remove name matcher from processed matcher list by @waynexia in #3213
• fix: fix default value cannot accept negative number by @WenyXu in #3217
• fix: only register region keeper while creating physical table by @WenyXu in #3223
• fix: fix MockInstance rebuild issue by @WenyXu in #3218
• fix: security update for shlex and h2 by @sunng87 in #3227
• fix: fix create table ddl return incorrect table id by @WenyXu in #3232
• fix: init parquet reader metrics twice by @evenyag in #3242
• fix: IntermediateWriter closes underlying writer twice by @waynexia in #3248
• fix: decouple columns in projection and prune by @waynexia in #3253
• fix(Copy From): fix incorrect type casts by @WenyXu in #3264
• fix: SQL insertion and column default constraint aware of timezone by @killme2008 in #3266
• fix: cli export database default value by @tisonkun in #3259
• fix: use fe_opts after setup_frontend_plugins in standalone by @shuiyisong in #3275
• fix: fix incorrect StatusCode parsing by @WenyXu in #3281
• fix(util): join_path function should not trim leading / by @dalprahcd in #3280
• fix: correct the case sensitivity behavior for PromQL by @waynexia in #3296
• fix: $TARGET_BIN not found when docker run the image by @daviderli614 in #3297
• fix(index): sanitize S3 upload buffer size by @zhongzc in #3300
• fix: commit_short sqlness test case by @tisonkun in #3313
• fix: bump libgit2-sys from 0.16.1+1.7.1 to 0.16.2+1.7.2 by @dependabot in #3316
• fix: split write metadata request by @fengjiachun in #3311
• fix(index): encode string type to original data to enable fst regex to work by @zhongzc in #3324
• fix: disable ansi contorl char when stdout is redirected by @waynexia in #3332
• fix: typo in lint config by @waynexia in #3358
• fix: treat "0" and "1" as valid boolean values. by @MichaelScofield in #3370
• fix: remove unused imports in memtable_util.rs by @v0y4g3r in #3376
• fix: resets dict builder keys counter and avoid unnecessary pruning by @evenyag in #3386
• fix: throw errors instead of panic by @WenyXu in #3391
• fix: some read metrics by @v0y4g3r in #3404
• fix: partition region id by @fengjiachun in #3414
• fix: show table names not complete from information_schema by @killme2008 in #3417
• fix: mitigate memory spike during startup by @niebayes in #3418
• fix: complete interceptors for all frontend entry by @shuiyisong in #3428

New Contributors

• @dalprahcd made their first contribution in #3280
• @caicancai made their first contribution in #3299
• @SteveLauC made their first contribution in #3352
• @KKould made their first contribution in #3360
• @J0HN50N133 made their first contribution in #3379

Full Changelog: v0.6.0...v0.7.0