0.23.8

This release includes an alpha version of stored procedure support, and fixes several performance and correctness issues related to merge and diff of large databases.

Merged PRs

• 1377: Upgraded go-mysql-server and unskipped many tests that now pass
• 1376: fix overflow issue
  Merges were failing when the changes were sufficiently large. This was really exacerbated by merges where the common ancestor is extremely old. getCompositeChildSequence would try to allocate contiguous blocks and encode the entire composite child sequence into it. If the size of the data exceeded the max value of a uint32 then the indexes being used to access the buffers would overflow and wrap around and cause all kinds of badness.
• 1375: Fix Schema Unmarshalling Error
  {Map, List, Set}.IterAll() spins up multiple go routines to parallelize the IO of loading a data structure, but when the data is small (unmarshalling Schema/SuperSchema) it seems to be causing issues. Changing these instances of IterAll() to Iter() fixes context cancelled errors seen during dolt merge. The errors have not yet been reproduced in a test case, but they have been fixed in the affected repos.
• 1372: fix docs links in readme
• 1371: improve conflict resolution error messages
  resolves #1366
• 321: Bug fix for parallelization
  We have been parallelizing Filter and Project nodes, but this is a problem for such nodes that contain a Subquery expression, since the query plan for that subquery may not tolerate parallelization. This fixes that bug and adds tests.
• 315: Added SHOW & DROP PROCEDURE, along with basic IF/ELSE support

0.23.7

This release is the first implementation of SQL Window Functions with ROW_NUMBER() being the first supported Window Function. More to come.

Other than that bug fixes and performance improvements on SQL JOINs and Dolt Merges.

Merged PRs

• 1369: disable foreign key checks on merge with warning
  Foreign key constraint violations on merge had the potential to make resolving conflicts between two commits impossible. Merges are now done without foreign key constraint validation. A new warning message was added when there is potential for this to be an issue. Foreign key constraint validation is enforced on commit.
• 1367: fix infinite loop when allocating encoding buffers larger than 2GBi
• 1365: Speed improvement. No need to flush() every row during a merge.
• 314: sql/analyzer/prune_columns.go: Turn off pruneColumns when a Subquery expression exists.
• 311: sql/analyzer/resolve_subqueries.go: Make subqueries which include subqueryaliases cacheable.
• 310: sql/plan/subquery.go: Make certain that analyzed nodes below a subquery that includes a subquery alias node see the row results they are expecting.
• 309: Follow specific rules for mysql float/double/error depending on the float ops.
• 308: Window functions
  This is a full implementation of window functions, with only the ROW_NUMBER() function supported. Partitions and order-by work for the OVER() clause, but other constructs (PRECEDING, FOLLOWING, etc.) are not supported.
  Other limitations:
  • Can't use window functions with GROUP BY
  • Can use non-window aggregates with window functions, but only if they are declared with an empty OVER clause

0.23.6

This release contains major performance improvements to dolt commit

Merged PRs

• 1361: save active merge commit during GC
• 1360: Diff based foreign key validation
  Previous validation implementation did full scans for every foreign key constraint. New implementation only validates against rows that have changed from the previous commit.
• 1359: err handling fix

0.23.5

This release contains support for CREATE, USE, and DROP database statements. The databases created are in memory and session local. These statements are supported for compliance with the SQL specification to make Dolt work with certain tools. Dolt natively does not support multiple databases.

This release contains major performance improvements to JOIN statements.

Merged PRs

• 1329: CREATE DATABASE (DBDDL)
  This pr brings CREATE DATABASE functionality to dolt and adds associated test cases.
• 1328: Feature version support.
  FeatureVersion is now checked every time a RootVal is read, and is written with every call to doltdb.WriteRootValue().
  A CLI flag was added to dolt.go to enable FeatureVersion testing.
• 305: sql/analyzer: resolve_subqueries.go: Add a missing stripQueryProcess in resolveSubqueries.
• 303: sql/analyzer: Make join_search faster
• 301: Naive/hacky show grants
  hard-coded SHOW GRANTS handling:

  > dolt sql -q "show grants for current_user"
  +-------------------------------------------------------------+
  | Grants for root@%                                            |
  +-------------------------------------------------------------+
  | GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION |
  +-------------------------------------------------------------+
  

• 300: Added Stored Procedure interface for integrators
• 299: sql/{plan,analyzer}: Add IndexedInSubqueryFilter for selecting the subquery results first and then making indexed lookups into the child.
• 298: Add functionality for CREATE DATABASE
• 297: Append error codes/sql state to error messages for downstream clients.
• 48: Add the constants
  Updates our vitess with a bunch of constants
• 47: add support for if exists and if not exists for create database
• 46: Added CREATE PROCEDURE & FUNCTION
  This adds:

  CREATE PROCEDURE
  SHOW PROCEDURE STATUS
  SHOW FUNCTION STATUS
  DROP PROCEDURE
  CALL
  

  The SHOW...CODE statements are being skipped for now, as they're relatively unused compared to the above statements. Additionally, the CREATE statements do not have a fully fleshed-out routine body, as we currently do not parse all valid statements anyway.
• 45: Preliminary vitess support for window funcs.

Closed Issues

• 1324: panic running great expectations unittests

0.23.2

This release supports ALTER TABLE MODIFY COLUMN statements.

Merged PRs

• 1321: disable query diff
  query diff was a little known feature that was not well documented and was the source of a large amount of maintenance work. We can recover this from git in the future, or reimplement once it is a priority.
• 1317: Removed extraneous diff.go file
• 1312: bats tests for primary key column change diff panic
• 1309: added two skipped bats
• 1306: Check constraints on commit
• 1299: Vinai/dolt merge Part 1. Just Fast Forward Merge
  This pr starts the Dolt_Merge process that works with a fast forward.
• 1290: Added DELIMITER support to the dolt shell
• 1281: ALTER TABLE MODIFY COLUMN
  Adds support for changing column types.

Closed Issues

• 1313: Dolt diff panics if you rename a primary key using ALTER

0.23.0

This release introduces several improvements to the query analyzer:

• Tables can now be joined to subqueries using indexes
• Indexes and other optimizations now work in INSERT ... SELECT statements
• Null-safe equals operator (<=>)

Merged PRs

• 1303: go/go.mod: Bump go-mysql-server
• 295: Changed analysis to isolate the values side of an INSERT statement
  Also cleaned up create trigger analysis, which broke when I did this at first. Now catches more errors than before.
• 294: Fixed a bug in join planning for Inserts.
  Table reordering was leaving nodes above the join with incorrect field indexes. This was getting fixed by other analyzer steps for some top-level nodes, but not Inserts.
• 293: Implement NullSafeEquals: The <=> operator in MySQL.
• 292: Make join planning work for insert statements
• 291: Consider SubqueryAlias nodes when planning indexed joins.

0.22.14

This release addresses correctness bugs and provides performance improvements, as well as continuing to flesh Dolt's version control features as SQL functions.

Merged PRs

• 1298: change table projection implementation
• 1293: Insertion optimizations
  Go from sql rows to types.Tuples directly and use those tuples for insertion without using row.Row.
• 1289: Added DOLT_SQL_DEBUG_LOG and DOLT_SQL_DEBUG_LOG_VERBOSE environment vars
  Used to turn on query analyzer debugging
• 1287: Implement SQL dolt_checkout function
  This pr adds DOLT_CHECKOUT functionality.
• 290: pushdown to indexed tables
• 288: remove slow span tag
• 287: Update README.md to reference Dolt
• 286: Fixed several bugs preventing indexes from being used in some joins
• 285: Implemented JOIN_ORDER optimizer hints
  Also:
  • Got rid of expensive comment stripping pass in the parser
  • Fixed test behavior of MySQL executed comment statements like /*!40101 SET NAMES utf8 */
  • Made SHOW VARIABLES output sorted
• 284: Updated copyright headers and added missing ones
• 283: Fixed type bugs

Closed Issues

0.22.13

This release addresses a number of correctness bugs and improves query performance.

Merged PRs

• 1285: Upgraded to latest go-mysql-server, and added a skip for a new query plan test
• 1284: fix projected columns with single partition indexes
• 1280: o/libraries/doltcore/table/editor: Add ability to configure map editor flush interval with environment variables.
• 1269: Implement StatisticsTable interface to provide table statistics to the analyzer.
• 1267: Fix bug where batched inserts containing subqueires chose non flushing option..
• 1264: adds ability to redirect stdin from a file
• 282: Bug fix for pushing a projection down to a table in a subquery more than once
• 278: remove sync.Once from autoincrement expression
• 275: add StatisticsTable interface
• 274: Fixed various bugs in subquery execution, added table ordering optimization
  Fixes many correctness issues in subquery execution:
  • Incorrect field indexes for subquery aliases in some cases
  • Incorrect field indexes for subquery expressions in the case of column pruning in other parts of the query
  • Queries selecting the same table more than once with aliases getting incorrect indexes applied
    Also introduces performance enhancements:
  • Table ordering optimizations based on rows counts of tables in a join
  • Better use of index pushdown for some join queries
• 273: Added the VALUES() function
  Requested in: #1225
  VALUES() is deprecated in the latest versions of MySQL (as of version 8.0.20, released April 2020), but it is recent enough that I feel its inclusion in the engine is justified.

Closed Issues

• 281: go get github.com/dolthub/go-mysql-server got the wrong path?
• 279: err:go get github.com/dolthub/go-mysql-server

0.22.12

This release has two notable features:

1. Binary type support. BINARY, VARBINARY, TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB now supported.
2. dolt diff --cached now supported

Other than that, this release contains performance improvements and bug fixes.

Merged PRs

• 1252: go/store/nbs: s3_table_reader: Parallelize loading large table file indexes.
• 1251: Implement dolt diff --cached command
  Made this competing PR with the open source contributed so I could commit bats test changes in conjunction.
• 1249: corrected sql row for init commit row in dolt_commit_ancestors table
• 1245: fixes panic in the atomic package
  The atomic package has the following limitation on 32-bit platforms:

  On both ARM and x86-32, it is the caller's responsibility to arrange for 64-bit alignment of 64-bit words accessed atomically. The first word in a variable or in an allocated struct, array, or slice can be relied upon to be 64-bit aligned.

• 1242: projected column fixes
  For both indexed and unindexed tables when the engine would call WithProjection on our sql.Table implementation we would return a new sql.Table that embedded the old one and had a member which was the list of projected columns. Then, when sql.Table.PartitionRows was called, it was called on the embedded object directly and the projected columns were not accessible.
• 1240: Skip And Less Optimizations
• 1239: Decrease the number of allocations for indexed reads
• 1237: Fixed a grammatical mistake in the man page for dolt conflicts resolve.
• 1233: fix panic when key does not exist in table
• 1230: change panic to error for '.' in Dataset name
  fix for #1144
• 1229: added sysbench scripts
• 1228: go/store/nbs: WithoutConjoiner() to configure NBS to not conjoin.
• 1221: cli output fix
  fixes issues where printf formatting was being unintentionally applied for some cli output
• 1220: Print pipeline fix
  Fixes issue #1219 by not calling on an iterator after it's already returned EOF
• 1218: remotestorage: Fix download aggregation to correctly aggregated based on prior chunk in batch, not first chunk.
• 1217: Restored proper DATE functionality
  For DATE, we accidentally forgot to add the time truncation during the perf improvements, so this adds it back in.
• 1215: Write to root on every loop of sql shell.
  This pr fixes a bug where the root gets written on every loop of the dolt sql shell.
• 1214: Automated release notes generation
• 1213: Read tuples from sequences directly without conversion to Values
  Read tuples from sequences directly instead of using values.
• 1212: decrease allocations made while reading values
• 1210: Vinai/refactor docs
  This pr refactors DocsReadWriter to simplify the interface. It then removes many of the methods in environment.go that are related to the d
  rw interface.
• 1209: Added binary types to dolt
  Added BINARY, VARBINARY, TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB types.
• 271: add ability to remove rules
• 269: Fix: not check tcp6 socket state while ipv6 is disabled
  If ipv6 is disabled on the system, tcp6 will not exist in the /proc/net dir.
  So open /proc/net/tcp6 will produce the error open /proc/net/tcp6: No such file or directory, we should not always check tcp6 socket state(
  unless it is opened).
• 268: Bug fix for #1219

Closed Issues

• 1219: show indexes crash
• 1211: Refactor Docs methods out of environment.go

0.22.11

This is a bug fix release for 0.22.10. This fixes a bug in clone that would sometimes keep large clones from succeeding by not retrying failed downloads.

Merged PRs

• 1208: go/store/datas: pull.go: Fix Sources refetch on clone download failure to actually try with the newly fetched URLs.