DataParser fixes/improvements. (#45)

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 Represents the **NuGet** versions.
 
+## v2.3.11
+- *Enhancement:* `DataParser.ParseJsonAsync` added to support JSON data file parsing in addition to the existing YAML. Files (embedded resources) can be mixed and matched as required.
+- *Enhancement:* `DataParser` supports specified schema of `*` to provide `DataConfig` that is applied to all tables in the YAML/JSON file.
+- *Enhancement:* `DataParser` supports hierarchical data (i.e parent/child) where the child row(s) are specified as a parent column value (must be an array). Will also attempt to update the child related column from the parent primary key where not specified to simplify specification.
+- *Fixed:* `DataParser` supports `bit` values of `0` and `1`, in addition to `false` and `true` respectively.
+- *Fixed:* `DataParser` constraint of table only being specified once in the YAML/JSON file has been removed; this is now supported. The underlying database insert or merge operations will be consolidated and executed in the sequence of initial specification.
+
 ## v2.3.10
 - *Fixed:* `DbTableSchema`, `DbColumnSchema` and `DataParserArgs` now correctly support the full range of by-convention properties (e.g. `RowVersion`, `TenantId` and `IsDeleted`). Both the `SqlServerSchemaConfig` and `MySqlSchemaConfig` updated to default names as appropriate. Additional properties added to `Db*Schema` to support additional code-generation scenarios, etc.
 

Common.targets

@@ -1,6 +1,6 @@
 <Project>
   <PropertyGroup>
-    <Version>2.3.10</Version>
+    <Version>2.3.11</Version>
     <LangVersion>preview</LangVersion>
     <Authors>Avanade</Authors>
     <Company>Avanade</Company>

README.md

@@ -118,14 +118,14 @@ The `Schema` folder is used to encourage the usage of database schemas. Therefor
 
 ### Data
 
-Data can be defined using [YAML](https://en.wikipedia.org/wiki/YAML) to enable simplified configuration that will be used to generate the required SQL statements to apply to the database.
+Data can be defined using [YAML](https://en.wikipedia.org/wiki/YAML) or JSON to enable simplified configuration that will be used to generate the required SQL statements to apply to the database.
 
 The data specified follows a basic indenting/levelling rule to enable:
 1. **Schema** - specifies Schema name.
 2. **Table** - specifies the Table name within the Schema; this will be validated to ensure it exists within the database as the underlying table schema (columns) will be inferred. The underyling rows will be [inserted](https://docs.microsoft.com/en-us/sql/t-sql/statements/insert-transact-sql) by default; or alternatively by prefixing with a `$` character a [merge](https://docs.microsoft.com/en-us/sql/t-sql/statements/merge-transact-sql) operation will be performed instead.
 3. **Rows** - each row specifies the column name and the corresponding values (except for reference data described below). The tooling will parse each column value according to the underying SQL type.
 
-Finally, SQL script files can also be provided in addition to YAML where explicit SQL is to be executed.
+Additionally, SQL script files can also be provided in addition to YAML and JSON where explicit SQL is to be executed.
 
 <br/>
 
@@ -139,7 +139,7 @@ Alternatively, a *Reference Data* reference could be the code itself, typically
 
 <br/>
 
-#### YAML configuration
+#### YAML/JSON configuration
 
 Example YAML configuration for *merging* reference data is as follows.
 
@@ -175,7 +175,7 @@ Demo:
     - { FirstName: Wendy, LastName: Jones, Gender: F, Birthday: 1985-03-18 }
 ```
 
-Finally, runtime values can be used within the YAML using the value lookup notation; this notation is `^(Key)`. This will either reference the [`DataParserArgs`](./src/DbEx/Migration/Data/DataParserArgs.cs) `Parameters` property using the specified key. There are two special parameters, being `UserName` and `DateTimeNow`, that reference the same named `DataParserArgs` properties. Where not found the extended notation `^(Namespace.Type.Property.Method().etc, AssemblyName)` is used. Where the `AssemblyName` is not specified then the default `mscorlib` is assumed. The `System` root namespace is optional, i.e. it will be attempted by default. The initial property or method for a `Type` must be `static`, in that the `Type` will not be instantiated. Example as follows. These parameters (`Name=Value` pairs) can also be command-line specified.
+Runtime values can be used within the YAML using the value lookup notation; this notation is `^(Key)`. This will either reference the [`DataParserArgs`](./src/DbEx/Migration/Data/DataParserArgs.cs) `Parameters` property using the specified key. There are two special parameters, being `UserName` and `DateTimeNow`, that reference the same named `DataParserArgs` properties. Where not found the extended notation `^(Namespace.Type.Property.Method().etc, AssemblyName)` is used. Where the `AssemblyName` is not specified then the default `mscorlib` is assumed. The `System` root namespace is optional, i.e. it will be attempted by default. The initial property or method for a `Type` must be `static`, in that the `Type` will not be instantiated. Example as follows. These parameters (`Name=Value` pairs) can also be command-line specified.
 
 ``` yaml
 Demo:
@@ -184,6 +184,9 @@ Demo:
     - { FirstName: Wendy, Username: ^(Beef.ExecutionContext.EnvironmentUsername,Beef.Core), Birthday: ^(DateTime.UtcNow) }
 ```
 
+Advanced capabilities, such as nested YAML/JSON can be specified to represent hierarchical relationships (see `contact->addresses` within test [`data.yaml`](./tests/DbEx.Test.Console/Data/Data.yaml) and related [`TableNameMappings`](./tests/DbEx.Test.Console/Program.cs) to map to the correct underlying database table). [`DataConfig`](./src/Dbex/Migration/Data/DataConfig.cs) can also be specified using the `*` schema to control the behaviour within the context of a YAML/JSON file as demonstrated by the test [`ContactType.json`](./tests/DbEx.Test.Console/Data/ContactType.json).
+
+
 <br/>
 
 ### Console application

src/DbEx.MySql/MySqlSchemaConfig.cs

@@ -214,7 +214,7 @@ public override string ToFormattedSqlType(DbColumnSchema schema, bool includeNul
         }
 
         /// <inheritdoc/>
-        public override string ToFormattedSqlStatementValue(DataParserArgs dataParserArgs, object? value) => value switch
+        public override string ToFormattedSqlStatementValue(DbColumnSchema dbColumnSchema, DataParserArgs dataParserArgs, object? value) => value switch
         {
             null => "NULL",
             string str => $"'{str.Replace("'", "''", StringComparison.Ordinal)}'",

src/DbEx.MySql/Resources/DatabaseData_sql.hbs

@@ -1,4 +1,8 @@
 {{! Copyright (c) Avanade. Licensed under the MIT License. See https://github.com/Avanade/DbEx }}
+{{#if PreSql}}
+{{PreSql}}
+
+{{/if}}
 {{#if IsMerge}}
   {{#each Rows}}
 INSERT INTO `{{Table.Name}}` ({{#each MergeInsertColumns}}`{{Name}}`{{#unless @last}}, {{/unless}}{{/each}}) VALUES ({{#each MergeInsertColumns}}{{#if UseForeignKeyQueryForId}}(SELECT `{{DbColumn.ForeignColumn}}` FROM `{{DbColumn.ForeignTable}}` WHERE `{{DbColumn.ForeignRefDataCodeColumn}}` = {{{SqlValue}}} LIMIT 1){{else}}{{{SqlValue}}}{{/if}}{{#unless @last}}, {{/unless}}{{/each}}) ON DUPLICATE KEY UPDATE {{#each MergeUpdateColumns}}`{{Name}}` = {{{SqlValue}}}{{#unless @last}}, {{/unless}}{{/each}};
@@ -9,4 +13,8 @@ SELECT {{Rows.Count}}; -- Total rows upserted
 INSERT INTO `{{Table.Name}}` ({{#each InsertColumns}}`{{Name}}`{{#unless @last}}, {{/unless}}{{/each}}) VALUES ({{#each InsertColumns}}{{#if UseForeignKeyQueryForId}}(SELECT `{{DbColumn.ForeignColumn}}` FROM `{{DbColumn.ForeignTable}}` WHERE `{{DbColumn.ForeignRefDataCodeColumn}}` = {{{SqlValue}}} LIMIT 1){{else}}{{{SqlValue}}}{{/if}}{{#unless @last}}, {{/unless}}{{/each}});
   {{/each}}
 SELECT {{Rows.Count}}; -- Total rows inserted
+{{/if}}
+{{#if PostSql}}
+
+{{PostSql}}
 {{/if}}

src/DbEx.SqlServer/Resources/DatabaseData_sql.hbs

@@ -1,4 +1,8 @@
 {{! Copyright (c) Avanade. Licensed under the MIT License. See https://github.com/Avanade/DbEx }}
+{{#if PreSql}}
+{{PreSql}}
+
+{{/if}}
 {{#if IsMerge}}
 CREATE TABLE #temp (
   {{#each Columns}}
@@ -28,4 +32,8 @@ DROP TABLE #temp
 INSERT INTO [{{Table.Schema}}].[{{Table.Name}}] ({{#each InsertColumns}}[{{Name}}]{{#unless @last}}, {{/unless}}{{/each}}) VALUES ({{#each InsertColumns}}{{#if UseForeignKeyQueryForId}}(SELECT TOP 1 [{{DbColumn.ForeignColumn}}] FROM [{{DbColumn.ForeignSchema}}].[{{DbColumn.ForeignTable}}] WHERE [{{DbColumn.ForeignRefDataCodeColumn}}] = {{{SqlValue}}}){{else}}{{{SqlValue}}}{{/if}}{{#unless @last}}, {{/unless}}{{/each}})
   {{/each}}
 SELECT {{Rows.Count}} -- Total rows inserted
+{{/if}}
+{{#if PostSql}}
+
+{{PostSql}}
 {{/if}}

src/DbEx.SqlServer/SqlServerSchemaConfig.cs

@@ -233,7 +233,7 @@ public override string ToFormattedSqlType(DbColumnSchema schema, bool includeNul
         }
 
         /// <inheritdoc/>
-        public override string ToFormattedSqlStatementValue(DataParserArgs dataParserArgs, object? value) => value switch
+        public override string ToFormattedSqlStatementValue(DbColumnSchema dbColumnSchema, DataParserArgs dataParserArgs, object? value) => value switch
         {
             null => "NULL",
             string str => $"N'{str.Replace("'", "''", StringComparison.Ordinal)}'",

src/DbEx/DatabaseSchemaConfig.cs

@@ -148,10 +148,11 @@ protected DatabaseSchemaConfig(string databaseName, bool supportsSchema = true)
         /// <summary>
         /// Gets the formatted SQL statement representation of the <paramref name="value"/>.
         /// </summary>
+        /// <param name="dbColumnSchema">The <see cref="DbColumnSchema"/>.</param>
         /// <param name="dataParserArgs">The <see cref="DataParserArgs"/>.</param>
         /// <param name="value">The value.</param>
         /// <returns>The formatted SQL statement representation.</returns>
-        public abstract string ToFormattedSqlStatementValue(DataParserArgs dataParserArgs, object? value);
+        public abstract string ToFormattedSqlStatementValue(DbColumnSchema dbColumnSchema, DataParserArgs dataParserArgs, object? value);
 
         /// <summary>
         /// Indicates whether the <paramref name="dbType"/> is considered an integer.

src/DbEx/DbSchema/DbTableSchema.cs

@@ -67,7 +67,7 @@ public static string CreateDotNetName(string name, bool removeKnownSuffix = fals
         }
 
         /// <summary>
-        /// Create a plural from the name.
+        /// Create a plural from the singular name.
         /// </summary>
         /// <param name="name">The name.</param>
         /// <returns>The pluralized name.</returns>
@@ -81,6 +81,21 @@ public static string CreatePluralName(string name)
             return string.Join(string.Empty, words);
         }
 
+        /// <summary>
+        /// Create a singular from the pluralized name.
+        /// </summary>
+        /// <param name="name">The name.</param>
+        /// <returns>The singular name.</returns>
+        public static string CreateSingularName(string name)
+        {
+            if (string.IsNullOrEmpty(name))
+                throw new ArgumentNullException(nameof(name));
+
+            var words = Regex.Split(name, WordSplitPattern).Where(x => !string.IsNullOrEmpty(x)).ToList();
+            words[^1] = StringConverter.ToSingle(words[^1]);
+            return string.Join(string.Empty, words);
+        }
+
         /// <summary>
         /// Initializes a new instance of the <see cref="DbTableSchema"/> class.
         /// </summary>

src/DbEx/Migration/Data/DataColumn.cs

@@ -19,6 +19,10 @@ internal DataColumn(DataTable table, string name)
         {
             Table = table ?? throw new ArgumentNullException(nameof(table));
             Name = name ?? throw new ArgumentNullException(nameof(name));
+
+            // Map the column name where specified.
+            if (table.ColumnNameMappings is not null && table.ColumnNameMappings.TryGetValue(name, out var mappedName))
+                Name = mappedName;
         }
 
         /// <summary>
@@ -50,6 +54,6 @@ internal DataColumn(DataTable table, string name)
         /// Gets the value formatted for use in a SQL statement.
         /// </summary>
         /// <returns>The value formatted for use in a SQL statement.</returns>
-        public string SqlValue => Table.DbTable.Config.ToFormattedSqlStatementValue(Table.Args, Value);
+        public string SqlValue => Table.DbTable.Config.ToFormattedSqlStatementValue(DbColumn ?? throw new InvalidOperationException("The DbColumn property must not be null."), Table.Args, Value);
     }
 }

src/DbEx/Migration/Data/DataConfig.cs

@@ -0,0 +1,25 @@
+// Copyright (c) Avanade. Licensed under the MIT License. See https://github.com/Avanade/DbEx
+
+namespace DbEx.Migration.Data
+{
+    /// <summary>
+    /// Respresents the data configuration.
+    /// </summary>
+    public class DataConfig
+    {
+        /// <summary>
+        /// Gets or sets the pre-condition SQL.
+        /// </summary>
+        public string? PreConditionSql { get; set; }
+
+        /// <summary>
+        /// Gets or sets the pre-SQL.
+        /// </summary>
+        public string? PreSql { get; set; }
+
+        /// <summary>
+        /// Gets or sets the post-SQL.
+        /// </summary>
+        public string? PostSql { get; set; }
+    }
+}