Fix issue #7. (#8)

* Fix issue #7. * Fix compile error on github.
Avanade · Jan 10, 2022 · 0646cbf · 0646cbf
1 parent 8cecb13
commit 0646cbf
Show file tree

Hide file tree

Showing 11 changed files with 133 additions and 58 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,5 +2,8 @@
 
 Represents the **NuGet** versions.
 
+## v1.0.2
+- *Fixed:* [Issue 7](https://github.com/Avanade/DbEx/issues/7) fixed. Documentation corrected and support for SQL files for both `Data` and `Execute` commands added.
+
 ## v1.0.1
 - *New:* Initial publish to GitHub/NuGet. This was originally harvested from, and will replace, the core database tooling within [Beef](https://github.com/Avanade/Beef/tree/master/tools/Beef.Database.Core).
diff --git a/README.md b/README.md
@@ -39,7 +39,7 @@ DbEx.Test.Console git:(main)> dotnet run -- -cv cs all
 
 Next, create your own console app, follow the structure of `DbEx.Test.Console` project, add reference to https://www.nuget.org/packages/DbEx and your SQL scripts.
 
-Currently, the easiest way of generating scripts from an existing database, is to use the `Generate Scripts` feature of SQL Server Management Studio and copy its output.
+Currently, the easiest way of generating scripts from an existing database, is to use the `Generate Scripts` feature of _SQL Server Management Studio_ and copy its output.
 
 <br/>
 
@@ -54,7 +54,7 @@ Command | Description
 [`Migrate`](#Migrate) | Being the upgrading of a database overtime using order-based migration scripts; the tool leverages the philosophy and NuGet packages of [DbUp](https://dbup.readthedocs.io/en/latest/philosophy-behind-dbup/) to enable.
 [`Schema`](#Schema) | There are a number of database schema objects that can be managed outside of the above migrations, that are dropped and (re-)applied to the database using their native `Create` statement.
 `Reset` | Resets the database by deleting all existing data (excludes `dbo` and `cdc` schema).
-[`Data`](#Data) | There is data, for example *Reference Data* that needs to be applied to a database. This provides a simpler configuration than specifying the required SQL statements directly. This is _also_ useful for setting up Master and Transaction data for the likes of testing scenarios.
+[`Data`](#Data) | There is data, for example *Reference Data* that needs to be applied to a database. This provides a simpler configuration than specifying the required SQL statements directly (which is also supported). This is _also_ useful for setting up Master and Transaction data for the likes of testing scenarios.
 
 Additional commands available are:
 
@@ -110,6 +110,8 @@ The data specified follows a basic indenting/levelling rule to enable:
 2. **Table** - specifies the Table name within the Schema; this will be validated to ensure it exists within the database as the underlying table schema (columns) will be inferred. The underyling rows will be [inserted](https://docs.microsoft.com/en-us/sql/t-sql/statements/insert-transact-sql) by default; or alternatively by prefixing with a `$` character a [merge](https://docs.microsoft.com/en-us/sql/t-sql/statements/merge-transact-sql) operation will be performed instead.
 3. **Rows** - each row specifies the column name and the corresponding values (except for reference data described below). The tooling will parse each column value according to the underying SQL type.
 
+Finally, SQL script files can also be provided in addition to YAML where explicit SQL is to be executed.
+
 <br/>
 
 #### Reference data
@@ -212,6 +214,16 @@ namespace DbEx.Test.Console
 }
 ```
 
+_Tip:_ To ensure all files are included as embedded resources add the following to the .NET project:
+
+``` xml
+<ItemGroup>
+  <EmbeddedResource Include="Schema\**\*" />
+  <EmbeddedResource Include="Migrations\**\*" />
+  <EmbeddedResource Include="Data\**\*" />
+</ItemGroup>
+```
+
 <br/>
 
 #### Script command
@@ -233,21 +245,25 @@ Sub-command | Argument(s) | Description
 Examples as follows.
 
 ```
-dotnet run scriptnew
-dotnet run scriptnew create Foo Bar
-dotnet run scriptnew refdata Foo Gender
-dotnet run scriptnew alter Foo Bar
-dotnet run scriptnew cdcdb
-dotnet run scriptnew cdc Foo Bar
+dotnet run script
+dotent run script schema Foo
+dotnet run script create Foo Bar
+dotnet run script refdata Foo Gender
+dotnet run script alter Foo Bar
+dotnet run script cdcdb
+dotnet run script cdc Foo Bar
 ```
 
 #### Execute command
 
-The execute command allows one or more SQL Statements to be executed directly against the database. This is intended for enabling commands to be executed only. No response other than success or failure will be acknowledged; as such this is not intended for performing queries.
+The execute command allows one or more SQL Statements, and/or Script files, to be executed directly against the database. This is intended for enabling commands to be executed only. No response other than success or failure will be acknowledged; as such this is not intended for performing queries.
 
 Examples as follows.
 
+```
 dotnet run execute "create schema [Xyz] authorization [dbo]"
+dotnet run execute ./schema/createscehma.sql
+```
 
 <br/>
 
@@ -287,6 +303,14 @@ The [`Database`](./src/DbEx/Database.cs) class provides a `SelectSchemaAsync` me
 
 <br/>
 
+## Other repos
+
+These other _Avanade_ repositories leverage _DbEx_:
+- [NTangle](https://github.com/Avanade/NTangle) - Change Data Capture (CDC) code generation tool and runtime.
+- [Beef](https://github.com/Avanade/Beef) - Business Entity Execution Framework to enable industralisation of API development.
+
+<br/>
+
 ## License
 
 _DbEx_ is open source under the [MIT license](./LICENSE) and is free for commercial use.

diff --git a/src/DbEx/DbEx.csproj b/src/DbEx/DbEx.csproj
@@ -3,7 +3,7 @@
   <PropertyGroup>
     <TargetFramework>netcoreapp3.1</TargetFramework>
     <RootNamespace>DbEx</RootNamespace>
-    <Version>1.0.1</Version>
+    <Version>1.0.2</Version>
     <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
     <Authors>DbEx Developers</Authors>
     <Company>Avanade</Company>

diff --git a/src/DbEx/Migration/DatabaseMigratorBase.cs b/src/DbEx/Migration/DatabaseMigratorBase.cs
@@ -11,7 +11,6 @@
 using System.IO;
 using System.Linq;
 using System.Reflection;
-using System.Text;
 using System.Threading.Tasks;
 
 namespace DbEx.Migration
@@ -223,8 +222,9 @@ protected async Task<bool> CommandExecuteAsync(string title, Func<Task<bool>> ac
         /// Execute the <paramref name="scripts"/>.
         /// </summary>
         /// <param name="scripts">The <see cref="SqlScript"/> list.</param>
+        /// <param name="includeExecutionLogging">Indicates whether to include detailed execution logging.</param>
         /// <returns>The <see cref="DatabaseUpgradeResult"/>.</returns>
-        public abstract Task<DatabaseUpgradeResult> ExecuteScriptsAsync(IEnumerable<SqlScript> scripts);
+        public abstract Task<DatabaseUpgradeResult> ExecuteScriptsAsync(IEnumerable<SqlScript> scripts, bool includeExecutionLogging);
 
         /// <summary>
         /// Check the <see cref="DatabaseUpgradeResult"/> and report error where not <see cref="DatabaseUpgradeResult.Successful"/>.
@@ -269,7 +269,7 @@ private async Task<bool> DatabaseMigrateAsync()
             var scripts = new List<SqlScript>();
             foreach (var ass in Assemblies)
             {
-                foreach (var name in ass.GetManifestResourceNames().Where(rn => Namespaces.Any(ns => rn.StartsWith($"{ns}.{MigrationsNamespace}.", StringComparison.InvariantCulture))))
+                foreach (var name in ass.GetManifestResourceNames().Where(rn => Namespaces.Any(ns => rn.StartsWith($"{ns}.{MigrationsNamespace}.", StringComparison.InvariantCulture))).OrderBy(x => x))
                 {
                     // Determine run order and add script to list.
                     var order = name.EndsWith(".pre.deploy.sql", StringComparison.InvariantCultureIgnoreCase) ? 1 :
@@ -287,7 +287,7 @@ private async Task<bool> DatabaseMigrateAsync()
             }
 
             Logger.LogInformation("  Migrate (using DbUp) the embedded resources...");
-            return CheckDatabaseUpgradeResult(await ExecuteScriptsAsync(scripts).ConfigureAwait(false));
+            return CheckDatabaseUpgradeResult(await ExecuteScriptsAsync(scripts, true).ConfigureAwait(false));
         }
 
         /// <summary>
@@ -318,7 +318,7 @@ private async Task<bool> DatabaseSchemaAsync()
             Logger.LogInformation($"  Probing for embedded resources: {string.Join(", ", GetNamespacesWithSuffix($"{SchemaNamespace}.*.sql"))}");
             foreach (var ass in Assemblies)
             {
-                foreach (var rn in ass.GetManifestResourceNames())
+                foreach (var rn in ass.GetManifestResourceNames().OrderBy(x => x))
                 {
                     // Filter on schema namespace prefix and suffix of '.sql'.
                     if (!(Namespaces.Any(x => rn.StartsWith($"{x}.{SchemaNamespace}.", StringComparison.InvariantCulture) && rn.EndsWith(".sql", StringComparison.InvariantCultureIgnoreCase))))
@@ -363,15 +363,15 @@ private async Task<bool> DatabaseSchemaAsync()
         /// </summary>
         private async Task<bool> DatabaseDataAsync()
         {
-            Logger.LogInformation($"  Probing for embedded resources: {string.Join(", ", GetNamespacesWithSuffix($"{DataNamespace}.*.sql", true))}");
+            Logger.LogInformation($"  Probing for embedded resources: {string.Join(", ", GetNamespacesWithSuffix($"{DataNamespace}.*.[sql|yaml]", true))}");
 
             var list = new List<(Assembly Assembly, string ResourceName)>();
             foreach (var ass in Assemblies)
             {
-                foreach (var rn in ass.GetManifestResourceNames())
+                foreach (var rn in ass.GetManifestResourceNames().OrderBy(x => x))
                 {
                     // Filter on schema namespace prefix and suffix of '.sql'.
-                    if (!Namespaces.Any(x => rn.StartsWith($"{x}.{DataNamespace}.", StringComparison.InvariantCulture) && (rn.EndsWith(".yaml", StringComparison.InvariantCultureIgnoreCase) || rn.EndsWith(".yml", StringComparison.InvariantCultureIgnoreCase))))
+                    if (!Namespaces.Any(x => rn.StartsWith($"{x}.{DataNamespace}.", StringComparison.InvariantCulture) && (rn.EndsWith(".sql", StringComparison.InvariantCultureIgnoreCase) || rn.EndsWith(".yaml", StringComparison.InvariantCultureIgnoreCase) || rn.EndsWith(".yml", StringComparison.InvariantCultureIgnoreCase))))
                         continue;
 
                     list.Add((ass, rn));
@@ -395,21 +395,37 @@ private async Task<bool> DatabaseDataAsync()
             var parser = new DataParser(dbTables, pargs);
             foreach (var item in list)
             {
-                try
+                using var sr = new StreamReader(item.Assembly.GetManifestResourceStream(item.ResourceName)!);
+
+                if (item.ResourceName.EndsWith(".sql", StringComparison.InvariantCultureIgnoreCase))
                 {
+                    // Execute the SQL script directly.
                     Logger.LogInformation(string.Empty);
-                    Logger.LogInformation($"** Parsing and executing: {item.ResourceName}");
-                    using var sr = new StreamReader(item.Assembly.GetManifestResourceStream(item.ResourceName)!);
-
-                    var tables = await parser.ParseYamlAsync(sr);
+                    Logger.LogInformation($"** Executing: {item.ResourceName}");
 
-                    if (!await DatabaseDataAsync(db, tables).ConfigureAwait(false))
+                    var ss = new SqlScript(item.ResourceName, await sr.ReadToEndAsync().ConfigureAwait(false), new SqlScriptOptions { ScriptType = DbUp.Support.ScriptType.RunAlways });
+                    var success = CheckDatabaseUpgradeResult(await ExecuteScriptsAsync(new SqlScript[] { ss }, false).ConfigureAwait(false));
+                    if (!success)
                         return false;
                 }
-                catch (DataParserException dpex)
+                else
                 {
-                    Logger.LogError(dpex.Message);
-                    return false;
+                    // Handle the YAML - parse and execute.
+                    try
+                    {
+                        Logger.LogInformation(string.Empty);
+                        Logger.LogInformation($"** Parsing and executing: {item.ResourceName}");
+
+                        var tables = await parser.ParseYamlAsync(sr);
+
+                        if (!await DatabaseDataAsync(db, tables).ConfigureAwait(false))
+                            return false;
+                    }
+                    catch (DataParserException dpex)
+                    {
+                        Logger.LogError(dpex.Message);
+                        return false;
+                    }
                 }
             }
 
@@ -530,7 +546,7 @@ private async Task<bool> CreateScriptInternalAsync(string? resourceName = null,
         }
 
         /// <summary>
-        /// Executes the raw SQL statements by creating the equivalent <see cref="SqlScript"/> and invoking <see cref="ExecuteScriptsAsync(IEnumerable{SqlScript})"/>.
+        /// Executes the raw SQL statements by creating the equivalent <see cref="SqlScript"/> and invoking <see cref="ExecuteScriptsAsync(IEnumerable{SqlScript}, bool)"/>.
         /// </summary>
         /// <param name="statements"></param>
         /// <returns><c>true</c> indicates success; otherwise, <c>false</c>.</returns>
@@ -557,16 +573,20 @@ private async Task<bool> ExecuteSqlStatementsInternalAsync(params string[] state
             var scripts = new List<SqlScript>();
             for (int i = 0; i < statements.Length; i++)
             {
-                scripts.Add(new SqlScript($"{sn}{i + 1:000}.sql", statements[i]));
+                if (File.Exists(statements[i]))
+                    scripts.Add(new SqlScript(statements[i], await File.ReadAllTextAsync(statements[i]).ConfigureAwait(false)));
+                else
+                    scripts.Add(new SqlScript($"{sn}{i + 1:000}.sql", statements[i]));
             }
 
-            var dur = await ExecuteScriptsAsync(scripts).ConfigureAwait(false);
+            var dur = await ExecuteScriptsAsync(scripts, false).ConfigureAwait(false);
             if (dur.Successful)
                 Logger.LogInformation($"  All scripts executed successfully.");
             else
             {
                 Logger.LogInformation(string.Empty);
-                Logger.LogError($"The following SQL statement failed with: {dur.Error.Message}");
+                Logger.LogError($"The SQL statement failed with: {dur.Error.Message}");
+                Logger.LogWarning($"Script '{dur.ErrorScript.Name}' contents:");
                 Logger.LogWarning(dur.ErrorScript.Contents);
             }
 

diff --git a/src/DbEx/Migration/NullSink.cs b/src/DbEx/Migration/NullSink.cs
@@ -0,0 +1,32 @@
+// Copyright (c) Avanade. Licensed under the MIT License. See https://github.com/Avanade/DbEx
+
+using DbUp.Engine.Output;
+
+namespace DbEx.Migration
+{
+    /// <summary>
+    /// Represents a <i>DbUp</i> <see cref="IUpgradeLog"/> <c>null</c> sink; i.e. all messages to be swallowed.
+    /// </summary>
+    public class NullSink : IUpgradeLog
+    {
+        /// <summary>
+        /// Initializes a new instance of the <see cref="NullSink"/> class.
+        /// </summary>
+        public NullSink() { }
+
+        /// <summary>
+        /// Writes/logs an error message.
+        /// </summary>
+        public void WriteError(string format, params object[] args) { }
+
+        /// <summary>
+        /// Writes/logs an informational message.
+        /// </summary>
+        public void WriteInformation(string format, params object[] args) { }
+
+        /// <summary>
+        /// Writes/logs a warning message.
+        /// </summary>
+        public void WriteWarning(string format, params object[] args) { }
+    }
+}
diff --git a/src/DbEx/Migration/SqlServer/SqlServerMigrator.cs b/src/DbEx/Migration/SqlServer/SqlServerMigrator.cs
@@ -4,6 +4,7 @@
 using DbEx.Migration.SqlServer.Internal;
 using DbUp;
 using DbUp.Engine;
+using DbUp.Engine.Output;
 using Microsoft.Data.SqlClient;
 using Microsoft.Extensions.Logging;
 using OnRamp.Utility;
@@ -84,7 +85,7 @@ protected override async Task<bool> DatabaseSchemaAsync(List<DatabaseMigrationSc
                 ss.Add(new SqlScript(sql, sql, new SqlScriptOptions { RunGroupOrder = i++, ScriptType = DbUp.Support.ScriptType.RunAlways }));
             }
 
-            var r = await ExecuteScriptsAsync(ss).ConfigureAwait(false);
+            var r = await ExecuteScriptsAsync(ss, true).ConfigureAwait(false);
             if (!r.Successful)
                 return false;
 
@@ -97,7 +98,7 @@ protected override async Task<bool> DatabaseSchemaAsync(List<DatabaseMigrationSc
                 ss.Add(new SqlScript(sor.ScriptName, sor.GetSql(), new SqlScriptOptions { RunGroupOrder = i++, ScriptType = DbUp.Support.ScriptType.RunAlways }));
             }
 
-            return CheckDatabaseUpgradeResult(await ExecuteScriptsAsync(ss).ConfigureAwait(false));
+            return CheckDatabaseUpgradeResult(await ExecuteScriptsAsync(ss, true).ConfigureAwait(false));
         }
 
         /// <inheritdoc/>
@@ -106,7 +107,7 @@ protected override async Task<bool> DatabaseResetAsync()
             Logger.LogInformation("    Deleting data from all tables (excludes schema 'dbo' and 'cdc').");
             using var sr = StreamLocator.GetResourcesStreamReader("SqlServer.DeleteAllAndReset.sql", new Assembly[] { typeof(IDatabase).Assembly }).StreamReader!;
             var ss = new SqlScript($"{typeof(IDatabase).Namespace}.SqlServer.DeleteAllAndReset.sql", await sr.ReadToEndAsync().ConfigureAwait(false), new SqlScriptOptions { ScriptType = DbUp.Support.ScriptType.RunAlways });
-            return CheckDatabaseUpgradeResult(await ExecuteScriptsAsync(new SqlScript[] { ss }).ConfigureAwait(false));
+            return CheckDatabaseUpgradeResult(await ExecuteScriptsAsync(new SqlScript[] { ss }, false).ConfigureAwait(false));
         }
 
         /// <inheritdoc/>
@@ -138,15 +139,15 @@ protected override async Task<bool> DatabaseDataAsync(IDatabase database, List<D
         protected override IDatabase CreateDatabase(string connectionString) => new Database<SqlConnection>(() => new SqlConnection(connectionString));
 
         /// <inheritdoc/>
-        public override async Task<DatabaseUpgradeResult> ExecuteScriptsAsync(IEnumerable<SqlScript> scripts)
+        public override async Task<DatabaseUpgradeResult> ExecuteScriptsAsync(IEnumerable<SqlScript> scripts, bool includeExecutionLogging)
         {
             for (int i = 0; true; i++)
             {
                 var dur = DbUp.DeployChanges.To
                     .SqlDatabase(ConnectionString)
                     .WithScripts(scripts)
                     .WithoutTransaction()
-                    .LogTo(new LoggerSink(Logger))
+                    .LogTo(includeExecutionLogging ? (IUpgradeLog)new LoggerSink(Logger) : (IUpgradeLog)new NullSink())
                     .Build()
                     .PerformUpgrade();
 

diff --git a/tests/DbEx.Test.Console/Data/Other.sql b/tests/DbEx.Test.Console/Data/Other.sql
@@ -0,0 +1 @@
+INSERT INTO [Test].[Contact] ([ContactId], [ContactTypeId], [GenderId], [Name], [DateOfBirth]) VALUES (3, (SELECT TOP 1 [ContactTypeId] FROM [Test].[ContactType] WHERE [Code] = 'E'), (SELECT TOP 1 [GenderId] FROM [Test].[Gender] WHERE [Code] = 'M'), 'Barry', '2001-10-22T00:00:00.0000000')