Skip to content

Commit

Permalink
Merge pull request #240 from wellcomecollection/develop
Browse files Browse the repository at this point in the history
Merge develop -> main
tomcrane authored May 16, 2023
2 parents dabb560 + 370d553 commit a195e4a
Showing 368 changed files with 19,588 additions and 5,469 deletions.
6 changes: 3 additions & 3 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -2,10 +2,10 @@ Dockerfile*
.dockerignore
.git
.gitignore
bin\
obj\
bin/
obj/
README.md
LICENSE
.idea
appsettings.Development.Example.json
appsettings.Development.json
appsettings.Development.json
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -17,6 +17,8 @@
mono_crash.*

appsettings.Development.json
appsettings.Development-Alt.json
appsettings.Development-Original.json

# Build results
[Dd]ebug/
@@ -363,4 +365,4 @@ ConnectedService.json

# Migration output as-per readme
dds.sql
dds_instr.sql
dds_instr.sql
3 changes: 3 additions & 0 deletions Dockerfile-dashboard
Original file line number Diff line number Diff line change
@@ -32,6 +32,9 @@ RUN dotnet publish "Wellcome.Dds.Dashboard.csproj" -c Release -o /app/publish
# Runtime image
FROM mcr.microsoft.com/dotnet/aspnet:6.0-bullseye-slim

ENV TZ Europe/London
ENV LANG en_GB.UTF8

LABEL maintainer="Donald Gray <donald.gray@digirati.com>,Tom Crane <tom.crane@digirati.com>"
LABEL org.opencontainers.image.source=https://github.com/wellcomecollection/iiif-builder

43 changes: 43 additions & 0 deletions docs/workflow-messaging.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
## Workflow queues and topics - questions/decisions

There are two existing SNS topics owned by _*storage*_:

- born-digital-bag-notifications-prod
- born-digital-bag-notifications-staging

And there are two new SNS topics for Goobi, owned by _*workflow*_

- digitised-bag-notifications-workflow-prod
- digitised-bag-notifications-workflow-staging

At the time these first two topics were created, queues were also created, subscribing to these topics. We don't use these queues:

* arn:aws:sqs:eu-west-1:975596993436:born-digital-notifications-prod (in storage acct)
* arn:aws:sqs:eu-west-1:653428163053:born-digital-notifications-prod (in digirati acct)

There is also an equivalent queue for staging storage.

## DDS Approach

In our terraform, we reference these four topics as data, but we creates our own queues for each environment:

- Two for each of our prod, stage, test (aka stage-prod) and also local dev environments.

We then subscribe some of these queues to the storage (born-digital) or workflow (goobi) topics.

This means the running DDS has no knowledge of the topics, and certainly never publishes to them. For the scenario of a manual initiation of a workflow from the dashboard, the dashboard sends a message on its appropriate, DDS-infra queue.

The WorkflowProcessor listens to a list of queues, set in DdsOptions `WorkflowMessageListenQueues` (so we can add others for testing), and the dashboard is capable of sending messages to one of two specific queues for its environment, one for born digital and one for digitised, the settings `DashboardPushDigitisedQueue` and `DashboardPushBornDigitalQueue`. In practice, the workflowProcessor in that environment is listening to these same two queues, but it can listen to others.

This means that DDS terraform manages all the notification queues it uses, the only point of contact with storage/workflow infrastructure is the queue subscription in terraform.

We can have more than one WorkflowProcessor service picking up a job, but we don't want more than one environment synchronising with the same DLCS space.
So DDS production and test each have their own queue(s) subscribed to both Goobi and born-digital topics.

- DDS production writes to space 5, as now.
- DDS staging writes to DLCS space 6, as now.
- DDS test (stage-prod) writes to space 7 (this is NEW)

In normal operation the `Test` environment WorkflowProcessor does not listen to queues. This is set via `WorkflowMessagePoll` in DdsOptions. It might be that Test WorkflowProcessor is deployed with an empty list for `WorkflowMessageListenQueues`.

Stage should listen to Goobi stage and Storage stage, and sync with DLCS, all the time it's running (assuming much less traffic comes through stage, and the traffic that does come through still needs to be looked at via IIIF/DLCS).
9 changes: 2 additions & 7 deletions src/Wellcome.Dds/CatalogueAPI.Tests/CatalogueTests.cs
Original file line number Diff line number Diff line change
@@ -1,25 +1,20 @@
using FluentAssertions;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
using System;
using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;
using NuGet.Frameworks;
using Test.Helpers;
using Wellcome.Dds.Common;
using Wellcome.Dds.Repositories.Catalogue;
using Xunit;

namespace CatalogueAPI.Tests
{
#nullable disable
[Trait("Category", "Manual")]
public class CatalogueTests
{
private readonly Wellcome.Dds.Repositories.Catalogue.WellcomeCollectionCatalogue sut;
private readonly JsonSerializerSettings serializer;
private readonly WellcomeCollectionCatalogue sut;

public CatalogueTests()
{
9 changes: 9 additions & 0 deletions src/Wellcome.Dds/CatalogueClient/CatalogueClient.csproj
Original file line number Diff line number Diff line change
@@ -4,6 +4,15 @@
<OutputType>Exe</OutputType>
<TargetFramework>net6.0</TargetFramework>
<LangVersion>latestmajor</LangVersion>
<Nullable>enable</Nullable>
</PropertyGroup>

<PropertyGroup Condition=" '$(Configuration)' == 'Debug' ">
<NoWarn>1701;1702;1591</NoWarn>
</PropertyGroup>

<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
<NoWarn>1701;1702;1591</NoWarn>
</PropertyGroup>

<ItemGroup>
8 changes: 5 additions & 3 deletions src/Wellcome.Dds/CatalogueClient/Program.cs
Original file line number Diff line number Diff line change
@@ -23,10 +23,10 @@ namespace CatalogueClient
class Program
{
static async Task Main(
string id = null,
FileInfo file = null,
string? id = null,
FileInfo? file = null,
bool update = false,
string bulkop = null,
string? bulkop = null,
int skip = 1)
{
var sw = new Stopwatch();
@@ -100,6 +100,8 @@ private static void CompareAvailabilities(DumpUtils dumpUtils, int skip)
Console.WriteLine("processing: " + counter);
}
var work = catalogue.FromDumpLine(line, options);
if(work == null) continue;

bool? online = work.IsOnline();
if (online == null) continue; // availabilities is NOT present

Original file line number Diff line number Diff line change
@@ -4,7 +4,7 @@ namespace CatalogueClient.ToolSupport
{
public class DumpLoopInfo
{
public string Filter;
public string? Filter;
public int Skip = 1;
public int Offset = 0;
public int TotalCount;
7 changes: 5 additions & 2 deletions src/Wellcome.Dds/CatalogueClient/ToolSupport/DumpUtils.cs
Original file line number Diff line number Diff line change
@@ -6,6 +6,7 @@
using System.Linq;
using System.Net.Http;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Threading;
using System.Threading.Tasks;
using ShellProgressBar;
@@ -116,8 +117,8 @@ public static JsonSerializerOptions GetSerialiserOptions()
var options = new JsonSerializerOptions
{
WriteIndented = true,
IgnoreNullValues = true,
PropertyNameCaseInsensitive = true
PropertyNameCaseInsensitive = true,
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull
};
return options;
}
@@ -132,6 +133,8 @@ public void FindDigitisedBNumbers(DumpLoopInfo info, ICatalogue catalogue)
{
info.UsedLines++;
var work = catalogue.FromDumpLine(line, options);
if(work == null) continue;

var sierraSystemBNumbers = work.GetSierraSystemBNumbers();
var digitalLocationBNumbers = work.GetDigitisedBNumbers();
foreach (var digitalLocationBNumber in digitalLocationBNumbers)
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.DependencyInjection;
using Wellcome.Dds.AssetDomain.Dashboard;
using Wellcome.Dds.AssetDomain.DigitalObjects;
using Wellcome.Dds.AssetDomain.Dlcs.Ingest;

namespace DlcsJobProcessor
1 change: 1 addition & 0 deletions src/Wellcome.Dds/DlcsJobProcessor/DlcsJobProcessor.csproj
Original file line number Diff line number Diff line change
@@ -4,6 +4,7 @@
<TargetFramework>net6.0</TargetFramework>
<LangVersion>latestmajor</LangVersion>
<EnableUnsafeBinaryFormatterSerialization>true</EnableUnsafeBinaryFormatterSerialization>
<Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
4 changes: 2 additions & 2 deletions src/Wellcome.Dds/DlcsJobProcessor/JobProcessorOptions.cs
Original file line number Diff line number Diff line change
@@ -11,12 +11,12 @@ public class JobProcessorOptions
/// Identifier filter to apply when fetching jobs.
/// Valid for "processqueue" mode only.
/// </summary>
public string Filter { get; set; }
public string? Filter { get; set; }

// TODO - validate this?
/// <summary>
/// The job mode "processqueue" or "updatestatus"
/// </summary>
public string Mode { get; set; }
public string? Mode { get; set; }
}
}
9 changes: 3 additions & 6 deletions src/Wellcome.Dds/DlcsJobProcessor/Startup.cs
Original file line number Diff line number Diff line change
@@ -11,11 +11,11 @@
using Utils.Caching;
using Utils.Storage;
using Wellcome.Dds.AssetDomain;
using Wellcome.Dds.AssetDomain.Dashboard;
using Wellcome.Dds.AssetDomain.DigitalObjects;
using Wellcome.Dds.AssetDomain.Dlcs.Ingest;
using Wellcome.Dds.AssetDomain.Mets;
using Wellcome.Dds.AssetDomainRepositories;
using Wellcome.Dds.AssetDomainRepositories.Dashboard;
using Wellcome.Dds.AssetDomainRepositories.DigitalObjects;
using Wellcome.Dds.AssetDomainRepositories.Ingest;
using Wellcome.Dds.AssetDomainRepositories.Mets;
using Wellcome.Dds.AssetDomainRepositories.Storage.WellcomeStorageService;
@@ -37,9 +37,6 @@ public Startup(IConfiguration configuration)

public void ConfigureServices(IServiceCollection services)
{
// Use pre-v6 handling of datetimes for npgsql
AppContext.SetSwitch("Npgsql.EnableLegacyTimestampBehavior", true);

services.AddDbContext<DdsInstrumentationContext>(options => options
.UseNpgsql(Configuration.GetConnectionString("DdsInstrumentation"))
.UseSnakeCaseNamingConvention());
@@ -72,7 +69,7 @@ public void ConfigureServices(IServiceCollection services)
.AddScoped<StorageServiceClient>()
.AddScoped<IMetsRepository, MetsRepository>()
.AddSingleton<ISimpleCache, ConcurrentSimpleMemoryCache>()
.AddScoped<IDashboardRepository, DashboardRepository>()
.AddScoped<IDigitalObjectRepository, DigitalObjectRepository>()
.AddScoped<IIngestJobProcessor, DashboardCloudServicesJobProcessor>()
.AddSingleton<UriPatterns>()
.AddHostedService<DashboardContinuousRunningStrategy>();
5 changes: 3 additions & 2 deletions src/Wellcome.Dds/DlcsJobProcessor/appsettings.Production.json
Original file line number Diff line number Diff line change
@@ -14,10 +14,11 @@
},
"Dlcs": {
"CustomerDefaultSpace": 5,
"ResourceEntryPoint": "https://iiif.wellcomecollection.org/"
"ResourceEntryPoint": "https://iiif.wellcomecollection.org/",
"SupportsDeliveryChannels": true
},
"Dds": {
"LinkedDataDomain": "https://iiif.wellcomecollection.org",
"LinkedDataDomain": "https://iiif.wellcomecollection.org"
},
"Storage": {
"StorageApiTemplate": "https://api.wellcomecollection.org/storage/v1/bags/{0}/{1}",
36 changes: 36 additions & 0 deletions src/Wellcome.Dds/DlcsJobProcessor/appsettings.Staging-New.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"Serilog": {
"MinimumLevel": {
"Default": "Debug",
"Override": {
"Microsoft": "Warning",
"System": "Warning"
}
},
"Properties": {
"ApplicationName": "Job-Processor",
"Environment": "Staging-New"
}
},
"Dlcs": {
"CustomerDefaultSpace": 6,
"SkeletonNamedQueryTemplate": "https://neworchestrator.dlcs.io/iiif-resource/wellcome/preview/{0}/{1}",
"ApiEntryPoint": "https://newapi.dlcs.io/",
"ResourceEntryPoint": "https://iiif-stage-new.wellcomecollection.org/",
"InternalResourceEntryPoint": "https://neworchestrator.dlcs.io/",
"SupportsDeliveryChannels": true
},
"Dds": {
"LinkedDataDomain": "https://iiif-stage-new.wellcomecollection.org"
},
"Storage": {
"StorageApiTemplate": "https://api-stage.wellcomecollection.org/storage/v1/bags/{0}/{1}",
"StorageApiTemplateIngest": "https://api-stage.wellcomecollection.org/storage/v1/ingests"
},
"BinaryObjectCache": {
"Wellcome.Dds.AssetDomainRepositories.Mets.WellcomeBagAwareArchiveStorageMap": {
"Container": "wellcomecollection-stage-new-iiif-storagemaps",
"Prefix": "stgmap-"
}
}
}
12 changes: 8 additions & 4 deletions src/Wellcome.Dds/DlcsJobProcessor/appsettings.Staging-Prod.json
Original file line number Diff line number Diff line change
@@ -13,19 +13,23 @@
}
},
"Dlcs": {
"CustomerDefaultSpace": 6,
"ResourceEntryPoint": "https://iiif-test.wellcomecollection.org/"
"CustomerDefaultSpace": 5,
"SkeletonNamedQueryTemplate": "https://neworchestrator.dlcs.io/iiif-resource/wellcome/preview/{0}/{1}",
"ApiEntryPoint": "https://newapi.dlcs.io/",
"ResourceEntryPoint": "https://iiif-test.wellcomecollection.org/",
"InternalResourceEntryPoint": "https://neworchestrator.dlcs.io/",
"SupportsDeliveryChannels": true
},
"Dds": {
"LinkedDataDomain": "https://iiif-test.wellcomecollection.org",
"LinkedDataDomain": "https://iiif-test.wellcomecollection.org"
},
"Storage": {
"StorageApiTemplate": "https://api.wellcomecollection.org/storage/v1/bags/{0}/{1}",
"StorageApiTemplateIngest": "https://api.wellcomecollection.org/storage/v1/ingests"
},
"BinaryObjectCache": {
"Wellcome.Dds.AssetDomainRepositories.Mets.WellcomeBagAwareArchiveStorageMap": {
"Container": "wellcomecollection-stage-iiif-storagemaps",
"Container": "wellcomecollection-test-iiif-storagemaps",
"Prefix": "stgmap-"
}
}
3 changes: 2 additions & 1 deletion src/Wellcome.Dds/DlcsJobProcessor/appsettings.json
Original file line number Diff line number Diff line change
@@ -36,6 +36,7 @@
"SkeletonNamedQueryTemplate": "https://dlcs.io/iiif-resource/wellcome/preview/{0}/{1}",
"ApiEntryPoint": "https://api.dlcs.io/",
"ResourceEntryPoint": "https://iiif-test.wellcomecollection.org/",
"InternalResourceEntryPoint": "https://dlcs.io/",
"BatchSize": 100,
"PreventSynchronisation": false,
"PdfQueryName": "pdf-item",
@@ -60,7 +61,7 @@
"LinkedDataDomain": "https://iiif.wellcomecollection.org",
"AvoidCaching": false,
"EarliestJobDateTime": "2016-07-07",
"MinimumJobAgeMinutes": 0,
"MinimumJobAgeMinutes": 1,
"CacheBuster": "TODO - may not need these but if we do it's an object { }",
"JobProcessorLog": "<firelens equivalent of> DlcsJobProcessor-ProcessQueue.log",
"WorkflowProcessorLog": "<firelens equivalent of> WorkflowProcessor.log",
36 changes: 19 additions & 17 deletions src/Wellcome.Dds/DlcsWebClient.Tests/Dlcs/DlcsTests.cs
Original file line number Diff line number Diff line change
@@ -13,6 +13,7 @@
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
using Test.Helpers;
using Wellcome.Dds.AssetDomain;
using Wellcome.Dds.AssetDomain.Dlcs;
using Wellcome.Dds.AssetDomain.Dlcs.Model;
using Wellcome.Dds.Common;
@@ -63,7 +64,7 @@ public async Task RegisterImages_PostsToCorrectUri_NotPriority()
httpHandler.RegisterCallback(r => message = r);

// Act
await sut.RegisterImages(request, false);
await sut.RegisterImages(request, new DlcsCallContext("[test]", "[id]"), false);

// Assert
httpHandler.CallsMade.Should().ContainSingle()
@@ -82,7 +83,7 @@ public async Task RegisterImages_PostsToCorrectUri_Priority()
httpHandler.RegisterCallback(r => message = r);

// Act
await sut.RegisterImages(request, true);
await sut.RegisterImages(request, new DlcsCallContext("[test]", "[id]"), true);

// Assert
httpHandler.CallsMade.Should().ContainSingle()
@@ -105,7 +106,7 @@ public async Task RegisterImages_PostsSerializedHydraBody(bool priority)
httpHandler.RegisterCallback(r => message = r);

// Act
await sut.RegisterImages(request, priority);
await sut.RegisterImages(request, new DlcsCallContext("[test]", "[id]"), priority);

// Assert
(await message.Content.ReadAsStringAsync()).Should().Be(expected);
@@ -122,7 +123,7 @@ public async Task PatchImages_PatchesToCorrectUri()
httpHandler.RegisterCallback(r => message = r);

// Act
await sut.PatchImages(request);
await sut.PatchImages(request, new DlcsCallContext("[test]", "[id]"));

// Assert
httpHandler.CallsMade.Should().ContainSingle()
@@ -148,7 +149,7 @@ public async Task PatchImages_PatchesSerializedHydraBody()
httpHandler.RegisterCallback(r => message = r);

// Act
await sut.PatchImages(request);
await sut.PatchImages(request, new DlcsCallContext("[test]", "[id]"));

// Assert
(await message.Content.ReadAsStringAsync()).Should().Be(expected);
@@ -171,7 +172,7 @@ public async Task PatchImages_UpdatesModelId_IfIncorrectFormat()
httpHandler.RegisterCallback(r => message = r);

// Act
await sut.PatchImages(request);
await sut.PatchImages(request, new DlcsCallContext("[test]", "[id]"));

// Assert
var actual = await message.Content.ReadAsAsync<HydraImageCollection>();
@@ -188,7 +189,7 @@ public async Task GetImages_CallsCorrectUri_UsingDefaultSpaceIfNoSpaceSpecified(
httpHandler.RegisterCallback(r => message = r);

// Act
await sut.GetImages(imageQuery, 999);
await sut.GetImages(imageQuery,999, new DlcsCallContext("[test]", "[id]"));

// Assert
httpHandler.CallsMade.Should().ContainSingle()
@@ -206,7 +207,7 @@ public async Task GetImages_CallsCorrectUri_UsingSpecifiedSpace()
httpHandler.RegisterCallback(r => message = r);

// Act
await sut.GetImages(imageQuery, 999);
await sut.GetImages(imageQuery, 999, new DlcsCallContext("[test]", "[id]"));

// Assert
httpHandler.CallsMade.Should().ContainSingle()
@@ -224,7 +225,7 @@ public async Task GetImages_NextUri_CallsCorrectUri()
httpHandler.RegisterCallback(r => message = r);

// Act
await sut.GetImages(uri);
await sut.GetImages(uri, new DlcsCallContext("[test]", "[id]"));

// Assert
httpHandler.CallsMade.Should().ContainSingle()
@@ -242,7 +243,7 @@ public async Task GetBatch_CallsCorrectUri()
httpHandler.RegisterCallback(r => message = r);

// Act
await sut.GetBatch(batchId);
await sut.GetBatch(batchId, new DlcsCallContext("[test]", "[id]"));

// Assert
httpHandler.CallsMade.Should().ContainSingle()
@@ -260,7 +261,7 @@ public async Task GetBatch_UsesPassedInBatchIdAsUri_IfValidUri()
httpHandler.RegisterCallback(r => message = r);

// Act
await sut.GetBatch(batchId);
await sut.GetBatch(batchId, new DlcsCallContext("[test]", "[id]"));

// Assert
httpHandler.CallsMade.Should().ContainSingle()
@@ -276,16 +277,17 @@ public void GetRoleUri_ReturnsClickThrough_ForRequiresRegistration()
var roleUriOa = sut.GetRoleUri(AccessCondition.OpenWithAdvisory);

// Assert
var expected = "https://api.dlcs.test/customers/99/roles/clickthrough";
// NOTE THAT THIS IS NOT api.dlcs.test - roles are Global URIs
var expected = "https://api.dlcs.io/customers/99/roles/clickthrough";
roleUriRr.Should().Be(expected);
roleUriOa.Should().Be(expected);
}

[Theory]
[InlineData(AccessCondition.Open, "https://api.dlcs.test/customers/99/roles/open")]
[InlineData(AccessCondition.ClinicalImages, "https://api.dlcs.test/customers/99/roles/clinicalImages")]
[InlineData(AccessCondition.RestrictedFiles, "https://api.dlcs.test/customers/99/roles/restrictedFiles")]
[InlineData(AccessCondition.Closed, "https://api.dlcs.test/customers/99/roles/closed")]
[InlineData(AccessCondition.Open, "https://api.dlcs.io/customers/99/roles/open")]
[InlineData(AccessCondition.ClinicalImages, "https://api.dlcs.io/customers/99/roles/clinicalImages")]
[InlineData(AccessCondition.RestrictedFiles, "https://api.dlcs.io/customers/99/roles/restrictedFiles")]
[InlineData(AccessCondition.Closed, "https://api.dlcs.io/customers/99/roles/closed")]
public void GetRoleUri_ReturnsExpected_ForNonRequiresRegistration(string accessCondition, string expected)
{
// Act
@@ -309,7 +311,7 @@ public async Task GetImagesForIssue_CallsCorrectUri()
httpHandler.RegisterCallback(r => message = r);

// Act
await sut.GetImagesForIssue(issueIdentifier);
await sut.GetImagesForIssue(issueIdentifier, new DlcsCallContext("[test]", "[id]"));

// Assert
httpHandler.CallsMade.Should().ContainSingle()
23 changes: 17 additions & 6 deletions src/Wellcome.Dds/DlcsWebClient/Config/DlcsOptions.cs
Original file line number Diff line number Diff line change
@@ -2,21 +2,32 @@
{
public class DlcsOptions
{
public string ApiKey { get; set; }
public string ApiSecret { get; set; }
public string? ApiKey { get; set; }
public string? ApiSecret { get; set; }
public int CustomerId { get; set; }
public string CustomerName { get; set; }
public string? CustomerName { get; set; }
public int CustomerDefaultSpace { get; set; }
public string ApiEntryPoint { get; set; }
public string ResourceEntryPoint { get; set; }
public string? ApiEntryPoint { get; set; }
public string? GlobalDlcsUrl { get; set; } = "https://api.dlcs.io/";
public string? ResourceEntryPoint { get; set; }
public string? InternalResourceEntryPoint { get; set; } = "https://dlcs.io/";
public int BatchSize { get; set; } = 100;
public string SkeletonNamedQueryTemplate { get; set; }
public string? SkeletonNamedQueryTemplate { get; set; }
public bool PreventSynchronisation { get; set; } = false;
public string PdfQueryName { get; set; } = "pdf";

/// <summary>
/// Default timeout (in ms) use for HttpClient.Timeout.
/// </summary>
public int DefaultTimeoutMs { get; set; } = 30000;

/// <summary>
/// Whether to call the DLCS using old Deliverator AssetFamily, or protagonist delivery channels
/// </summary>
public bool SupportsDeliveryChannels { get; set; } = false;

public string? PortalPageTemplate { get; set; }
public string? PortalBatchTemplate { get; set; }
public string? SingleAssetManifestTemplate { get; set; }
}
}
422 changes: 257 additions & 165 deletions src/Wellcome.Dds/DlcsWebClient/Dlcs/Dlcs.cs

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions src/Wellcome.Dds/DlcsWebClient/Dlcs/ServiceCollectionX.cs
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@
using DlcsWebClient.Config;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Utils;
using Utils.Web;
using Wellcome.Dds.AssetDomain.Dlcs;

@@ -27,6 +28,10 @@ public static IHttpClientBuilder AddDlcsClient(this IServiceCollection services,
{
client.DefaultRequestHeaders.Accept
.Add(new MediaTypeWithQualityHeaderValue("application/json"));
if (dlcsOptions.ApiKey.IsNullOrWhiteSpace() || dlcsOptions.ApiSecret.IsNullOrWhiteSpace())
{
throw new InvalidOperationException("Missing DLCS API key/secret in config");
}
client.DefaultRequestHeaders.AddBasicAuth(dlcsOptions.ApiKey, dlcsOptions.ApiSecret);
client.Timeout = TimeSpan.FromMilliseconds(dlcsOptions.DefaultTimeoutMs);
});
1 change: 1 addition & 0 deletions src/Wellcome.Dds/DlcsWebClient/DlcsWebClient.csproj
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@
<PropertyGroup>
<TargetFramework>net6.0</TargetFramework>
<LangVersion>latestmajor</LangVersion>
<Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
11 changes: 10 additions & 1 deletion src/Wellcome.Dds/PdfThumbGenerator/PdfThumbGenerator.csproj
Original file line number Diff line number Diff line change
@@ -3,9 +3,18 @@
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net6.0</TargetFramework>
<Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
<PropertyGroup Condition=" '$(Configuration)' == 'Debug' ">
<NoWarn>1701;1702;1591</NoWarn>
</PropertyGroup>

<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
<NoWarn>1701;1702;1591</NoWarn>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="AWSSDK.SecurityToken" Version="3.7.1.167" />
<PackageReference Include="CsvHelper" Version="27.2.1" />
<PackageReference Include="GhostScript.NetCore" Version="1.0.1" />
28 changes: 20 additions & 8 deletions src/Wellcome.Dds/PdfThumbGenerator/PdfThumbnailUtil.cs
Original file line number Diff line number Diff line change
@@ -14,12 +14,21 @@

namespace PdfThumbGenerator
{
/// <summary>
/// Wrapper round GhostScript executable call to generate an image from the first page of a PDF
/// </summary>
public class PdfThumbnailUtil
{
private readonly ILogger<PdfThumbnailUtil> logger;
private readonly IAmazonS3 ddsServiceS3;
private readonly DdsOptions ddsOptions;

/// <summary>
///
/// </summary>
/// <param name="s3ClientFactory"></param>
/// <param name="ddsOptions"></param>
/// <param name="logger"></param>
public PdfThumbnailUtil(
INamedAmazonS3ClientFactory s3ClientFactory,
IOptions<DdsOptions> ddsOptions,
@@ -30,6 +39,11 @@ public PdfThumbnailUtil(
this.ddsOptions = ddsOptions.Value;
}

/// <summary>
///
/// </summary>
/// <param name="pdfStreamSource"></param>
/// <param name="identifier"></param>
public async Task EnsurePdfThumbnails(Func<Task<Stream>> pdfStreamSource, string identifier)
{
var folder = Path.Combine(Path.GetTempPath(), "pdf_thumbs");
@@ -79,7 +93,7 @@ private async Task GenerateJpgFromPdf(string outputJpgPath, string inputPdfPath,
$"-dNOPAUSE -dBATCH -r96 -sDEVICE=jpeg -sOutputFile=\"{outputJpgPath}\" -dLastPage=1 \"{inputPdfPath}\"";
logger.LogInformation("Calling ghostscript with args {Args}", args);

using (var ghostScriptProcess = new Process
using var ghostScriptProcess = new Process
{
StartInfo = new ProcessStartInfo
{
@@ -89,13 +103,11 @@ private async Task GenerateJpgFromPdf(string outputJpgPath, string inputPdfPath,
UseShellExecute = false,
RedirectStandardOutput = true,
}
})
{
ghostScriptProcess.Start();
var output = await ghostScriptProcess.StandardOutput.ReadToEndAsync();
logger.LogInformation(output);
await ghostScriptProcess.WaitForExitAsync();
}
};
ghostScriptProcess.Start();
var output = await ghostScriptProcess.StandardOutput.ReadToEndAsync();
logger.LogInformation(output);
await ghostScriptProcess.WaitForExitAsync();
}

private async Task ResizeImageToS3(string outputJpgPath, string key, string identifier)
29 changes: 17 additions & 12 deletions src/Wellcome.Dds/PdfThumbGenerator/Program.cs
Original file line number Diff line number Diff line change
@@ -23,8 +23,10 @@

namespace PdfThumbGenerator
{
// ReSharper disable once UnusedType.Global
class Program
{
// ReSharper disable once UnusedMember.Local
static async Task Main(string file = @"C:\temp\Digitised_PDF_list.csv")
{
await Host.CreateDefaultBuilder()
@@ -38,11 +40,11 @@ await Host.CreateDefaultBuilder()
.AddSingleton<IWorkStorageFactory, ArchiveStorageServiceWorkStorageFactory>()
.AddSingleton(typeof(IBinaryObjectCache<>), typeof(BinaryObjectCache<>))
.AddSingleton<PdfThumbnailUtil>()
.AddHostedService<PdfGenerator>(provider =>
.AddHostedService(provider =>
new PdfGenerator(
provider.GetService<ILogger<PdfGenerator>>(),
provider.GetService<IMetsRepository>(),
provider.GetService<PdfThumbnailUtil>(),
provider.GetService<ILogger<PdfGenerator>>()!,
provider.GetService<IMetsRepository>()!,
provider.GetService<PdfThumbnailUtil>()!,
file));

services.AddHttpClient<OAuth2ApiConsumer>();
@@ -71,6 +73,7 @@ public class PdfGenerator : IHostedService
private readonly PdfThumbnailUtil pdfThumbnailUtil;
private readonly string file;


public PdfGenerator(ILogger<PdfGenerator> logger, IMetsRepository metsRepository,
PdfThumbnailUtil pdfThumbnailUtil, string file)
{
@@ -80,6 +83,7 @@ public PdfGenerator(ILogger<PdfGenerator> logger, IMetsRepository metsRepository
this.file = file;
}


public async Task StartAsync(CancellationToken cancellationToken)
{
// read csv
@@ -93,6 +97,7 @@ public async Task StartAsync(CancellationToken cancellationToken)
}
}


public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;

private static List<BornDigitalPdf> GetBornDigitalPdfs(string file)
@@ -109,20 +114,20 @@ private async Task ProcessPdf(BornDigitalPdf pdf)
try
{
logger.LogDebug("Processing {Identifier}", pdf.Identifier);
IMetsResource resource = await metsRepository.GetAsync(pdf.Identifier);
if (resource is ICollection multipleManifestation)
IMetsResource? resource = await metsRepository.GetAsync(pdf.Identifier!);
if (resource is ICollection)
{
throw new InvalidOperationException(
$"{pdf.Identifier} is a multiple manifestation - update handling");
}
var manifestation = resource as IManifestation;
var pdfItems = manifestation!.Sequence.Where(s => s.MimeType == "application/pdf");
var pdfItems = manifestation!.Sequence!.Where(s => s.MimeType == "application/pdf");

foreach (var pdfItem in pdfItems)
{
await pdfThumbnailUtil.EnsurePdfThumbnails(
() => pdfItem.WorkStore.GetStreamForPathAsync(pdfItem.RelativePath),
pdf.Identifier);
() => pdfItem.WorkStore.GetStreamForPathAsync(pdfItem.RelativePath!),
pdf.Identifier!);
}
}
catch (Exception ex)
@@ -134,8 +139,8 @@ await pdfThumbnailUtil.EnsurePdfThumbnails(

public class BornDigitalPdf
{
public string Identifier { get; set; }
public string Processed { get; set; }
public string Title { get; set; }
public string? Identifier { get; set; }
public string? Processed { get; set; }
public string? Title { get; set; }
}
}
36 changes: 8 additions & 28 deletions src/Wellcome.Dds/Utils.Aws/S3/S3CacheAwareStorage.cs
Original file line number Diff line number Diff line change
@@ -46,8 +46,13 @@ public ISimpleStoredFileInfo GetCachedFileInfo(string container, string fileName
public Task DeleteCacheFile(string container, string fileName)
=> amazonS3.DeleteObjectAsync(container, fileName);

public async Task<T> Read<T>(ISimpleStoredFileInfo fileInfo) where T : class
public async Task<T?> Read<T>(ISimpleStoredFileInfo fileInfo) where T : class
{
if (fileInfo.Container.IsNullOrWhiteSpace())
{
logger.LogError("No Container specified for ISimpleStoredFileInfo object");
return default;
}
var path = options.ReadProtobuf ? GetProtobufKey(fileInfo.Path) : fileInfo.Path;
try
{
@@ -82,24 +87,7 @@ public async Task Write<T>(T t, ISimpleStoredFileInfo fileInfo, bool writeFailTh
var sw = Stopwatch.StartNew();
if (options.WriteBinary)
{
var request = new PutObjectRequest
{
BucketName = fileInfo.Container, Key = fileInfo.Path
};
logger.LogDebug("Writing binary cache file '{Bucket}/{Path}' to S3", request.BucketName,
request.Key);
IFormatter formatter = new BinaryFormatter();
await using (request.InputStream = new MemoryStream())
{
sw.Restart();

formatter.Serialize(request.InputStream, t);
await amazonS3.PutObjectAsync(request);

logger.LogDebug("Wrote stream for '{Bucket}/{Path}' in {Elapsed}ms", request.BucketName,
request.Key,
sw.ElapsedMilliseconds);
}
throw new NotSupportedException("Binary Serializer is no longer supported");
}

if (options.WriteProtobuf)
@@ -170,15 +158,7 @@ private T Deserialize<T>(Stream source, ISimpleStoredFileInfo fileInfo)
return ProtoBuf.Serializer.Deserialize<T>(source);
}

IFormatter formatter = new BinaryFormatter();
var obj = formatter.Deserialize(source);
if (obj is T binaryDeserialized)
{
return binaryDeserialized;
}

throw new InvalidOperationException(
$"Attempt to deserialize '{fileInfo.Uri}' from S3 failed, expected {typeof(T)}, found {obj.GetType()}");
throw new NotSupportedException("Binary Serializer is no longer supported");
}

// TODO - having this knowledge here isn't great but is only temporary until everything is moved to protobuf
7 changes: 6 additions & 1 deletion src/Wellcome.Dds/Utils.Aws/S3/S3Storage.cs
Original file line number Diff line number Diff line change
@@ -33,8 +33,13 @@ public ISimpleStoredFileInfo GetCachedFileInfo(string container, string fileName
public Task DeleteCacheFile(string container, string fileName)
=> amazonS3.DeleteObjectAsync(container, fileName);

public async Task<T> Read<T>(ISimpleStoredFileInfo fileInfo) where T : class
public async Task<T?> Read<T>(ISimpleStoredFileInfo fileInfo) where T : class
{
if (fileInfo.Container.IsNullOrWhiteSpace())
{
logger.LogError("No Container specified for ISimpleStoredFileInfo object");
return default;
}
try
{
await using var stream = await GetStream(fileInfo.Container, fileInfo.Path);
77 changes: 77 additions & 0 deletions src/Wellcome.Dds/Utils.Aws/SQS/QueueMessage.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
using System;
using System.Collections.Generic;
using Newtonsoft.Json.Linq;

namespace Utils.Aws.SQS;

/// <summary>
/// Generic representation of message pulled from queue.
/// Adapted from protagonist for NewtonSoft.JSON
/// </summary>
public class QueueMessage
{
public QueueMessage(JObject body, Dictionary<string, string> attributes, string messageId, string queueName)
{
Body = body;
Attributes = attributes;
MessageId = messageId;
QueueName = queueName;
}

/// <summary>
/// The full message body property
/// </summary>
public JObject Body { get; set; }

/// <summary>
/// Any attributes associated with message
/// </summary>
public Dictionary<string, string> Attributes { get; set; }

/// <summary>
/// Unique identifier for message
/// </summary>
public string MessageId { get; set; }

/// <summary>
/// The name of the queue that this message was from
/// </summary>
public string QueueName { get; set; }
}

public static class QueueMessageX
{
/// <summary>
/// Get a <see cref="JsonObject"/> representing the contents of the message as raised by source system. This helps
/// when the message can be from SNS->SQS, SNS->SQS with RawDelivery or SQS directly.
///
/// If originating from SNS and Raw Message Delivery is disabled (default) then the <see cref="QueueMessage"/>
/// object will have additional fields about topic etc, and the message will be embedded in a "Message" property.
/// e.g. { "Type": "Notification", "MessageId": "1234", "Message": { \"key\": \"value\" } }
///
/// If originating from SQS, or from SNS with Raw Message Delivery enabled, the <see cref="QueueMessage"/> Body
/// property will contain the full message only.
/// e.g. { "key": "value" }
/// </summary>
/// <remarks>See https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html </remarks>
public static JObject? GetMessageContents(this QueueMessage queueMessage)
{
const string messageKey = "Message";
if (queueMessage.Body.ContainsKey("TopicArn") && queueMessage.Body.ContainsKey(messageKey))
{
// From SNS without Raw Message Delivery
try
{
var value = queueMessage.Body[messageKey]!.Value<string>();
return value.HasText() ? JObject.Parse(value) : null;
}
catch (Exception)
{
return null;
}
}

// From SQS or SNS with Raw Message Delivery
return queueMessage.Body;
}
}
2 changes: 2 additions & 0 deletions src/Wellcome.Dds/Utils.Aws/Utils.Aws.csproj
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@
<PropertyGroup>
<TargetFramework>net6.0</TargetFramework>
<LangVersion>latestmajor</LangVersion>
<Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
@@ -13,6 +14,7 @@
<PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="3.7.2" />
<PackageReference Include="AWSSDK.S3" Version="3.7.9.18" />
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="6.0.0" />
<PackageReference Include="Newtonsoft.Json" Version="13.0.1" />
<PackageReference Include="protobuf-net" Version="3.1.4" />
</ItemGroup>

Original file line number Diff line number Diff line change
@@ -33,7 +33,8 @@ private BinaryObjectCache<FakeStoredFileInfo> GetSut(bool hasMemoryCache = true,
MemoryCacheSeconds = cacheSeconds,
AvoidCaching = avoidCaching,
AvoidSaving = avoidSaving,
Container = ContainerName
Container = ContainerName,
WriteFailThrowsException = false
};
var byType = new BinaryObjectCacheOptionsByType();
byType["Utils.Tests.Caching.FakeStoredFileInfo"] = fakeStoredFileInfoOptions;
39 changes: 39 additions & 0 deletions src/Wellcome.Dds/Utils.Tests/StringProcessingTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
using System.Collections.Generic;
using FluentAssertions;
using Wellcome.Dds.Repositories.Presentation.LicencesAndRights;
using Xunit;

namespace Utils.Tests;

public class StringProcessingTests
{
private static readonly Dictionary<string, string> LicenseMap = new()
{
["PDM"] = "https://creativecommons.org/publicdomain/mark/1.0/",
["CC0"] = "https://creativecommons.org/publicdomain/zero/1.0/",
["CC-BY"] = "https://creativecommons.org/licenses/by/4.0/",
["CC-BY-NC"] = "https://creativecommons.org/licenses/by-nc/4.0/",
["CC-BY-NC-ND"] = "https://creativecommons.org/licenses/by-nc-nd/4.0/",
["CC-BY-ND"] = "https://creativecommons.org/licenses/by-nd/4.0/",
["CC-BY-SA"] = "https://creativecommons.org/licenses/by-sa/4.0/",
["CC-BY-NC-SA"] = "https://creativecommons.org/licenses/by-nc-sa/4.0/",
["OGL"] = "http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/",
["OPL"] = "http://www.parliament.uk/site-information/copyright/open-parliament-licence/",
["ARR"] = "https://en.wikipedia.org/wiki/All_rights_reserved",
["All Rights Reserved"] = "https://en.wikipedia.org/wiki/All_rights_reserved",
};

[Theory]
[InlineData("This is CC-BY", @"This is <a href=""https://creativecommons.org/licenses/by/4.0/"">CC-BY</a>")]
[InlineData("This is CC-BY hello", @"This is <a href=""https://creativecommons.org/licenses/by/4.0/"">CC-BY</a> hello")]
[InlineData("This is CC-BY-NC hello", @"This is <a href=""https://creativecommons.org/licenses/by-nc/4.0/"">CC-BY-NC</a> hello")]
[InlineData("This is CC-BY-NC hello and CC-BY", @"This is <a href=""https://creativecommons.org/licenses/by-nc/4.0/"">CC-BY-NC</a> hello and <a href=""https://creativecommons.org/licenses/by/4.0/"">CC-BY</a>")]
[InlineData("This is OGL and CC-BY-NC-SA and CC-BY-NC and OGL again",
@"This is <a href=""http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/"">OGL</a> and <a href=""https://creativecommons.org/licenses/by-nc-sa/4.0/"">CC-BY-NC-SA</a> and <a href=""https://creativecommons.org/licenses/by-nc/4.0/"">CC-BY-NC</a> and <a href=""http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/"">OGL</a> again")]
public void LicenseMap_Codes_Are_Replaced(string raw, string expected)
{
var processed = LicenceHelpers.GetUsageWithHtmlLinks(raw);

processed.Should().Be(expected);
}
}
53 changes: 53 additions & 0 deletions src/Wellcome.Dds/Utils.Tests/StringUtilsTests.cs
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
using System;
using FluentAssertions;
using Xunit;

@@ -144,5 +145,57 @@ public void Chomp_RemovesEndOfString()
// Assert
actual.Should().Be(expected);
}

[Theory]
[InlineData(null, "")]
[InlineData("", "")]
[InlineData("path", "path")]
[InlineData("/path", "path")]
[InlineData("https://example.org/some/path", "path")]
[InlineData("https://example.org/some/long/path", "https://example.org/some/long/path")]
[InlineData("https://example.org/some/xxx/path", "https://example.org/some/long/path")]
[InlineData("https://example.org/some/long/path", "https://example.org/some/long/path", 2)]
[InlineData("https://example.org/xxx/long/path", "https://example.org/some/long/path", 2)]
public void PathElements_Are_Equivalent(string path1, string path2, int walkback = 1)
{
StringUtils.EndWithSamePathElements(path1, path2, walkback).Should().BeTrue();
}

[Theory]
[InlineData(null, "xxx")]
[InlineData("", "xxx")]
[InlineData("path", "xxx")]
[InlineData("/path", "xxx")]
[InlineData("https://example.org/some/path", "xxx")]
[InlineData("https://example.org/some/long/path", "https://example.org/some/long/xxx")]
[InlineData("https://example.org/some/long/path", "https://example.org/some/xxx/path", 2)]
[InlineData("https://example.org/xxx/long/path", "https://example.org/some/long/path", 3)]
public void PathElements_Are_Not_Equivalent(string path1, string path2, int walkback = 1)
{
StringUtils.EndWithSamePathElements(path1, path2, walkback).Should().BeFalse();
}

[Fact]
public void GetFriendlyAge_Supports_UtcDateTime()
{
var dtUtcNow = DateTime.UtcNow;

var friendly = StringUtils.GetFriendlyAge(dtUtcNow);
var local = dtUtcNow.ToLocalTime().ToString("yyyy-MM-dd HH:mm:ss");

friendly.Should().StartWith(local);
}


[Fact]
public void GetFriendlyAge_Supports_NonUtcDateTime()
{
var dtNow = DateTime.Now;

var friendly = StringUtils.GetFriendlyAge(dtNow);
var local = dtNow.ToLocalTime().ToString("yyyy-MM-dd HH:mm:ss");

friendly.Should().StartWith(local);
}
}
}
1 change: 1 addition & 0 deletions src/Wellcome.Dds/Utils.Tests/Utils.Tests.csproj
Original file line number Diff line number Diff line change
@@ -27,6 +27,7 @@

<ItemGroup>
<ProjectReference Include="..\Utils\Utils.csproj" />
<ProjectReference Include="..\Wellcome.Dds.Repositories\Wellcome.Dds.Repositories.csproj" />
</ItemGroup>

</Project>
88 changes: 43 additions & 45 deletions src/Wellcome.Dds/Utils/Caching/BinaryObjectCache.cs
Original file line number Diff line number Diff line change
@@ -14,7 +14,7 @@ public class BinaryObjectCache<T> : IBinaryObjectCache<T>
private readonly ILogger<BinaryObjectCache<T>> logger;
private readonly BinaryObjectCacheOptions options;
private readonly IStorage storage;
private readonly IMemoryCache memoryCache;
private readonly IMemoryCache? memoryCache;
private readonly TimeSpan cacheDuration;
private bool memoryCacheEnabled = true;

@@ -24,11 +24,11 @@ public BinaryObjectCache(
ILogger<BinaryObjectCache<T>> logger,
IOptions<BinaryObjectCacheOptionsByType> binaryObjectCacheOptionsByType,
IStorage storage,
IMemoryCache memoryCache
IMemoryCache? memoryCache
)
{
this.logger = logger;
options = binaryObjectCacheOptionsByType.Value[typeof(T).FullName];
options = binaryObjectCacheOptionsByType.Value[typeof(T).FullName!];
this.storage = storage;
this.memoryCache = memoryCache;
cacheDuration = TimeSpan.FromSeconds(this.options.MemoryCacheSeconds);
@@ -50,19 +50,19 @@ public Task DeleteCacheFile(string key)
return storage.DeleteCacheFile(options.Container, fileName);
}

public Task<T> GetCachedObject(string key, Func<Task<T>> getFromSource)
public Task<T?> GetCachedObject(string key, Func<Task<T?>> getFromSource)
=> GetCachedObject(key, getFromSource, null);

public Task<T> GetCachedObject(string key, Func<Task<T>> getFromSource, Predicate<T> storedVersionIsStale)
public Task<T?> GetCachedObject(string key, Func<Task<T?>>? getFromSource, Predicate<T?>? storedVersionIsStale)
=> GetCachedObject(key, getFromSource, storedVersionIsStale, true);

public Task<T> GetCachedObjectFromLocal(string key, Func<Task<T>> getFromSource)
public Task<T?> GetCachedObjectFromLocal(string key, Func<Task<T?>> getFromSource)
=> GetCachedObject(key, getFromSource, null, false);

private async Task<T> GetCachedObject(string key, Func<Task<T>> getFromSource,
Predicate<T> storedVersionIsStale, bool readFromStorage)
private async Task<T?> GetCachedObject(string key, Func<Task<T?>>? getFromSource,
Predicate<T?>? storedVersionIsStale, bool readFromStorage)
{
T t = default;
T? t = default;
if (options.AvoidCaching)
{
if (getFromSource == null) return t;
@@ -79,7 +79,7 @@ private async Task<T> GetCachedObject(string key, Func<Task<T>> getFromSource,

var memoryCacheKey = GetMemoryCacheKey(key);

if (memoryCache != null && memoryCacheEnabled)
if (memoryCacheEnabled)
{
t = memoryCache.Get(memoryCacheKey) as T;
}
@@ -89,56 +89,54 @@ private async Task<T> GetCachedObject(string key, Func<Task<T>> getFromSource,
bool memoryCacheMiss = false;
var cachedFile = GetCachedFile(key);

using (var processLock = await GetLock(key))
using var processLock = await GetLock(key);
// check in memoryCache cache again
if (memoryCacheEnabled)
{
// check in memoryCache cache again
if (memoryCache != null && memoryCacheEnabled)
{
t = memoryCache.Get(memoryCacheKey) as T;
}
t = memoryCache.Get(memoryCacheKey) as T;
}

if (t == null)
if (t == null)
{
memoryCacheMiss = true;
if (readFromStorage)
{
memoryCacheMiss = true;
if (readFromStorage)
if (logger.IsEnabled(LogLevel.Debug))
{
if (logger.IsEnabled(LogLevel.Debug))
{
logger.LogDebug("Cache MISS for {MemoryCacheKey}, will attempt read from disk",
memoryCacheKey);
}

t = await storage.Read<T>(cachedFile);
logger.LogDebug("Cache MISS for {MemoryCacheKey}, will attempt read from disk",
memoryCacheKey);
}

t = await storage.Read<T>(cachedFile);
}
}

if (t != null && storedVersionIsStale != null && storedVersionIsStale(t))
if (t != null && storedVersionIsStale != null && storedVersionIsStale(t))
{
t = null;
}

if (t == null)
{
if (logger.IsEnabled(LogLevel.Debug))
{
t = null;
logger.LogDebug("Disk MISS for {MemoryCacheKey}, will attempt read from source",
memoryCacheKey);
}

if (t == null)
if (getFromSource != null)
{
if (logger.IsEnabled(LogLevel.Debug))
t = await getFromSource();
if (t != null)
{
logger.LogDebug("Disk MISS for {MemoryCacheKey}, will attempt read from source",
memoryCacheKey);
}

if (getFromSource != null)
{
t = await getFromSource();
if (t != null)
{
await storage.Write(t, cachedFile, options.WriteFailThrowsException);
}
await storage.Write(t, cachedFile, options.WriteFailThrowsException);
}
}
}

if (t != null && memoryCacheMiss && memoryCache != null && memoryCacheEnabled)
{
PutInMemoryCache(t, memoryCacheKey);
}
if (t != null && memoryCacheMiss && memoryCacheEnabled)
{
PutInMemoryCache(t, memoryCacheKey);
}

return t;
2 changes: 2 additions & 0 deletions src/Wellcome.Dds/Utils/Caching/BinaryObjectCacheOptions.cs
Original file line number Diff line number Diff line change
@@ -6,6 +6,8 @@ public class BinaryObjectCacheOptionsByType : Dictionary<string, BinaryObjectCac

public class BinaryObjectCacheOptions
{
#nullable disable

public bool AvoidCaching { get; set; }
public bool AvoidSaving { get; set; }
public bool WriteFailThrowsException { get; set; }
6 changes: 3 additions & 3 deletions src/Wellcome.Dds/Utils/Caching/IBinaryObjectCache.cs
Original file line number Diff line number Diff line change
@@ -6,14 +6,14 @@ namespace Utils.Caching
{
public interface IBinaryObjectCache<T> where T : class
{
Task<T> GetCachedObject(string key, Func<Task<T>> getFromSource);
Task<T?> GetCachedObject(string key, Func<Task<T?>> getFromSource);

Task<T> GetCachedObject(string key, Func<Task<T>> getFromSource, Predicate<T> storedVersionIsStale);
Task<T?> GetCachedObject(string key, Func<Task<T?>>? getFromSource, Predicate<T?>? storedVersionIsStale);

/// <summary>
/// Get cached object from local cache only, do not attempt to read from backing store.
/// </summary>
Task<T> GetCachedObjectFromLocal(string key, Func<Task<T>> getFromSource);
Task<T?> GetCachedObjectFromLocal(string key, Func<Task<T?>> getFromSource);

ISimpleStoredFileInfo GetCachedFile(string key);

18 changes: 9 additions & 9 deletions src/Wellcome.Dds/Utils/HtmlUtils.cs
Original file line number Diff line number Diff line change
@@ -271,7 +271,7 @@ Match the character ">" literally «>»
/// </summary>
/// <param name="input">The HTML to clean</param>
/// <returns></returns>
public static string StripEmptyParagraphTags(string input)
public static string? StripEmptyParagraphTags(string? input)
{
if (input != null)
{
@@ -289,7 +289,7 @@ public static string StripEmptyParagraphTags(string input)
/// </summary>
/// <param name="input">The HTML to clean</param>
/// <returns></returns>
public static string StripEmptyOrNbspParagraphTags(string input)
public static string? StripEmptyOrNbspParagraphTags(string? input)
{
if (input != null)
{
@@ -305,7 +305,7 @@ public static string StripEmptyOrNbspParagraphTags(string input)
/// </summary>
/// <param name="markup"></param>
/// <returns></returns>
public static string TextOnly(string markup)
public static string? TextOnly(string? markup)
{
if (markup != null)
{
@@ -314,7 +314,7 @@ public static string TextOnly(string markup)
return null;
}

public static string TextOnlyWithSpaces(string markup)
public static string? TextOnlyWithSpaces(string? markup)
{
if (markup != null)
{
@@ -333,7 +333,7 @@ public static string TextOnlyWithSpaces(string markup)
/// Usually, the CSS won't be designed for that scenario and will be adding its own borders/margins
/// to images in text.
/// </summary>
public static string StripParagraphTagsAroundImages(string input)
public static string? StripParagraphTagsAroundImages(string? input)
{
if (input != null)
{
@@ -351,7 +351,7 @@ public static string StripParagraphTagsAroundImages(string input)
/// This is fine in a long segment of body text but for some more tightly controlled designs
/// (e.g., ImageCallout) the CSS isn't designed to style extra p tags.
/// </summary>
public static string StripParagraphTagsAroundAnchorTags(string input)
public static string? StripParagraphTagsAroundAnchorTags(string? input)
{
if (input != null)
{
@@ -368,7 +368,7 @@ public static string StripParagraphTagsAroundAnchorTags(string input)
/// <param name="input">An HTML string</param>
/// <param name="elementRegex">a Regular Expression that will match the HTML element</param>
/// <returns></returns>
public static string StripParagraphTagsAroundElement(string input, string elementRegex)
public static string? StripParagraphTagsAroundElement(string? input, string elementRegex)
{
if (input != null)
{
@@ -451,7 +451,7 @@ public static string AddBreakTags(string s)
/// </summary>
/// <param name="raw">A string of html (or plain text)</param>
/// <returns>html with at least one p tag - i.e., no "free-standing" text</returns>
public static string EnsureParagraphTags(string raw)
public static string? EnsureParagraphTags(string? raw)
{
if (raw == null) return null;
if (raw.IndexOf("<p", System.StringComparison.Ordinal) == -1)
@@ -468,7 +468,7 @@ public static string EnsureParagraphTags(string raw)
/// </summary>
/// <param name="raw"></param>
/// <returns></returns>
public static string ConvertDoubleBreakTagsToParas(string raw)
public static string? ConvertDoubleBreakTagsToParas(string? raw)
{
// don't do this if the supplied string already contains paragraphs
if (raw == null) return null;
20 changes: 10 additions & 10 deletions src/Wellcome.Dds/Utils/Link.cs
Original file line number Diff line number Diff line change
@@ -4,9 +4,9 @@
namespace Utils
{
/// <summary>
/// Represention of the properties of an anchor tag (hyperlink element) - its attributes and inner text.
/// Representation of the properties of an anchor tag (hyperlink element) - its attributes and inner text.
///
/// This is useful in databinding scenarios, and in manipulating HTML.
/// This is useful in data-binding scenarios, and in manipulating HTML.
///
/// Do not add any properties to this class that do not directly represent attributes of a link element
/// This represents a hyperlink HTML element (anchor tag), it is NOT a lightweight IItem
@@ -18,20 +18,20 @@ namespace Utils
[Serializable]
public class Link : IComparable<Link>
{
public string Href { get; set; }
public string Text { get; set; }
public string CssClass { get; set; }
public string Rel { get; set; }
public string Title { get; set; }
public string? Href { get; set; }
public string? Text { get; set; }
public string? CssClass { get; set; }
public string? Rel { get; set; }
public string? Title { get; set; }

public int CompareTo(Link other)
public int CompareTo(Link? other)
{
return String.CompareOrdinal(Text, other.Text);
return String.CompareOrdinal(Text, other?.Text);
}

public override string ToString()
{
return String.Format("[{0} - {1}]", Href, Text);
return $"[{Href} - {Text}]";
}

public string ToHtml()
68 changes: 68 additions & 0 deletions src/Wellcome.Dds/Utils/Logging/BatchMetrics.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
using System.Diagnostics;

namespace Utils.Logging;

public class BatchMetrics
{
public int BatchCounter { get; set; }
public long LastBatchTime { get; set; }
public long TotalTime { get; set; }
public long MinBatchTime { get; set; }
public int MinBatchCount { get; set; }
public long MaxBatchTime { get; set; }
public int MaxBatchCount { get; set; }

public int BatchSize { get; set; }

public long AverageBatchTime => TotalTime / BatchCounter;

public string Summary => $"Total batches: {BatchCounter}; " +
$"Avg batch time {AverageBatchTime}; " +
$"Min batch time {MinBatchTime} ({MinBatchCount} items); " +
$"Max Batch time {MaxBatchTime} ({MaxBatchCount} items)";

private readonly Stopwatch stopwatch;

public BatchMetrics()
{
stopwatch = new Stopwatch();
}

public void BeginBatch(int batchSize = -1)
{
stopwatch.Restart();
BatchCounter++;
BatchSize = batchSize;
}

public void EndBatch(int batchSize = -1)
{
if (batchSize > 0)
{
// you might not know BatchSize until you end the batch
BatchSize = batchSize;
}
LastBatchTime = stopwatch.ElapsedMilliseconds;
if (BatchCounter == 1)
{
MinBatchTime = LastBatchTime;
MinBatchCount = BatchSize;
MaxBatchTime = LastBatchTime;
MaxBatchCount = BatchSize;
}
else
{
if (LastBatchTime < MinBatchTime)
{
MinBatchTime = LastBatchTime;
MinBatchCount = BatchSize;
}
else if (LastBatchTime >= MaxBatchTime)
{
MaxBatchTime = LastBatchTime;
MaxBatchCount = BatchSize;
}
}
TotalTime += LastBatchTime;
}
}
2 changes: 1 addition & 1 deletion src/Wellcome.Dds/Utils/Logging/LoggingEvent.cs
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ namespace Utils.Logging

public class LoggingEvent
{
public string Message { get; set; }
public string? Message { get; set; }
public long Split { get; set; }
public long Total { get; set; }

4 changes: 2 additions & 2 deletions src/Wellcome.Dds/Utils/Logging/SmallJobLogger.cs
Original file line number Diff line number Diff line change
@@ -16,10 +16,10 @@ public List<Tuple<long, long, string>> GetEvents()
}

// this allows you to supply a message to your preferred logging mechanism
private readonly Action<string> callback;
private readonly Action<string>? callback;
private readonly string callbackPrefix;

public SmallJobLogger(string callbackPrefix, Action<string> callback)
public SmallJobLogger(string callbackPrefix, Action<string>? callback)
{
this.callbackPrefix = callbackPrefix;
this.callback = callback;
8 changes: 6 additions & 2 deletions src/Wellcome.Dds/Utils/PathStringUtils.cs
Original file line number Diff line number Diff line change
@@ -13,11 +13,15 @@ public static class PathStringUtils
/// </summary>
/// <param name="path"></param>
/// <returns></returns>
public static string GetSimpleNameFromPath(string path)
public static string? GetSimpleNameFromPath(string? path)
{
if (path.IsNullOrWhiteSpace())
{
return String.Empty;
}
if (path.EndsWith("/"))
{
path = path.Substring(0, path.Length - 1);
path = path[..^1];
}
int spos = path.LastIndexOf("/", StringComparison.Ordinal) + 1;
var simpleName = path.Substring(spos);
27 changes: 18 additions & 9 deletions src/Wellcome.Dds/Utils/SequenceExtensions.cs
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Linq;

namespace Utils
@@ -33,13 +34,13 @@ public static IEnumerable<TSource> DistinctBy<TSource, TKey>(
/// <typeparam name="T"></typeparam>
/// <param name="enumerable"></param>
/// <returns></returns>
public static bool IsNullOrEmpty<T>(this IEnumerable<T> enumerable)
public static bool IsNullOrEmpty<T>([NotNullWhen(false)] this IEnumerable<T>? enumerable)
=> enumerable == null || !enumerable.Any();

/// <summary>
/// Does the sequence contain anything (allows null sequence)?
/// </summary>
public static bool IsNullOrEmpty<T>(this IList<T> enumerable)
public static bool IsNullOrEmpty<T>([NotNullWhen(false)] this IList<T>? enumerable)
=> enumerable == null || enumerable.Count == 0;

/// <summary>
@@ -48,28 +49,36 @@ public static bool IsNullOrEmpty<T>(this IList<T> enumerable)
/// <typeparam name="T"></typeparam>
/// <param name="enumerable"></param>
/// <returns></returns>
public static bool HasItems<T>(this IEnumerable<T> enumerable) => !enumerable.IsNullOrEmpty();
public static bool HasItems<T>([NotNullWhen(true)] this IEnumerable<T>? enumerable) => !enumerable.IsNullOrEmpty();

/// <summary>
/// Does the sequence contain anything (allows null sequence)?
/// </summary>
public static bool HasItems<T>(this IList<T> enumerable) => !enumerable.IsNullOrEmpty();
public static bool HasItems<T>([NotNullWhen(true)] this IList<T>? enumerable) => !enumerable.IsNullOrEmpty();


public static IEnumerable<T> AnyItems<T>(this IEnumerable<T>? items)
{
if (items == null)
{
return Enumerable.Empty<T>();
}

return items;
}
/// <summary>
/// Generate collection of IEnumerables of specified size.
/// Generate collection of IEnumerable of specified size.
/// </summary>
/// <remarks>From morelinq. Consider importing whole library</remarks>
public static IEnumerable<IEnumerable<T>> Batch<T>(
this IEnumerable<T> source, int size)
{
T[] bucket = null;
T[]? bucket = null;
var count = 0;

foreach (var item in source)
{
if (bucket == null)
bucket = new T[size];

bucket ??= new T[size];
bucket[count++] = item;

if (count != size)
12 changes: 9 additions & 3 deletions src/Wellcome.Dds/Utils/Storage/FileSystem/FileSystemStorage.cs
Original file line number Diff line number Diff line change
@@ -32,10 +32,12 @@ public Task DeleteCacheFile(string container, string fileName)
return Task.CompletedTask;
}

public async Task<T> Read<T>(ISimpleStoredFileInfo fileInfo) where T : class
public Task<T?> Read<T>(ISimpleStoredFileInfo fileInfo) where T : class
{
throw new NotSupportedException("File System Storage is not supported until re-written for protobuf");
/*
if (!await fileInfo.DoesExist()) return null;
T t = default(T);
var t = default(T);
try
{
IFormatter formatter = new BinaryFormatter();
@@ -55,10 +57,13 @@ public async Task<T> Read<T>(ISimpleStoredFileInfo fileInfo) where T : class
logger.LogError("Attempt to deserialize '" + fileInfo.Uri + "' from disk failed", ex);
}
return t;
*/
}

public async Task Write<T>(T t, ISimpleStoredFileInfo fileInfo, bool writeFailThrowsException) where T : class
public Task Write<T>(T t, ISimpleStoredFileInfo fileInfo, bool writeFailThrowsException) where T : class
{
throw new NotSupportedException("File System Storage is not supported until re-written for protobuf");
/*
logger.LogInformation("Writing cache file '" + fileInfo.Uri + "' to disk");
try
{
@@ -78,6 +83,7 @@ public async Task Write<T>(T t, ISimpleStoredFileInfo fileInfo, bool writeFailTh
throw;
}
}
*/
}

public Task<Stream?> GetStream(string container, string fileName)
Original file line number Diff line number Diff line change
@@ -18,7 +18,7 @@ public FileSystemStoredFileInfo(FileInfo fileInfo)

public Task<DateTime?> GetLastWriteTime() => Task.FromResult<DateTime?>(fileInfo.LastWriteTime);

public string Container => fileInfo.DirectoryName;
public string? Container => fileInfo.DirectoryName;

public string Path => fileInfo.Name;
}
2 changes: 1 addition & 1 deletion src/Wellcome.Dds/Utils/Storage/ISimpleStoredFile.cs
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@ public interface ISimpleStoredFileInfo
Task<DateTime?> GetLastWriteTime();

// add folder/key idea here...
public string Container { get; }
public string? Container { get; }

public string Path { get; }
}
2 changes: 1 addition & 1 deletion src/Wellcome.Dds/Utils/Storage/IStorage.cs
Original file line number Diff line number Diff line change
@@ -22,7 +22,7 @@ public interface IStorage
/// <summary>
/// Read object represented by <see cref="ISimpleStoredFileInfo"/>
/// </summary>
Task<T> Read<T>(ISimpleStoredFileInfo fileInfo) where T : class;
Task<T?> Read<T>(ISimpleStoredFileInfo fileInfo) where T : class;

Task<Stream?> GetStream(string container, string fileName);
}
2 changes: 1 addition & 1 deletion src/Wellcome.Dds/Utils/StreamUtils.cs
Original file line number Diff line number Diff line change
@@ -14,7 +14,7 @@ public static async Task CopyToAsync (
this Stream source,
Stream destination,
int bufferSize,
IProgress<long> progress = null,
IProgress<long>? progress = null,
CancellationToken cancellationToken = default (CancellationToken))
{
if (bufferSize < 0)
202 changes: 153 additions & 49 deletions src/Wellcome.Dds/Utils/StringUtils.cs
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.Linq;
using System.Text;
using System.Text.RegularExpressions;
@@ -8,7 +9,14 @@ namespace Utils
{
public static class StringUtils
{
public static bool IsNullOrWhiteSpace(this string s)
private static readonly string[] FileSizeSuffixes;
static StringUtils()
{
//Longs run out around EB
FileSizeSuffixes = new[] { "B", "KB", "MB", "GB", "TB", "PB", "EB" };
}

public static bool IsNullOrWhiteSpace([NotNullWhen(false)] this string? s)
{
return string.IsNullOrWhiteSpace(s);
}
@@ -19,9 +27,9 @@ public static bool IsNullOrWhiteSpace(this string s)
/// <remarks>
/// This may seem trivial but it helps code readability.
/// </remarks>
/// <param name="s"></param>
/// <param name="str"></param>
/// <returns></returns>
public static bool HasText(this string s) => !string.IsNullOrWhiteSpace(s);
public static bool HasText([NotNullWhen(true)] this string? str) => !string.IsNullOrWhiteSpace(str);

/// <summary>
/// Removes separator from the start of str if it's there, otherwise leave it alone.
@@ -33,7 +41,7 @@ public static bool IsNullOrWhiteSpace(this string s)
/// <param name="str"></param>
/// <param name="start"></param>
/// <returns></returns>
public static string RemoveStart(this string str, string start)
public static string? RemoveStart(this string? str, string start)
{
if (str == null) return null;
if (str == string.Empty) return string.Empty;
@@ -46,17 +54,14 @@ public static string RemoveStart(this string str, string start)
return str;
}

public static DateTime? GetNullableDateTime(string s)
public static DateTime? GetNullableDateTime(string? s)
{
DateTime date;
if (DateTime.TryParse(s, out date))
if (DateTime.TryParse(s, out var date))
{
return date;
}
else
{
return null;
}

return null;
}


@@ -81,7 +86,7 @@ public static string ToAlphanumeric(this string s)

/// <summary>
/// remove leading and trailing characters that are not alphanumeric
/// TODO - imporve this, not very efficient
/// TODO - improve this, not very efficient
/// </summary>
/// <param name="s"></param>
/// <returns></returns>
@@ -140,7 +145,7 @@ public static string ToAlphanumericOrUnderscore(this string s)
/// <param name="s"></param>
/// <param name="exceptions"></param>
/// <returns></returns>
public static string ToAlphanumericOrWhitespace(this string s, char[] exceptions = null)
public static string ToAlphanumericOrWhitespace(this string s, char[]? exceptions = null)
{
var sb = new StringBuilder();
foreach (char c in s)
@@ -190,12 +195,12 @@ public static string ToNumber(this string s)
/// <param name="source"></param>
/// <param name="delimiter"></param>
/// <returns></returns>
public static string[] SplitByDelimiterIntoArray(this string source, char delimiter)
public static string[] SplitByDelimiterIntoArray(this string? source, char delimiter)
{
var strings = SplitByDelimiter(source, delimiter);
if (strings == null)
{
return new string[0];
return Array.Empty<string>();
}
return strings.ToArray();
}
@@ -207,15 +212,13 @@ public static string[] SplitByDelimiterIntoArray(this string source, char delimi
/// <param name="source"></param>
/// <param name="delimiter"></param>
/// <returns></returns>
public static IEnumerable<string> SplitByDelimiter(this string source, char delimiter)
public static IEnumerable<string>? SplitByDelimiter(this string? source, char delimiter)
{
if (!String.IsNullOrEmpty(source))
{
// this trims whitespace by default - implement another one if required
var strings = source.Split(new[] { delimiter });
return strings.Where(s => s.HasText()).Select(s => s.Trim());
}
return null;
if (string.IsNullOrEmpty(source)) return null;

// this trims whitespace by default - implement another one if required
var strings = source.Split(new[] { delimiter });
return strings.Where(s => s.HasText()).Select(s => s.Trim());
}


@@ -263,18 +266,33 @@ public static string NormaliseSpaces(this string s)
///
/// </summary>
/// <param name="sizeInBytes"></param>
/// <param name="withSpace">include a space between number and unit</param>
/// <returns></returns>
public static string FormatFileSize(long sizeInBytes)
public static string FormatFileSize(long sizeInBytes, bool withSpace = false)
{
string[] suf = { "B", "KB", "MB", "GB", "TB", "PB", "EB" }; //Longs run out around EB
var spacer = withSpace ? " " : "";
if (sizeInBytes == 0)
return "0" + suf[0];
return "0" + spacer + FileSizeSuffixes[0];
long bytes = Math.Abs(sizeInBytes);
int place = Convert.ToInt32(Math.Floor(Math.Log(bytes, 1024)));
double num = Math.Round(bytes / Math.Pow(1024, place), 1);
return (Math.Sign(sizeInBytes) * num) + suf[place];
return (Math.Sign(sizeInBytes) * num) + spacer + FileSizeSuffixes[place];
}

/// <summary>
/// Create a nice display format for file size given a raw string value
/// </summary>
/// <param name="rawSize"></param>
/// <param name="withSpace"></param>
/// <returns></returns>
public static string? FormatFileSize(string? rawSize, bool withSpace = false)
{
if (long.TryParse(rawSize, out var asLong))
{
return FormatFileSize(asLong, withSpace);
}
return rawSize;
}

/// <summary>
/// like String.Replace, but only replaces the first instance of search in str
@@ -307,37 +325,57 @@ public static string GetFriendlyAge(DateTime? dtn)
return "(no date)";
}

public static string GetLocalDate(DateTime dt)
{
DateTime localTime = dt.ToLocalTime();
return localTime.ToString("yyyy-MM-dd HH:mm:ss");
}

public static string GetLocalDate(DateTime? dt)
{
if (!dt.HasValue)
{
return "";
}

return GetLocalDate(dt.Value);
}

public static string GetFriendlyAge(DateTime dt)
{
DateTime dttz = dt.ToLocalTime();
var s = dttz.ToString("yyyy-MM-dd HH:mm:ss") + " (";
DateTime localTime = dt.ToLocalTime();
var s = localTime.ToString("yyyy-MM-dd HH:mm:ss") + " (";
var dtNow = DateTime.Now;
if (dttz.Date == dtNow.Date)
if (localTime.Date == dtNow.Date)
{
s += "today";
}
else if (dttz.Date == dtNow.AddDays(-1).Date)
else if (localTime.Date == dtNow.AddDays(-1).Date)
{
s += "yesterday";
}
else
{
var td = (dtNow.Date - dttz).TotalDays;
var td = (dtNow.Date - localTime).TotalDays;
var d = Math.Ceiling(td);
s += d + " days ago";
}
return s + ")";
}

public static string GetFileName(this string s)
public static string? GetFileName(this string s)
{
var parts = s.Split(new [] {'/', '\\'});
return parts.LastOrDefault();
}

public static string GetFileExtension(this string s)
{
var fn = GetFileName(s);
string? fn = GetFileName(s);
if (fn.IsNullOrWhiteSpace())
{
return String.Empty;
}
var idx = fn.LastIndexOf('.');
if (idx != -1 && fn.Length > idx)
{
@@ -417,7 +455,7 @@ public static string SummariseWithEllipsis(this string fullField, int maxChars)
/// </summary>
/// <param name="strings"></param>
/// <returns></returns>
public static bool AllHaveText(params string[] strings)
public static bool AllHaveText(params string?[] strings)
{
return strings.AllHaveText();
}
@@ -427,9 +465,9 @@ public static bool AllHaveText(params string[] strings)
/// </summary>
/// <param name="strings"></param>
/// <returns></returns>
public static bool AllHaveText(this IEnumerable<string> strings)
public static bool AllHaveText(this IEnumerable<string?> strings)
{
foreach (string s in strings)
foreach (string? s in strings)
{
if (!HasText(s)) return false;
}
@@ -441,16 +479,9 @@ public static bool AllHaveText(this IEnumerable<string> strings)
/// </summary>
/// <param name="tests"></param>
/// <returns></returns>
public static bool AnyHaveText(params string[] tests)
public static bool AnyHaveText(params string?[] tests)
{
foreach (string s in tests)
{
if (s.HasText())
{
return true;
}
}
return false;
return tests.Any(s => s.HasText());
}

/// <summary>
@@ -504,10 +535,25 @@ public static string ReplaceFromDictionary(this string s, Dictionary<string, str
// https://stackoverflow.com/a/14033595
return dict.Aggregate(s, (current, kvp) => current.Replace(kvp.Key, kvp.Value));
}

public static string ReplaceFromDictionary(this string s, Dictionary<string, string> dict, string template)
{
// https://stackoverflow.com/a/14033595
return dict.Aggregate(s, (current, kvp) => current.Replace(kvp.Key, string.Format(template, kvp.Key, kvp.Value)));
// return dict.Aggregate(s, (current, kvp) => current.Replace(kvp.Key, string.Format(template, kvp.Key, kvp.Value)));

var byLength = dict.OrderByDescending(kvp => kvp.Key.Length).ToArray();
for (var index = 0; index < byLength.Length; index++)
{
var pair = byLength[index];
s = s.Replace(pair.Key, $"%%${index}$%%");
}

for (var index = 0; index < byLength.Length; index++)
{
var pair = byLength[index];
s = s.Replace($"%%${index}$%%", string.Format(template, pair.Key, pair.Value));
}
return s;
}

/// <summary>
@@ -517,9 +563,9 @@ public static string ReplaceFromDictionary(this string s, Dictionary<string, str
/// <param name="arguments"></param>
/// <param name="permittedOperations"></param>
/// <returns></returns>
public static (string operation, string parameter) GetOperationAndParameter(string[] arguments, string[] permittedOperations)
public static (string? operation, string? parameter) GetOperationAndParameter(string[] arguments, string[] permittedOperations)
{
string operation = null, parameter = null;
string? operation = null, parameter = null;
for (int i = 0; i < arguments.Length; i++)
{
var op = arguments[i];
@@ -536,6 +582,64 @@ public static (string operation, string parameter) GetOperationAndParameter(stri
return (operation, parameter);
}

public static int? ToNullableInt(this string? s)
{
if (int.TryParse(s, out var i)) return i;
return null;
}

public static double? ToNullableDouble(this string? s)
{
if (double.TryParse(s, out var i)) return i;
return null;
}


private static readonly Regex Roman = new Regex(
"^M{0,4}(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$",
RegexOptions.IgnoreCase);

public static bool IsRomanNumeral(string s)
{
return Roman.IsMatch(s);
}

public static string ToCommaDelimitedList(this IEnumerable<string>? strings)
{
if (strings == null)
{
return String.Empty;
}

return String.Join(',', strings);
}

/// <summary>
/// To be used carefully when comparing raw strings and RESt resources
/// </summary>
/// <param name="s1"></param>
/// <param name="s2"></param>
/// <param name="countBack">How many path elements at the end to consider when matching</param>
/// <returns></returns>
public static bool EndWithSamePathElements(string? s1, string? s2, int countBack = 1)
{
if (s1 == s2) return true;
if (s1.IsNullOrEmpty() && s2.IsNullOrEmpty()) return true;
var parts1 = s1.SplitByDelimiterIntoArray('/');
var parts2 = s2.SplitByDelimiterIntoArray('/');
if (parts1.Length < countBack || parts2.Length < countBack)
{
return false;
}
for (int i = 1; i <= countBack; i++)
{
if (parts1[^i] != parts2[^i])
{
return false;
}
}

return true;
}
}
}
13 changes: 9 additions & 4 deletions src/Wellcome.Dds/Utils/Threading/AsyncKeyedLock.cs
Original file line number Diff line number Diff line change
@@ -10,13 +10,13 @@ public sealed class AsyncKeyedLock
public IDisposable Lock(object key)
{
GetOrCreate(key).Wait();
return new Releaser { Key = key };
return new Releaser(key);
}

public async Task<IDisposable> LockAsync(object key)
{
await GetOrCreate(key).WaitAsync();
return new Releaser { Key = key };
return new Releaser(key);
}

public async Task<IDisposable> LockAsync(object key, TimeSpan timeout, bool throwIfNoLock = false)
@@ -28,12 +28,12 @@ public async Task<IDisposable> LockAsync(object key, TimeSpan timeout, bool thro
$"Unable to attain lock for {key} within timeout of {timeout.TotalMilliseconds}ms");
}

return new Releaser { Key = key, HaveLock = success};
return new Releaser(key) { HaveLock = success };
}

private SemaphoreSlim GetOrCreate(object key)
{
RefCounted<SemaphoreSlim> item;
RefCounted<SemaphoreSlim>? item;
lock (SemaphoreSlims)
{
if (SemaphoreSlims.TryGetValue(key, out item))
@@ -65,6 +65,11 @@ public RefCounted(T value)

public sealed class Releaser : IDisposable
{
public Releaser(object key)
{
Key = key;
}

public object Key { get; set; }

public bool HaveLock { get; set; } = true;
2 changes: 1 addition & 1 deletion src/Wellcome.Dds/Utils/Threading/TaskEx.cs
Original file line number Diff line number Diff line change
@@ -14,7 +14,7 @@ public static class TaskEx
/// <param name="throwOnTimeout">If true, throw <exception cref="TimeoutException">TimeoutException</exception>
/// if timeout exceeded. If false return default{T}</param>
/// <returns>Result of awaited task, or default{T}</returns>
public static async Task<T> TimeoutAfter<T>(this Task<T> task, int millisecondsTimeout,
public static async Task<T?> TimeoutAfter<T>(this Task<T> task, int millisecondsTimeout,
bool throwOnTimeout = false)
{
using (var cts = new CancellationTokenSource())
1 change: 1 addition & 0 deletions src/Wellcome.Dds/Utils/Utils.csproj
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@
<PropertyGroup>
<TargetFramework>net6.0</TargetFramework>
<LangVersion>latestmajor</LangVersion>
<Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
14 changes: 7 additions & 7 deletions src/Wellcome.Dds/Utils/XmlUtils.cs
Original file line number Diff line number Diff line change
@@ -12,24 +12,24 @@ public static XElement GetSingleElementWithAttribute(this XElement xel, XName el
{
return xel
.Elements(elementName)
.Single(x => (string)x.Attribute(attributeName) == attributeValue);
.Single(x => (string?)x.Attribute(attributeName) == attributeValue);
}

public static XElement GetSingleDescendantWithAttribute(this XElement xel, XName elementName, string attributeName, string attributeValue)
{
return xel
.Descendants(elementName)
.Single(x => (string)x.Attribute(attributeName) == attributeValue);
.Single(x => (string?)x.Attribute(attributeName) == attributeValue);
}

public static IEnumerable<XElement> GetAllDescendantsWithAttribute(this XElement xel, XName elementName, string attributeName, string attributeValue)
{
return xel
.Descendants(elementName)
.Where(x => (string)x.Attribute(attributeName) == attributeValue);
.Where(x => (string?)x.Attribute(attributeName) == attributeValue);
}

public static string GetDesendantElementValue(this XDocument xel, XName elementName)
public static string? GetDescendantElementValue(this XDocument xel, XName elementName)
{
try
{
@@ -45,7 +45,7 @@ public static string GetDesendantElementValue(this XDocument xel, XName elementN
}
return null;
}
public static string GetDesendantElementValue(this XElement xel, XName elementName)
public static string? GetDescendantElementValue(this XElement xel, XName elementName)
{
try
{
@@ -62,7 +62,7 @@ public static string GetDesendantElementValue(this XElement xel, XName elementNa
return null;
}

public static IEnumerable<string> GetDesendantElementValues(this XDocument xel, XName elementName)
public static IEnumerable<string> GetDescendantElementValues(this XDocument xel, XName elementName)
{
var els = xel.Descendants(elementName);
return els.Select(el => el.Value.Trim());
@@ -82,7 +82,7 @@ public static string GetRequiredAttributeValue(this XElement xel, XName attribut
return attr.Value;
}

public static string GetAttributeValue(this XElement xel, XName attributeName, string valueIfMissing)
public static string? GetAttributeValue(this XElement xel, XName attributeName, string? valueIfMissing)
{
if (xel == null)
{
62 changes: 62 additions & 0 deletions src/Wellcome.Dds/Wellcome.Dds.AssetDomain/AssetExtensions.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
using System.Linq;
using IIIF;
using Wellcome.Dds.AssetDomain.DigitalObjects;
using Wellcome.Dds.AssetDomain.Mets;

namespace Wellcome.Dds.AssetDomain;

public static class AssetExtensions
{
public static bool IsVideoMimeType(this string? mimeType)
{
return mimeType != null && mimeType.StartsWith("video/");
}

public static bool IsAudioMimeType(this string? mimeType)
{
return mimeType != null && mimeType.StartsWith("audio/");
}

public static bool IsTimeBasedMimeType(this string? mimeType)
{
return mimeType.IsVideoMimeType() || mimeType.IsAudioMimeType();
}

public static bool IsImageMimeType(this string? mimeType)
{
return mimeType != null && mimeType.StartsWith("image/");
}

public static bool IsTextMimeType(this string? mimeType)
{
return mimeType != null && mimeType.StartsWith("text/");
}

public static IStoredFile? GetDefaultFile(this IPhysicalFile asset)
{
return asset.Files!.SingleOrDefault(f => f.StorageIdentifier == asset.StorageIdentifier);
}

public static IProcessingBehaviour GetDefaultProcessingBehaviour(this IPhysicalFile asset)
{
var defaultStoredFile = asset.GetDefaultFile();
return defaultStoredFile!.ProcessingBehaviour;
}

public static Size? GetWhSize(this IPhysicalFile file)
{
var dimensions = file.AssetMetadata?.GetMediaDimensions();
if (dimensions == null) return null;

var w = dimensions.Width.GetValueOrDefault();
var h = dimensions.Height.GetValueOrDefault();
if (w > 0 && h > 0)
{
return new Size(w, h);
}

return null;
}


}

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

2,221 changes: 2,221 additions & 0 deletions src/Wellcome.Dds/Wellcome.Dds.AssetDomain/Data/pronom_map.json

Large diffs are not rendered by default.

32 changes: 32 additions & 0 deletions src/Wellcome.Dds/Wellcome.Dds.AssetDomain/DeliveredFile.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
namespace Wellcome.Dds.AssetDomain;

/// <summary>
/// Represents files and derivatives of files served by DLCS
/// </summary>
public class DeliveredFile
{
public string? PublicUrl { get; set; }
public string? DlcsUrl { get; set; }
public string? DeliveryChannel { get; set; }

public string? MediaType { get; set; }
public int? Width { get; set; }
public int? Height { get; set; }
public double? Duration { get; set; }

public string GetSummary()
{
var s = MediaType;
if (Duration.HasValue)
{
s += $"\nDuration: {Duration}";
}

if (Height.HasValue)
{
s += $"\nWidth: {Width}, Height: {Height}";
}

return s ?? "(no summary information)";
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
using System.Collections.Generic;
using Wellcome.Dds.AssetDomain.Mets;

namespace Wellcome.Dds.AssetDomain.DigitalObjects
{
public interface IDigitalCollection : IDigitalObject
{
ICollection? MetsCollection { get; set; }
IEnumerable<IDigitalManifestation>? Manifestations { get; set; }
IEnumerable<IDigitalCollection>? Collections { get; set; }
}
}
Original file line number Diff line number Diff line change
@@ -3,27 +3,27 @@
using Wellcome.Dds.AssetDomain.Dlcs.Model;
using Wellcome.Dds.AssetDomain.Mets;

namespace Wellcome.Dds.AssetDomain.Dashboard
namespace Wellcome.Dds.AssetDomain.DigitalObjects
{
public interface IDigitisedManifestation : IDigitisedResource
public interface IDigitalManifestation : IDigitalObject
{
IManifestation MetsManifestation { get; set; }
IManifestation MetsManifestation { get; }

/// <summary>
/// The images held by the DLCS that match the metadata for this manifestation
/// based on string3
/// </summary>
IEnumerable<Image> DlcsImages { get; set; }
IEnumerable<Image>? DlcsImages { get; set; }
bool JobExactMatchForManifestation(DlcsIngestJob job);

/// <summary>
/// TODO: This doesn't belong here! Only here for PDF link to work and be same as in IIIF manifest
/// </summary>
// int SequenceIndex { get; set; }

string DlcsStatus { get; set; }
string DlcsResponse { get; set; }
string? DlcsStatus { get; set; }
string? DlcsResponse { get; set; }

IPdf PdfControlFile { get; set; }
IPdf? PdfControlFile { get; set; }
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
using Wellcome.Dds.Common;

namespace Wellcome.Dds.AssetDomain.DigitalObjects
{
public interface IDigitalObject
{
DdsIdentifier? Identifier { get; set; }
bool Partial { get; set; }
bool? InSyncWithDlcs { get; }
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
using System.Collections.Generic;
using System.Threading.Tasks;
using Wellcome.Dds.AssetDomain.Dlcs;
using Wellcome.Dds.AssetDomain.Dlcs.Ingest;
using Wellcome.Dds.AssetDomain.Dlcs.Model;
using Wellcome.Dds.AssetDomain.Mets;
using Wellcome.Dds.Common;

namespace Wellcome.Dds.AssetDomain.DigitalObjects
{
public interface IDigitalObjectRepository
{
Task<IDigitalObject> GetDigitalObject(DdsIdentifier identifier, DlcsCallContext dlcsCallContext, bool includePdfDetails = false);
Task<SyncOperation> GetDlcsSyncOperation(
IDigitalManifestation digitisedManifestation,
bool reIngestErrorImages,
DlcsCallContext dlcsCallContext);

Task ExecuteDlcsSyncOperation(SyncOperation syncOperation, bool usePriorityQueue, DlcsCallContext dlcsCallContext);
int DefaultSpace { get; }
Task<IEnumerable<DlcsIngestJob>> GetMostRecentIngestJobs(string identifier, int number);
//IEnumerable<DlcsIngestJob> GetUpdatedIngestJobs(string identifier, SyncOperation syncOperation);
Task<Batch?> GetBatch(string batchId, DlcsCallContext dlcsCallContext);

Task<JobActivity> GetRationalisedJobActivity(SyncOperation syncOperation, DlcsCallContext dlcsCallContext);

Task<IEnumerable<Batch>> GetBatchesForImages(IEnumerable<Image> images, DlcsCallContext dlcsCallContext);
Task<IEnumerable<ErrorByMetadata>> GetErrorsByMetadata(DlcsCallContext dlcsCallContext);
Task<Page<ErrorByMetadata>> GetErrorsByMetadata(int page, DlcsCallContext dlcsCallContext);

Task<int> FindSequenceIndex(string identifier);
Task<bool> DeletePdf(string identifier);
Task<int> RemoveOldJobs(string id);
Task<int> DeleteOrphans(string id, DlcsCallContext dlcsCallContext);

IngestAction LogAction(string manifestationId, int? jobId, string userName, string action, string? description = null);
IEnumerable<IngestAction> GetRecentActions(int count, string? user = null);
Task<Dictionary<string, long>> GetDlcsQueueLevel();


DeliveredFile[] GetDeliveredFiles(IPhysicalFile physicalFile);
DeliveredFile[] GetDeliveredFiles(IStoredFile? storedFile);

}
}
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
using System;

namespace Wellcome.Dds.AssetDomain.Dashboard
namespace Wellcome.Dds.AssetDomain.DigitalObjects
{
public interface IPdf
{
// The DLCS URL of the PDF
string Url { get; set; }
string? Url { get; set; }
// False if the DLCS hasn't started or finished making it yet
bool Exists { get; set; }
//true if the DLCS has started the creation process
@@ -15,7 +15,7 @@ public interface IPdf
// The deduced roles, from the constituent images, for the DLCS to enforce access control.
// The hard-coded rule is that open and clickthrough are included, anything else is replaced by a placeholder page.
// So this in practice for Wellcome will either be empty, or clickthrough.
string[] Roles { get; set; }
string[]? Roles { get; set; }

int PageCount { get; set; }
long SizeBytes { get; set; }
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
using System.Collections.Generic;
using IIIF;
using Wellcome.Dds.AssetDomain.Dlcs;

namespace Wellcome.Dds.AssetDomain.DigitalObjects;

public interface IProcessingBehaviour
{
HashSet<string> DeliveryChannels { get; }
string? ImageOptimisationPolicy { get; }
Size? GetVideoSize(string deliveryChannel);
AssetFamily AssetFamily { get; }
}
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@
using System.Threading;
using System.Threading.Tasks;

namespace Wellcome.Dds.AssetDomain.Dashboard
namespace Wellcome.Dds.AssetDomain.DigitalObjects
{
public interface IStatusProvider
{
Original file line number Diff line number Diff line change
@@ -2,12 +2,17 @@
using Wellcome.Dds.AssetDomain.Dlcs.Ingest;
using Wellcome.Dds.AssetDomain.Dlcs.Model;

namespace Wellcome.Dds.AssetDomain.Dashboard
namespace Wellcome.Dds.AssetDomain.DigitalObjects
{
public class JobActivity
{
public JobActivity(List<Batch> batchesForCurrentImages, List<DlcsIngestJob> updatedJobs)
{
BatchesForCurrentImages = batchesForCurrentImages;
UpdatedJobs = updatedJobs;
}

public List<Batch> BatchesForCurrentImages { get; set; }
public List<Batch> BatchesForImagesRequiringSync { get; set; }
public List<DlcsIngestJob> UpdatedJobs { get; set; }
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
using System;
using System.Collections.Generic;
using System.Linq;
using Utils;
using Wellcome.Dds.AssetDomain.Dlcs.Ingest;
using Wellcome.Dds.AssetDomain.Dlcs.Model;
using Wellcome.Dds.AssetDomain.Mets;

namespace Wellcome.Dds.AssetDomain.DigitalObjects
{
/// <summary>
/// Contains the result of a synchronisation request.
/// Some of the tasks may have been done there and then (synchronously), others might have been queued, in batches
/// represents the set of differences between the METS view of the world and the DLCS view
/// </summary>
public class SyncOperation
{
public Guid SyncOperationIdentifier { get; }

/// <summary>
/// The DLCS Ingest Job, if there is one, that is using this SyncOperation
/// </summary>
public int? JobIdentifier { get; }

public string? ManifestationIdentifier { get; set; }
public int LegacySequenceIndex { get; set; }

// Ingest Ops
public List<Batch> Batches { get; set; }
public List<DlcsBatch> BatchIngestOperationInfos { get; set; }

// Patch Ops
public List<DlcsBatch> BatchPatchOperationInfos { get; set; }

public bool Succeeded { get; set; }
public string? Message { get; set; }

/// <summary>
/// The key is the physicalFile.StorageIdentifier
/// The value is a Dlcs API Image.
/// The value will be null if the image is not on the DLCS
/// </summary>
public Dictionary<string, Image?>? ImagesCurrentlyOnDlcs { get; set; }

/// <summary>
/// Any ingestible image that could be registered with the DLCS
/// </summary>
public Dictionary<string, Image?>? ImagesThatShouldBeOnDlcs { get; set; }
public List<Image>? DlcsImagesToIngest { get; set; }
public List<Image>? DlcsImagesToPatch { get; set; }
public Dictionary<string, List<string>>? Mismatches { get; set; }
public List<Image>? DlcsImagesCurrentlyIngesting { get; set; }
public List<Image>? Orphans { get; set; }
/// <summary>
/// Not every file mentioned in METS needs to be synced with the DLCS.
/// This is an optimisation so we don't have to look stuff up all the time
/// </summary>
public List<string>? StorageIdentifiersToIgnore { get; set; }

public bool RequiresSync => DlcsImagesToIngest.HasItems() || DlcsImagesToPatch.HasItems();

/// <summary>
/// The sync operation has at least one invalid access condition and should not be synced
/// </summary>
public bool HasInvalidAccessCondition { get; set; }

/// <summary>
/// Files with no access condition in METS
/// </summary>
public List<IStoredFile>? MissingAccessConditions { get; set; }

public SyncOperation(DlcsCallContext dlcsCallContext)
{
SyncOperationIdentifier = Guid.NewGuid();
JobIdentifier = dlcsCallContext.JobId;
dlcsCallContext.SyncOperationId = SyncOperationIdentifier;
Batches = new List<Batch>();
BatchIngestOperationInfos = new List<DlcsBatch>();
BatchPatchOperationInfos = new List<DlcsBatch>();
}

public string[] GetSummary()
{
string syncReasons = "(no sync messages)";
if (Mismatches != null && Mismatches.Count > 0)
{
var first = Mismatches.First().Value;
syncReasons = string.Join(", ", first);
}
var summary = new List<string>
{
$"SyncOperationIdentifier: {SyncOperationIdentifier}",
$"JobId: {JobIdentifier}",
$"RequiresSync: {RequiresSync}",
$"DlcsImagesToIngest: {DlcsImagesToIngest!.Count}",
$"DlcsImagesToPatch: {DlcsImagesToPatch!.Count}",
$"DlcsImagesCurrentlyIngesting: {DlcsImagesCurrentlyIngesting!.Count}",
$"Ignored storage identifiers: {StorageIdentifiersToIgnore!.Count}",
$"Orphans: {Orphans!.Count}",
$"HasInvalidAccessCondition: {HasInvalidAccessCondition}",
$"Message: {Message}",
$"SyncReason: {syncReasons}"
};
return summary.ToArray();
}
}
}
31 changes: 7 additions & 24 deletions src/Wellcome.Dds/Wellcome.Dds.AssetDomain/Dlcs/AssetFamily.cs
Original file line number Diff line number Diff line change
@@ -1,25 +1,8 @@
namespace Wellcome.Dds.AssetDomain.Dlcs
{
public enum AssetFamily
{
Image = 'I',
TimeBased = 'T',
File = 'F'
}
namespace Wellcome.Dds.AssetDomain.Dlcs;

public static class AssetFamilyUtils
{
public static AssetFamily GetAssetFamily(this string mediaType)
{
if (mediaType.StartsWith("image/"))
{
return AssetFamily.Image;
}
if (mediaType.StartsWith("video/") || mediaType.StartsWith("audio/"))
{
return AssetFamily.TimeBased;
}
return AssetFamily.File;
}
}
}
public enum AssetFamily
{
Image = 'I',
TimeBased = 'T',
File = 'F'
}
55 changes: 34 additions & 21 deletions src/Wellcome.Dds/Wellcome.Dds.AssetDomain/Dlcs/IDlcs.cs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
using System.Collections.Generic;
using System.Threading.Tasks;
using Wellcome.Dds.AssetDomain.Dashboard;
using Wellcome.Dds.AssetDomain.DigitalObjects;
using Wellcome.Dds.AssetDomain.Dlcs.Model;
using Wellcome.Dds.AssetDomain.Dlcs.RestOperations;

@@ -18,25 +18,32 @@ public interface IDlcs
/// Image[]
/// </summary>
/// <param name="images"></param>
/// <param name="dlcsCallContext"></param>
/// <param name="priority">add the jobs to the priority queue rather than the default</param>
/// <returns></returns>
Task<Operation<HydraImageCollection, Batch>> RegisterImages(HydraImageCollection images, bool priority = false);
Task<Operation<HydraImageCollection, Batch>> RegisterImages(
HydraImageCollection images,
DlcsCallContext dlcsCallContext,
bool priority = false);

Task<Operation<HydraImageCollection, HydraImageCollection>> PatchImages(HydraImageCollection images);
Task<Operation<HydraImageCollection, HydraImageCollection>> PatchImages(
HydraImageCollection images,
DlcsCallContext dlcsCallContext);

/// <summary>
/// Query the queue for ingest status
/// GET /c/queue?q={imageQuery}
/// </summary>
/// <param name="query"></param>
/// <param name="defaultSpace"></param>
/// <param name="dlcsCallContext"></param>
/// <returns></returns>
Task<Operation<ImageQuery, HydraImageCollection>> GetImages(ImageQuery query, int defaultSpace);
Task<Operation<ImageQuery, HydraImageCollection>> GetImages(ImageQuery query, int defaultSpace, DlcsCallContext dlcsCallContext);

// TODO - this should be a Uri
Task<Operation<ImageQuery, HydraImageCollection>> GetImages(string nextUri);
Task<Operation<ImageQuery, HydraImageCollection>> GetImages(string nextUri, DlcsCallContext dlcsCallContext);

Task<Operation<string, Batch>> GetBatch(string batchId);
Task<Operation<string, Batch>> GetBatch(string batchId, DlcsCallContext dlcsCallContext);

// TODO - this should be return a Uri, or change name
string GetRoleUri(string accessCondition);
@@ -47,37 +54,37 @@ public interface IDlcs
// these methods give us our images back for checking

// any identifier - delegate to any of the FOUR following by same logic as metsrepository
Task<IEnumerable<Image>> GetImagesForIdentifier(string anyIdentifier);
Task<IEnumerable<Image>> GetImagesForIdentifier(string anyIdentifier, DlcsCallContext dlcsCallContext);

// string 3
Task<IEnumerable<Image>> GetImagesForIssue(string issueIdentifier);
Task<IEnumerable<Image>> GetImagesForIssue(string issueIdentifier, DlcsCallContext dlcsCallContext);

Task<IEnumerable<Image>> GetImagesForString3(string identfier);
Task<IEnumerable<Image>> GetImagesForString3(string identifier, DlcsCallContext dlcsCallContext);

// string 2
Task<IEnumerable<Image>> GetImagesForVolume(string volumeIdentifier);
Task<IEnumerable<Image>> GetImagesForVolume(string volumeIdentifier, DlcsCallContext dlcsCallContext);

// string 1
Task<IEnumerable<Image>> GetImagesForBNumber(string identfier);
Task<IEnumerable<Image>> GetImagesForBNumber(string identifier, DlcsCallContext dlcsCallContext);

Task<IEnumerable<ErrorByMetadata>> GetErrorsByMetadata();
Task<IEnumerable<ErrorByMetadata>> GetErrorsByMetadata(DlcsCallContext dlcsCallContext);

Task<Page<ErrorByMetadata>> GetErrorsByMetadata(int page);
Task<Page<ErrorByMetadata>> GetErrorsByMetadata(int page, DlcsCallContext dlcsCallContext);

/// <summary>
/// Used for images in the DLCS without string2 and string3 (volume and issue identifiers)
/// </summary>
/// <param name="identifier"></param>
/// <param name="sequenceIndex"></param>
/// <returns></returns>
Task<IEnumerable<Image>> GetImagesBySequenceIndex(string identifier, int sequenceIndex);
Task<IEnumerable<Image>> GetImagesBySequenceIndex(string identifier, int sequenceIndex, DlcsCallContext dlcsCallContext);

Task<IEnumerable<Image>> GetImagesByDlcsIdentifiers(List<string> identifiers);
Task<IEnumerable<Image>> GetImagesByDlcsIdentifiers(List<string> identifiers, DlcsCallContext dlcsCallContext);

// If you have a lot of images to register, in batches, call RegisterImages above.
// and this is for new Tizer ingests - where the DLCS does not have these images yet.
// This is immediate, for a small number of images.
IEnumerable<Image> RegisterNewImages(List<Image> images);
IEnumerable<Image> RegisterNewImages(List<Image> images, DlcsCallContext dlcsCallContext);

// You can only call this if you are sure that none of the images are in use anywhere.
// This means scoped to b nnumber in the event of a reorganisation of volumes.
@@ -86,25 +93,31 @@ public interface IDlcs
// Any in the first that are not in the second can be deleted safely.
// This probably can only run as a background job, unless it's a single b number. But for a single
// b number, this kind of misalignment is unlikely.
Task<int> DeleteImages(List<Image> images);
Task<int> DeleteImages(List<Image> images, DlcsCallContext dlcsCallContext);

/// <summary>
/// TODO: This MUST be changed to use string3 as soon as the manifest has that info to emit into the rendering
/// </summary>
// Task<IPdf> GetPdfDetails(string string1, int number1);
Task<IPdf> GetPdfDetails(string identifier);
Task<IPdf?> GetPdfDetails(string identifier);

// Task<bool> DeletePdf(string string1, int number1);
Task<bool> DeletePdf(string identifier);

int DefaultSpace { get; }
int DefaultCustomer { get; }

int BatchSize { get; }

bool PreventSynchronisation { get; }
Task<List<Batch>> GetTestedImageBatches(List<Batch> imageBatches);
Task<List<Batch>> GetTestedImageBatches(List<Batch> imageBatches, DlcsCallContext dlcsCallContext);
Task<Dictionary<string, long>> GetDlcsQueueLevel();

List<AVDerivative> GetAVDerivatives(Image dlcsImage);

string ResourceEntryPoint { get; }
string InternalResourceEntryPoint { get; }

bool SupportsDeliveryChannels { get; }
Task<Image?> GetImage(int space, string id, DlcsCallContext dlcsCallContext);
Task<Image?> ReingestImage(int space, string id, DlcsCallContext dlcsCallContext);
}
}
8 changes: 4 additions & 4 deletions src/Wellcome.Dds/Wellcome.Dds.AssetDomain/Dlcs/ImageQuery.cs
Original file line number Diff line number Diff line change
@@ -5,14 +5,14 @@ public class ImageQuery
{
public int? Space { get; set; }

public string String1 { get; set; }
public string String2 { get; set; }
public string String3 { get; set; }
public string? String1 { get; set; }
public string? String2 { get; set; }
public string? String3 { get; set; }

public long? Number1 { get; set; }
public long? Number2 { get; set; }
public long? Number3 { get; set; }

public string[] Tags { get; set; }
public string[]? Tags { get; set; }
}
}
Original file line number Diff line number Diff line change
@@ -11,11 +11,11 @@ public class DlcsBatch
public int Id { get; set; }
public int DlcsIngestJobId { get; set; }
public DateTime? RequestSent { get; set; }
public string RequestBody { get; set; }
public string? RequestBody { get; set; }
public DateTime? Finished { get; set; }
public string ResponseBody { get; set; }
public string? ResponseBody { get; set; }
public int ErrorCode { get; set; }
public string ErrorText { get; set; }
public string? ErrorText { get; set; }
public int BatchSize { get; set; }

[DatabaseGenerated(DatabaseGeneratedOption.Computed)]
Original file line number Diff line number Diff line change
@@ -12,23 +12,29 @@ namespace Wellcome.Dds.AssetDomain.Dlcs.Ingest
[Index(nameof(Created))]
public class DlcsIngestJob
{
public int Id { get; set; }
public DlcsIngestJob(string identifier)
{
Created = DateTime.UtcNow;
Identifier = identifier;
}

public int Id { get; set; }
public DateTime Created { get; set; }
public string Identifier { get; set; }
public string Label { get; set; }
public string? Label { get; set; }
public int SequenceIndex { get; set; }
public string VolumePart { get; set; }
public string IssuePart { get; set; }
public string? VolumePart { get; set; }
public string? IssuePart { get; set; }
public int ImageCount { get; set; }
public DateTime? StartProcessed { get; set; }
public DateTime? EndProcessed { get; set; }
public string AssetType { get; set; }
public string? AssetType { get; set; }

public bool Succeeded { get; set; }
public string Data { get; set; }
public string? Data { get; set; }
public int ReadyImageCount { get; set; }

public virtual ICollection<DlcsBatch> DlcsBatches { get; set; }
public virtual ICollection<DlcsBatch> DlcsBatches { get; set; } = null!;

/// <summary>
/// Determine the field of this job that is equivalent to a manifest identifier
Loading

0 comments on commit a195e4a

Please sign in to comment.