-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #240 from wellcomecollection/develop
Merge develop -> main
Showing
368 changed files
with
19,588 additions
and
5,469 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
## Workflow queues and topics - questions/decisions | ||
|
||
There are two existing SNS topics owned by _*storage*_: | ||
|
||
- born-digital-bag-notifications-prod | ||
- born-digital-bag-notifications-staging | ||
|
||
And there are two new SNS topics for Goobi, owned by _*workflow*_ | ||
|
||
- digitised-bag-notifications-workflow-prod | ||
- digitised-bag-notifications-workflow-staging | ||
|
||
At the time these first two topics were created, queues were also created, subscribing to these topics. We don't use these queues: | ||
|
||
* arn:aws:sqs:eu-west-1:975596993436:born-digital-notifications-prod (in storage acct) | ||
* arn:aws:sqs:eu-west-1:653428163053:born-digital-notifications-prod (in digirati acct) | ||
|
||
There is also an equivalent queue for staging storage. | ||
|
||
## DDS Approach | ||
|
||
In our terraform, we reference these four topics as data, but we creates our own queues for each environment: | ||
|
||
- Two for each of our prod, stage, test (aka stage-prod) and also local dev environments. | ||
|
||
We then subscribe some of these queues to the storage (born-digital) or workflow (goobi) topics. | ||
|
||
This means the running DDS has no knowledge of the topics, and certainly never publishes to them. For the scenario of a manual initiation of a workflow from the dashboard, the dashboard sends a message on its appropriate, DDS-infra queue. | ||
|
||
The WorkflowProcessor listens to a list of queues, set in DdsOptions `WorkflowMessageListenQueues` (so we can add others for testing), and the dashboard is capable of sending messages to one of two specific queues for its environment, one for born digital and one for digitised, the settings `DashboardPushDigitisedQueue` and `DashboardPushBornDigitalQueue`. In practice, the workflowProcessor in that environment is listening to these same two queues, but it can listen to others. | ||
|
||
This means that DDS terraform manages all the notification queues it uses, the only point of contact with storage/workflow infrastructure is the queue subscription in terraform. | ||
|
||
We can have more than one WorkflowProcessor service picking up a job, but we don't want more than one environment synchronising with the same DLCS space. | ||
So DDS production and test each have their own queue(s) subscribed to both Goobi and born-digital topics. | ||
|
||
- DDS production writes to space 5, as now. | ||
- DDS staging writes to DLCS space 6, as now. | ||
- DDS test (stage-prod) writes to space 7 (this is NEW) | ||
|
||
In normal operation the `Test` environment WorkflowProcessor does not listen to queues. This is set via `WorkflowMessagePoll` in DdsOptions. It might be that Test WorkflowProcessor is deployed with an empty list for `WorkflowMessageListenQueues`. | ||
|
||
Stage should listen to Goobi stage and Storage stage, and sync with DLCS, all the time it's running (assuming much less traffic comes through stage, and the traffic that does come through still needs to be looked at via IIIF/DLCS). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
36 changes: 36 additions & 0 deletions
36
src/Wellcome.Dds/DlcsJobProcessor/appsettings.Staging-New.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
{ | ||
"Serilog": { | ||
"MinimumLevel": { | ||
"Default": "Debug", | ||
"Override": { | ||
"Microsoft": "Warning", | ||
"System": "Warning" | ||
} | ||
}, | ||
"Properties": { | ||
"ApplicationName": "Job-Processor", | ||
"Environment": "Staging-New" | ||
} | ||
}, | ||
"Dlcs": { | ||
"CustomerDefaultSpace": 6, | ||
"SkeletonNamedQueryTemplate": "https://neworchestrator.dlcs.io/iiif-resource/wellcome/preview/{0}/{1}", | ||
"ApiEntryPoint": "https://newapi.dlcs.io/", | ||
"ResourceEntryPoint": "https://iiif-stage-new.wellcomecollection.org/", | ||
"InternalResourceEntryPoint": "https://neworchestrator.dlcs.io/", | ||
"SupportsDeliveryChannels": true | ||
}, | ||
"Dds": { | ||
"LinkedDataDomain": "https://iiif-stage-new.wellcomecollection.org" | ||
}, | ||
"Storage": { | ||
"StorageApiTemplate": "https://api-stage.wellcomecollection.org/storage/v1/bags/{0}/{1}", | ||
"StorageApiTemplateIngest": "https://api-stage.wellcomecollection.org/storage/v1/ingests" | ||
}, | ||
"BinaryObjectCache": { | ||
"Wellcome.Dds.AssetDomainRepositories.Mets.WellcomeBagAwareArchiveStorageMap": { | ||
"Container": "wellcomecollection-stage-new-iiif-storagemaps", | ||
"Prefix": "stgmap-" | ||
} | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
using System; | ||
using System.Collections.Generic; | ||
using Newtonsoft.Json.Linq; | ||
|
||
namespace Utils.Aws.SQS; | ||
|
||
/// <summary> | ||
/// Generic representation of message pulled from queue. | ||
/// Adapted from protagonist for NewtonSoft.JSON | ||
/// </summary> | ||
public class QueueMessage | ||
{ | ||
public QueueMessage(JObject body, Dictionary<string, string> attributes, string messageId, string queueName) | ||
{ | ||
Body = body; | ||
Attributes = attributes; | ||
MessageId = messageId; | ||
QueueName = queueName; | ||
} | ||
|
||
/// <summary> | ||
/// The full message body property | ||
/// </summary> | ||
public JObject Body { get; set; } | ||
|
||
/// <summary> | ||
/// Any attributes associated with message | ||
/// </summary> | ||
public Dictionary<string, string> Attributes { get; set; } | ||
|
||
/// <summary> | ||
/// Unique identifier for message | ||
/// </summary> | ||
public string MessageId { get; set; } | ||
|
||
/// <summary> | ||
/// The name of the queue that this message was from | ||
/// </summary> | ||
public string QueueName { get; set; } | ||
} | ||
|
||
public static class QueueMessageX | ||
{ | ||
/// <summary> | ||
/// Get a <see cref="JsonObject"/> representing the contents of the message as raised by source system. This helps | ||
/// when the message can be from SNS->SQS, SNS->SQS with RawDelivery or SQS directly. | ||
/// | ||
/// If originating from SNS and Raw Message Delivery is disabled (default) then the <see cref="QueueMessage"/> | ||
/// object will have additional fields about topic etc, and the message will be embedded in a "Message" property. | ||
/// e.g. { "Type": "Notification", "MessageId": "1234", "Message": { \"key\": \"value\" } } | ||
/// | ||
/// If originating from SQS, or from SNS with Raw Message Delivery enabled, the <see cref="QueueMessage"/> Body | ||
/// property will contain the full message only. | ||
/// e.g. { "key": "value" } | ||
/// </summary> | ||
/// <remarks>See https://docs.aws.amazon.com/sns/latest/dg/sns-large-payload-raw-message-delivery.html </remarks> | ||
public static JObject? GetMessageContents(this QueueMessage queueMessage) | ||
{ | ||
const string messageKey = "Message"; | ||
if (queueMessage.Body.ContainsKey("TopicArn") && queueMessage.Body.ContainsKey(messageKey)) | ||
{ | ||
// From SNS without Raw Message Delivery | ||
try | ||
{ | ||
var value = queueMessage.Body[messageKey]!.Value<string>(); | ||
return value.HasText() ? JObject.Parse(value) : null; | ||
} | ||
catch (Exception) | ||
{ | ||
return null; | ||
} | ||
} | ||
|
||
// From SQS or SNS with Raw Message Delivery | ||
return queueMessage.Body; | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
using System.Collections.Generic; | ||
using FluentAssertions; | ||
using Wellcome.Dds.Repositories.Presentation.LicencesAndRights; | ||
using Xunit; | ||
|
||
namespace Utils.Tests; | ||
|
||
public class StringProcessingTests | ||
{ | ||
private static readonly Dictionary<string, string> LicenseMap = new() | ||
{ | ||
["PDM"] = "https://creativecommons.org/publicdomain/mark/1.0/", | ||
["CC0"] = "https://creativecommons.org/publicdomain/zero/1.0/", | ||
["CC-BY"] = "https://creativecommons.org/licenses/by/4.0/", | ||
["CC-BY-NC"] = "https://creativecommons.org/licenses/by-nc/4.0/", | ||
["CC-BY-NC-ND"] = "https://creativecommons.org/licenses/by-nc-nd/4.0/", | ||
["CC-BY-ND"] = "https://creativecommons.org/licenses/by-nd/4.0/", | ||
["CC-BY-SA"] = "https://creativecommons.org/licenses/by-sa/4.0/", | ||
["CC-BY-NC-SA"] = "https://creativecommons.org/licenses/by-nc-sa/4.0/", | ||
["OGL"] = "http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/", | ||
["OPL"] = "http://www.parliament.uk/site-information/copyright/open-parliament-licence/", | ||
["ARR"] = "https://en.wikipedia.org/wiki/All_rights_reserved", | ||
["All Rights Reserved"] = "https://en.wikipedia.org/wiki/All_rights_reserved", | ||
}; | ||
|
||
[Theory] | ||
[InlineData("This is CC-BY", @"This is <a href=""https://creativecommons.org/licenses/by/4.0/"">CC-BY</a>")] | ||
[InlineData("This is CC-BY hello", @"This is <a href=""https://creativecommons.org/licenses/by/4.0/"">CC-BY</a> hello")] | ||
[InlineData("This is CC-BY-NC hello", @"This is <a href=""https://creativecommons.org/licenses/by-nc/4.0/"">CC-BY-NC</a> hello")] | ||
[InlineData("This is CC-BY-NC hello and CC-BY", @"This is <a href=""https://creativecommons.org/licenses/by-nc/4.0/"">CC-BY-NC</a> hello and <a href=""https://creativecommons.org/licenses/by/4.0/"">CC-BY</a>")] | ||
[InlineData("This is OGL and CC-BY-NC-SA and CC-BY-NC and OGL again", | ||
@"This is <a href=""http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/"">OGL</a> and <a href=""https://creativecommons.org/licenses/by-nc-sa/4.0/"">CC-BY-NC-SA</a> and <a href=""https://creativecommons.org/licenses/by-nc/4.0/"">CC-BY-NC</a> and <a href=""http://www.nationalarchives.gov.uk/doc/open-government-licence/version/2/"">OGL</a> again")] | ||
public void LicenseMap_Codes_Are_Replaced(string raw, string expected) | ||
{ | ||
var processed = LicenceHelpers.GetUsageWithHtmlLinks(raw); | ||
|
||
processed.Should().Be(expected); | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,68 @@ | ||
using System.Diagnostics; | ||
|
||
namespace Utils.Logging; | ||
|
||
public class BatchMetrics | ||
{ | ||
public int BatchCounter { get; set; } | ||
public long LastBatchTime { get; set; } | ||
public long TotalTime { get; set; } | ||
public long MinBatchTime { get; set; } | ||
public int MinBatchCount { get; set; } | ||
public long MaxBatchTime { get; set; } | ||
public int MaxBatchCount { get; set; } | ||
|
||
public int BatchSize { get; set; } | ||
|
||
public long AverageBatchTime => TotalTime / BatchCounter; | ||
|
||
public string Summary => $"Total batches: {BatchCounter}; " + | ||
$"Avg batch time {AverageBatchTime}; " + | ||
$"Min batch time {MinBatchTime} ({MinBatchCount} items); " + | ||
$"Max Batch time {MaxBatchTime} ({MaxBatchCount} items)"; | ||
|
||
private readonly Stopwatch stopwatch; | ||
|
||
public BatchMetrics() | ||
{ | ||
stopwatch = new Stopwatch(); | ||
} | ||
|
||
public void BeginBatch(int batchSize = -1) | ||
{ | ||
stopwatch.Restart(); | ||
BatchCounter++; | ||
BatchSize = batchSize; | ||
} | ||
|
||
public void EndBatch(int batchSize = -1) | ||
{ | ||
if (batchSize > 0) | ||
{ | ||
// you might not know BatchSize until you end the batch | ||
BatchSize = batchSize; | ||
} | ||
LastBatchTime = stopwatch.ElapsedMilliseconds; | ||
if (BatchCounter == 1) | ||
{ | ||
MinBatchTime = LastBatchTime; | ||
MinBatchCount = BatchSize; | ||
MaxBatchTime = LastBatchTime; | ||
MaxBatchCount = BatchSize; | ||
} | ||
else | ||
{ | ||
if (LastBatchTime < MinBatchTime) | ||
{ | ||
MinBatchTime = LastBatchTime; | ||
MinBatchCount = BatchSize; | ||
} | ||
else if (LastBatchTime >= MaxBatchTime) | ||
{ | ||
MaxBatchTime = LastBatchTime; | ||
MaxBatchCount = BatchSize; | ||
} | ||
} | ||
TotalTime += LastBatchTime; | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
62 changes: 62 additions & 0 deletions
62
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/AssetExtensions.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
using System.Linq; | ||
using IIIF; | ||
using Wellcome.Dds.AssetDomain.DigitalObjects; | ||
using Wellcome.Dds.AssetDomain.Mets; | ||
|
||
namespace Wellcome.Dds.AssetDomain; | ||
|
||
public static class AssetExtensions | ||
{ | ||
public static bool IsVideoMimeType(this string? mimeType) | ||
{ | ||
return mimeType != null && mimeType.StartsWith("video/"); | ||
} | ||
|
||
public static bool IsAudioMimeType(this string? mimeType) | ||
{ | ||
return mimeType != null && mimeType.StartsWith("audio/"); | ||
} | ||
|
||
public static bool IsTimeBasedMimeType(this string? mimeType) | ||
{ | ||
return mimeType.IsVideoMimeType() || mimeType.IsAudioMimeType(); | ||
} | ||
|
||
public static bool IsImageMimeType(this string? mimeType) | ||
{ | ||
return mimeType != null && mimeType.StartsWith("image/"); | ||
} | ||
|
||
public static bool IsTextMimeType(this string? mimeType) | ||
{ | ||
return mimeType != null && mimeType.StartsWith("text/"); | ||
} | ||
|
||
public static IStoredFile? GetDefaultFile(this IPhysicalFile asset) | ||
{ | ||
return asset.Files!.SingleOrDefault(f => f.StorageIdentifier == asset.StorageIdentifier); | ||
} | ||
|
||
public static IProcessingBehaviour GetDefaultProcessingBehaviour(this IPhysicalFile asset) | ||
{ | ||
var defaultStoredFile = asset.GetDefaultFile(); | ||
return defaultStoredFile!.ProcessingBehaviour; | ||
} | ||
|
||
public static Size? GetWhSize(this IPhysicalFile file) | ||
{ | ||
var dimensions = file.AssetMetadata?.GetMediaDimensions(); | ||
if (dimensions == null) return null; | ||
|
||
var w = dimensions.Width.GetValueOrDefault(); | ||
var h = dimensions.Height.GetValueOrDefault(); | ||
if (w > 0 && h > 0) | ||
{ | ||
return new Size(w, h); | ||
} | ||
|
||
return null; | ||
} | ||
|
||
|
||
} |
36 changes: 0 additions & 36 deletions
36
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/Dashboard/IDashboardRepository.cs
This file was deleted.
Oops, something went wrong.
12 changes: 0 additions & 12 deletions
12
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/Dashboard/IDigitisedCollection.cs
This file was deleted.
Oops, something went wrong.
9 changes: 0 additions & 9 deletions
9
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/Dashboard/IDigitisedResource.cs
This file was deleted.
Oops, something went wrong.
69 changes: 0 additions & 69 deletions
69
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/Dashboard/SyncOperation.cs
This file was deleted.
Oops, something went wrong.
2,221 changes: 2,221 additions & 0 deletions
2,221
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/Data/pronom_map.json
Large diffs are not rendered by default.
Oops, something went wrong.
32 changes: 32 additions & 0 deletions
32
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/DeliveredFile.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
namespace Wellcome.Dds.AssetDomain; | ||
|
||
/// <summary> | ||
/// Represents files and derivatives of files served by DLCS | ||
/// </summary> | ||
public class DeliveredFile | ||
{ | ||
public string? PublicUrl { get; set; } | ||
public string? DlcsUrl { get; set; } | ||
public string? DeliveryChannel { get; set; } | ||
|
||
public string? MediaType { get; set; } | ||
public int? Width { get; set; } | ||
public int? Height { get; set; } | ||
public double? Duration { get; set; } | ||
|
||
public string GetSummary() | ||
{ | ||
var s = MediaType; | ||
if (Duration.HasValue) | ||
{ | ||
s += $"\nDuration: {Duration}"; | ||
} | ||
|
||
if (Height.HasValue) | ||
{ | ||
s += $"\nWidth: {Width}, Height: {Height}"; | ||
} | ||
|
||
return s ?? "(no summary information)"; | ||
} | ||
} |
12 changes: 12 additions & 0 deletions
12
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/DigitalObjects/IDigitalCollection.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
using System.Collections.Generic; | ||
using Wellcome.Dds.AssetDomain.Mets; | ||
|
||
namespace Wellcome.Dds.AssetDomain.DigitalObjects | ||
{ | ||
public interface IDigitalCollection : IDigitalObject | ||
{ | ||
ICollection? MetsCollection { get; set; } | ||
IEnumerable<IDigitalManifestation>? Manifestations { get; set; } | ||
IEnumerable<IDigitalCollection>? Collections { get; set; } | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
11 changes: 11 additions & 0 deletions
11
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/DigitalObjects/IDigitalObject.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
using Wellcome.Dds.Common; | ||
|
||
namespace Wellcome.Dds.AssetDomain.DigitalObjects | ||
{ | ||
public interface IDigitalObject | ||
{ | ||
DdsIdentifier? Identifier { get; set; } | ||
bool Partial { get; set; } | ||
bool? InSyncWithDlcs { get; } | ||
} | ||
} |
45 changes: 45 additions & 0 deletions
45
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/DigitalObjects/IDigitalObjectRepository.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
using System.Collections.Generic; | ||
using System.Threading.Tasks; | ||
using Wellcome.Dds.AssetDomain.Dlcs; | ||
using Wellcome.Dds.AssetDomain.Dlcs.Ingest; | ||
using Wellcome.Dds.AssetDomain.Dlcs.Model; | ||
using Wellcome.Dds.AssetDomain.Mets; | ||
using Wellcome.Dds.Common; | ||
|
||
namespace Wellcome.Dds.AssetDomain.DigitalObjects | ||
{ | ||
public interface IDigitalObjectRepository | ||
{ | ||
Task<IDigitalObject> GetDigitalObject(DdsIdentifier identifier, DlcsCallContext dlcsCallContext, bool includePdfDetails = false); | ||
Task<SyncOperation> GetDlcsSyncOperation( | ||
IDigitalManifestation digitisedManifestation, | ||
bool reIngestErrorImages, | ||
DlcsCallContext dlcsCallContext); | ||
|
||
Task ExecuteDlcsSyncOperation(SyncOperation syncOperation, bool usePriorityQueue, DlcsCallContext dlcsCallContext); | ||
int DefaultSpace { get; } | ||
Task<IEnumerable<DlcsIngestJob>> GetMostRecentIngestJobs(string identifier, int number); | ||
//IEnumerable<DlcsIngestJob> GetUpdatedIngestJobs(string identifier, SyncOperation syncOperation); | ||
Task<Batch?> GetBatch(string batchId, DlcsCallContext dlcsCallContext); | ||
|
||
Task<JobActivity> GetRationalisedJobActivity(SyncOperation syncOperation, DlcsCallContext dlcsCallContext); | ||
|
||
Task<IEnumerable<Batch>> GetBatchesForImages(IEnumerable<Image> images, DlcsCallContext dlcsCallContext); | ||
Task<IEnumerable<ErrorByMetadata>> GetErrorsByMetadata(DlcsCallContext dlcsCallContext); | ||
Task<Page<ErrorByMetadata>> GetErrorsByMetadata(int page, DlcsCallContext dlcsCallContext); | ||
|
||
Task<int> FindSequenceIndex(string identifier); | ||
Task<bool> DeletePdf(string identifier); | ||
Task<int> RemoveOldJobs(string id); | ||
Task<int> DeleteOrphans(string id, DlcsCallContext dlcsCallContext); | ||
|
||
IngestAction LogAction(string manifestationId, int? jobId, string userName, string action, string? description = null); | ||
IEnumerable<IngestAction> GetRecentActions(int count, string? user = null); | ||
Task<Dictionary<string, long>> GetDlcsQueueLevel(); | ||
|
||
|
||
DeliveredFile[] GetDeliveredFiles(IPhysicalFile physicalFile); | ||
DeliveredFile[] GetDeliveredFiles(IStoredFile? storedFile); | ||
|
||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
13 changes: 13 additions & 0 deletions
13
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/DigitalObjects/IProcessingBehaviour.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
using System.Collections.Generic; | ||
using IIIF; | ||
using Wellcome.Dds.AssetDomain.Dlcs; | ||
|
||
namespace Wellcome.Dds.AssetDomain.DigitalObjects; | ||
|
||
public interface IProcessingBehaviour | ||
{ | ||
HashSet<string> DeliveryChannels { get; } | ||
string? ImageOptimisationPolicy { get; } | ||
Size? GetVideoSize(string deliveryChannel); | ||
AssetFamily AssetFamily { get; } | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
107 changes: 107 additions & 0 deletions
107
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/DigitalObjects/SyncOperation.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,107 @@ | ||
using System; | ||
using System.Collections.Generic; | ||
using System.Linq; | ||
using Utils; | ||
using Wellcome.Dds.AssetDomain.Dlcs.Ingest; | ||
using Wellcome.Dds.AssetDomain.Dlcs.Model; | ||
using Wellcome.Dds.AssetDomain.Mets; | ||
|
||
namespace Wellcome.Dds.AssetDomain.DigitalObjects | ||
{ | ||
/// <summary> | ||
/// Contains the result of a synchronisation request. | ||
/// Some of the tasks may have been done there and then (synchronously), others might have been queued, in batches | ||
/// represents the set of differences between the METS view of the world and the DLCS view | ||
/// </summary> | ||
public class SyncOperation | ||
{ | ||
public Guid SyncOperationIdentifier { get; } | ||
|
||
/// <summary> | ||
/// The DLCS Ingest Job, if there is one, that is using this SyncOperation | ||
/// </summary> | ||
public int? JobIdentifier { get; } | ||
|
||
public string? ManifestationIdentifier { get; set; } | ||
public int LegacySequenceIndex { get; set; } | ||
|
||
// Ingest Ops | ||
public List<Batch> Batches { get; set; } | ||
public List<DlcsBatch> BatchIngestOperationInfos { get; set; } | ||
|
||
// Patch Ops | ||
public List<DlcsBatch> BatchPatchOperationInfos { get; set; } | ||
|
||
public bool Succeeded { get; set; } | ||
public string? Message { get; set; } | ||
|
||
/// <summary> | ||
/// The key is the physicalFile.StorageIdentifier | ||
/// The value is a Dlcs API Image. | ||
/// The value will be null if the image is not on the DLCS | ||
/// </summary> | ||
public Dictionary<string, Image?>? ImagesCurrentlyOnDlcs { get; set; } | ||
|
||
/// <summary> | ||
/// Any ingestible image that could be registered with the DLCS | ||
/// </summary> | ||
public Dictionary<string, Image?>? ImagesThatShouldBeOnDlcs { get; set; } | ||
public List<Image>? DlcsImagesToIngest { get; set; } | ||
public List<Image>? DlcsImagesToPatch { get; set; } | ||
public Dictionary<string, List<string>>? Mismatches { get; set; } | ||
public List<Image>? DlcsImagesCurrentlyIngesting { get; set; } | ||
public List<Image>? Orphans { get; set; } | ||
/// <summary> | ||
/// Not every file mentioned in METS needs to be synced with the DLCS. | ||
/// This is an optimisation so we don't have to look stuff up all the time | ||
/// </summary> | ||
public List<string>? StorageIdentifiersToIgnore { get; set; } | ||
|
||
public bool RequiresSync => DlcsImagesToIngest.HasItems() || DlcsImagesToPatch.HasItems(); | ||
|
||
/// <summary> | ||
/// The sync operation has at least one invalid access condition and should not be synced | ||
/// </summary> | ||
public bool HasInvalidAccessCondition { get; set; } | ||
|
||
/// <summary> | ||
/// Files with no access condition in METS | ||
/// </summary> | ||
public List<IStoredFile>? MissingAccessConditions { get; set; } | ||
|
||
public SyncOperation(DlcsCallContext dlcsCallContext) | ||
{ | ||
SyncOperationIdentifier = Guid.NewGuid(); | ||
JobIdentifier = dlcsCallContext.JobId; | ||
dlcsCallContext.SyncOperationId = SyncOperationIdentifier; | ||
Batches = new List<Batch>(); | ||
BatchIngestOperationInfos = new List<DlcsBatch>(); | ||
BatchPatchOperationInfos = new List<DlcsBatch>(); | ||
} | ||
|
||
public string[] GetSummary() | ||
{ | ||
string syncReasons = "(no sync messages)"; | ||
if (Mismatches != null && Mismatches.Count > 0) | ||
{ | ||
var first = Mismatches.First().Value; | ||
syncReasons = string.Join(", ", first); | ||
} | ||
var summary = new List<string> | ||
{ | ||
$"SyncOperationIdentifier: {SyncOperationIdentifier}", | ||
$"JobId: {JobIdentifier}", | ||
$"RequiresSync: {RequiresSync}", | ||
$"DlcsImagesToIngest: {DlcsImagesToIngest!.Count}", | ||
$"DlcsImagesToPatch: {DlcsImagesToPatch!.Count}", | ||
$"DlcsImagesCurrentlyIngesting: {DlcsImagesCurrentlyIngesting!.Count}", | ||
$"Ignored storage identifiers: {StorageIdentifiersToIgnore!.Count}", | ||
$"Orphans: {Orphans!.Count}", | ||
$"HasInvalidAccessCondition: {HasInvalidAccessCondition}", | ||
$"Message: {Message}", | ||
$"SyncReason: {syncReasons}" | ||
}; | ||
return summary.ToArray(); | ||
} | ||
} | ||
} |
31 changes: 7 additions & 24 deletions
31
src/Wellcome.Dds/Wellcome.Dds.AssetDomain/Dlcs/AssetFamily.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,25 +1,8 @@ | ||
namespace Wellcome.Dds.AssetDomain.Dlcs | ||
{ | ||
public enum AssetFamily | ||
{ | ||
Image = 'I', | ||
TimeBased = 'T', | ||
File = 'F' | ||
} | ||
namespace Wellcome.Dds.AssetDomain.Dlcs; | ||
|
||
public static class AssetFamilyUtils | ||
{ | ||
public static AssetFamily GetAssetFamily(this string mediaType) | ||
{ | ||
if (mediaType.StartsWith("image/")) | ||
{ | ||
return AssetFamily.Image; | ||
} | ||
if (mediaType.StartsWith("video/") || mediaType.StartsWith("audio/")) | ||
{ | ||
return AssetFamily.TimeBased; | ||
} | ||
return AssetFamily.File; | ||
} | ||
} | ||
} | ||
public enum AssetFamily | ||
{ | ||
Image = 'I', | ||
TimeBased = 'T', | ||
File = 'F' | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.