Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Large] Create logic for editing metadata #372

Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
f8db60c
Start work on MMS endpoint
aswallace Oct 4, 2024
3a56a6a
Add edit files logic and support in fileservices
SeanLeRoy Oct 4, 2024
a669973
Update to use FES stored annotationID
SeanLeRoy Oct 7, 2024
d405170
Update comment
SeanLeRoy Oct 7, 2024
8eafa0f
Remove MMS
SeanLeRoy Oct 7, 2024
f4a1eb9
Map FES base URL to MMS base url
SeanLeRoy Oct 7, 2024
d39f514
Change minor spelling
aswallace Oct 15, 2024
a1392ba
Fix editFileLogic tests
aswallace Oct 16, 2024
64d8bad
Start work on MMS endpoint
aswallace Oct 4, 2024
47cbad7
Add edit files logic and support in fileservices
SeanLeRoy Oct 4, 2024
d773efb
Update to use FES stored annotationID
SeanLeRoy Oct 7, 2024
d249446
Remove MMS
SeanLeRoy Oct 7, 2024
90dcd75
Map FES base URL to MMS base url
SeanLeRoy Oct 7, 2024
a528766
Change minor spelling
aswallace Oct 15, 2024
18ce082
Start work on MMS endpoint
aswallace Oct 4, 2024
97d6710
Add edit files logic and support in fileservices
SeanLeRoy Oct 4, 2024
4bb3c33
Update to use FES stored annotationID
SeanLeRoy Oct 7, 2024
6f66c59
Remove MMS
SeanLeRoy Oct 7, 2024
85ecfe2
Map FES base URL to MMS base url
SeanLeRoy Oct 7, 2024
55649ba
Change minor spelling
aswallace Oct 15, 2024
b0e6544
temporarily skip out-of-date tests
aswallace Dec 10, 2024
2d42179
add refresh for edits and account for annotations with spaces
aswallace Dec 17, 2024
4b8da5e
update file id references to match UID changes
aswallace Dec 18, 2024
80d9536
fix MMS endpoint base urls and request body
aswallace Dec 19, 2024
e885706
remove logic for getting metadata from MMS
aswallace Dec 19, 2024
e5114f8
trim value strings to prevent empty submits
aswallace Dec 20, 2024
6684902
Merge branch 'feature/metadata-editing/develop' of github.com:AllenIn…
aswallace Jan 7, 2025
3dfc6a1
fix references to MMS endpoint url
aswallace Jan 7, 2025
4e91442
fix edit logic test
aswallace Jan 7, 2025
fcfa8f9
try addressing edit test timouts by awaiting all logic branches
aswallace Jan 7, 2025
404f5c9
test modifications to async edit logic
aswallace Jan 8, 2025
a1517c5
add FES test endpoint to mock state
aswallace Jan 8, 2025
ec5277b
verify that mocked download service is not necessary
aswallace Jan 9, 2025
03fd374
clean up debugging code
aswallace Jan 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
import { IComboBoxOption } from "@fluentui/react";
import classNames from "classnames";
import * as React from "react";
import { useDispatch } from "react-redux";

import MetadataDetails, { ValueCountItem } from "./MetadataDetails";
import { PrimaryButton, SecondaryButton } from "../Buttons";
import ComboBox from "../ComboBox";
import { interaction } from "../../state";

import styles from "./EditMetadata.module.css";

Expand All @@ -20,6 +22,7 @@ interface ExistingAnnotationProps {
* and then entering values for the selected files
*/
export default function ExistingAnnotationPathway(props: ExistingAnnotationProps) {
const dispatch = useDispatch();
const [newValues, setNewValues] = React.useState<string>();
const [valueCount, setValueCount] = React.useState<ValueCountItem[]>();
const [selectedAnnotation, setSelectedAnnotation] = React.useState<string | undefined>();
Expand Down Expand Up @@ -65,7 +68,9 @@ export default function ExistingAnnotationPathway(props: ExistingAnnotationProps
};

function onSubmit() {
// TO DO: endpoint logic is in progress on a different branch
if (selectedAnnotation && newValues) {
dispatch(interaction.actions.editFiles({ [selectedAnnotation]: [newValues] }));
}
props.onDismiss();
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
width: 40vw;
height: 100%;
left: 30px;
/* FluentUI modal z-index is set to 1000000 */
BrianWhitneyAI marked this conversation as resolved.
Show resolved Hide resolved
z-index: 1000;
pointer-events: none;
}
Expand Down
6 changes: 6 additions & 0 deletions packages/core/entity/Annotation/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ import { AnnotationValue } from "../../services/AnnotationService";
* Expected JSON structure of an annotation returned from the query service.
*/
export interface AnnotationResponse {
// Undefined when pulled from a non-AICS FMS data source
annotationId?: number;
annotationDisplayName: string;
annotationName: string;
description: string;
Expand Down Expand Up @@ -70,6 +72,10 @@ export default class Annotation {
return this.annotation.units;
}

public get id(): number | undefined {
return this.annotation.annotationId;
}

/**
* Get the annotation this instance represents from a given FmsFile. An annotation on an FmsFile
* can either be at the "top-level" of the document or it can be within it's "annotations" list.
Expand Down
30 changes: 12 additions & 18 deletions packages/core/hooks/useFileAccessContextMenu.ts
Original file line number Diff line number Diff line change
Expand Up @@ -136,24 +136,18 @@ export default (filters?: FileFilter[], onDismiss?: () => void) => {
],
},
},
...(isQueryingAicsFms
? [
{
key: "edit",
text: "Edit metadata",
title: "Edit metadata of selected files",
disabled: !filters && fileSelection.count() === 0,
iconProps: {
iconName: "Edit",
},
onClick() {
dispatch(
interaction.actions.setVisibleModal(ModalType.EditMetadata)
);
},
},
]
: []),
{
key: "edit",
text: "Edit metadata",
title: "Edit metadata for selected files",
disabled: !filters && fileSelection.count() === 0,
iconProps: {
iconName: "Edit",
},
onClick() {
dispatch(interaction.actions.setVisibleModal(ModalType.EditMetadata));
},
},
{
key: "download",
text: "Download",
Expand Down
2 changes: 1 addition & 1 deletion packages/core/services/DatabaseService/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ export default abstract class DatabaseService {
_uri: string | File
): Promise<void>;

protected abstract execute(_sql: string): Promise<void>;
public abstract execute(_sql: string): Promise<void>;

private static columnTypeToAnnotationType(columnType: string): string {
switch (columnType) {
Expand Down
46 changes: 45 additions & 1 deletion packages/core/services/FileService/DatabaseFileService/index.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
import { isEmpty, isNil, uniqueId } from "lodash";

import FileService, { GetFilesRequest, SelectionAggregationResult, Selection } from "..";
import FileService, {
GetFilesRequest,
SelectionAggregationResult,
Selection,
AnnotationNameToValuesMap,
} from "..";
import DatabaseService from "../../DatabaseService";
import DatabaseServiceNoop from "../../DatabaseService/DatabaseServiceNoop";
import FileDownloadService, { DownloadResolution, DownloadResult } from "../../FileDownloadService";
Expand Down Expand Up @@ -184,4 +189,43 @@ export default class DatabaseFileService implements FileService {
uniqueId()
);
}

public editFile(fileId: string, annotations: AnnotationNameToValuesMap): Promise<void> {
const tableName = this.dataSourceNames.sort().join(", ");
const columnAssignments = Object.entries(annotations).map(
([name, values]) => `"${name}" = '${values.join(DatabaseService.LIST_DELIMITER)}'`
);
const sql = `\
UPDATE '${tableName}' \
SET ${columnAssignments.join(", ")} \
WHERE ${DatabaseService.HIDDEN_UID_ANNOTATION} = '${fileId}'; \
`;
return this.databaseService.execute(sql);
}

public async getEditableFileMetadata(
aswallace marked this conversation as resolved.
Show resolved Hide resolved
fileIds: string[]
): Promise<{ [fileId: string]: AnnotationNameToValuesMap }> {
const sql = new SQLBuilder()
.from(this.dataSourceNames)
.where(`${DatabaseService.HIDDEN_UID_ANNOTATION} IN (${fileIds.join(", ")})`)
.toSQL();

const rows = await this.databaseService.query(sql);
return rows
.map((row) => DatabaseFileService.convertDatabaseRowToFileDetail(row))
.reduce(
(acc, file) => ({
...acc,
[file.uid]: file.annotations.reduce(
(annoAcc, annotation) => ({
...annoAcc,
[annotation.name]: annotation.values,
}),
{} as AnnotationNameToValuesMap
),
}),
{} as { [fileId: string]: AnnotationNameToValuesMap }
);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,39 @@ describe("DatabaseFileService", () => {
});
});

describe("getEditableFileMetadata", () => {
it("converts response into a map of file uids to metadata", async () => {
// Arrange
const databaseFileService = new DatabaseFileService({
dataSourceNames: ["mock source name", "another mock source name"],
databaseService,
downloadService: new FileDownloadServiceNoop(),
});

// Act
const response = await databaseFileService.getEditableFileMetadata([
"abc123",
"def456",
]);

// Assert
expect(response).to.deep.equal({
abc123: {
"File Name": ["file"],
"File Path": ["path/to/file"],
"File Size": ["432226"],
num_files: ["6"],
},
def456: {
"File Name": ["file"],
"File Path": ["path/to/file"],
"File Size": ["432226"],
num_files: ["6"],
},
});
});
});

describe("getAggregateInformation", () => {
it("issues request for aggregated information about given files", async () => {
// Arrange
Expand Down
14 changes: 9 additions & 5 deletions packages/core/services/FileService/FileServiceNoop.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import FileService, { SelectionAggregationResult } from ".";
import FileService, { AnnotationNameToValuesMap, SelectionAggregationResult } from ".";
import { DownloadResolution, DownloadResult } from "../FileDownloadService";
import FileDetail from "../../entity/FileDetail";

Expand All @@ -11,15 +11,19 @@ export default class FileServiceNoop implements FileService {
return Promise.resolve({ count: 0, size: 0 });
}

public getFiles(): Promise<FileDetail[]> {
return Promise.resolve([]);
public getEditableFileMetadata(): Promise<{ [fileId: string]: AnnotationNameToValuesMap }> {
return Promise.resolve({});
}

public getFilesAsBuffer(): Promise<Uint8Array> {
return Promise.resolve(new Uint8Array());
public getFiles(): Promise<FileDetail[]> {
return Promise.resolve([]);
}

public download(): Promise<DownloadResult> {
return Promise.resolve({ downloadRequestId: "", resolution: DownloadResolution.CANCELLED });
}

public editFile(): Promise<void> {
return Promise.resolve();
}
}
78 changes: 77 additions & 1 deletion packages/core/services/FileService/HttpFileService/index.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,16 @@
import { compact, join, uniqueId } from "lodash";

import FileService, { GetFilesRequest, SelectionAggregationResult, Selection } from "..";
import FileService, {
GetFilesRequest,
SelectionAggregationResult,
Selection,
AnnotationNameToValuesMap,
} from "..";
import FileDownloadService, { DownloadResult } from "../../FileDownloadService";
import FileDownloadServiceNoop from "../../FileDownloadService/FileDownloadServiceNoop";
import HttpServiceBase, { ConnectionConfig } from "../../HttpServiceBase";
import { FileExplorerServiceBaseUrl } from "../../../constants";
import Annotation from "../../../entity/Annotation";
import FileSelection from "../../../entity/FileSelection";
import FileSet from "../../../entity/FileSet";
import FileDetail, { FmsFile } from "../../../entity/FileDetail";
Expand All @@ -16,13 +23,30 @@ interface Config extends ConnectionConfig {
downloadService: FileDownloadService;
}

// Used for the GET request to MMS for file metadata
interface EditableFileMetadata {
fileId: string;
annotations?: {
annotationId: number;
values: string[];
}[];
templateId?: number;
}

const FESBaseUrlToMMSBaseUrlMap = {
[FileExplorerServiceBaseUrl.LOCALHOST]: "https://localhost:9060",
[FileExplorerServiceBaseUrl.STAGING]: "https://stg-aics.corp.alleninstitute.org",
[FileExplorerServiceBaseUrl.PRODUCTION]: "https://stg-aics.corp.alleninstitute.org", // TO DO: unsure if using stg for prod was intentional
aswallace marked this conversation as resolved.
Show resolved Hide resolved
};

/**
* Service responsible for fetching file related metadata.
*/
export default class HttpFileService extends HttpServiceBase implements FileService {
private static readonly ENDPOINT_VERSION = "3.0";
public static readonly BASE_FILES_URL = `file-explorer-service/${HttpFileService.ENDPOINT_VERSION}/files`;
public static readonly BASE_FILE_COUNT_URL = `${HttpFileService.BASE_FILES_URL}/count`;
public static readonly BASE_EDIT_FILES_URL = `metadata-management-service/1.0/filemetadata`;
public static readonly SELECTION_AGGREGATE_URL = `${HttpFileService.BASE_FILES_URL}/selection/aggregate`;
private static readonly CSV_ENDPOINT_VERSION = "2.0";
public static readonly BASE_CSV_DOWNLOAD_URL = `file-explorer-service/${HttpFileService.CSV_ENDPOINT_VERSION}/files/selection/manifest`;
Expand Down Expand Up @@ -127,4 +151,56 @@ export default class HttpFileService extends HttpServiceBase implements FileServ
uniqueId()
);
}

public async editFile(
fileId: string,
annotationNameToValuesMap: AnnotationNameToValuesMap,
annotationNameToAnnotationMap?: Record<string, Annotation>
): Promise<void> {
const mmsBaseUrl = FESBaseUrlToMMSBaseUrlMap[this.baseUrl as FileExplorerServiceBaseUrl];
const url = `${mmsBaseUrl}/${HttpFileService.BASE_EDIT_FILES_URL}/${fileId}`;
const annotations = Object.entries(annotationNameToValuesMap).map(([name, values]) => {
const annotationId = annotationNameToAnnotationMap?.[name].id;
if (!annotationId) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it might be better to aggregate all missing annotations and throw a single error

throw new Error(
`Unable to edit file. Failed to find annotation id for annotation ${name}`
);
}
return { annotationId, values };
});
const requestBody = JSON.stringify({ customMetadata: { annotations } });
await this.put(url, requestBody);
SeanLeRoy marked this conversation as resolved.
Show resolved Hide resolved
}

public async getEditableFileMetadata(
aswallace marked this conversation as resolved.
Show resolved Hide resolved
fileIds: string[],
annotationIdToAnnotationMap?: Record<number, Annotation>
): Promise<{ [fileId: string]: AnnotationNameToValuesMap }> {
const mmsBaseUrl = FESBaseUrlToMMSBaseUrlMap[this.baseUrl as FileExplorerServiceBaseUrl];
const url = `${mmsBaseUrl}/${HttpFileService.BASE_EDIT_FILES_URL}/${fileIds.join(",")}`;
const response = await this.get<EditableFileMetadata>(url);

// Group files by fileId
return response.data.reduce(
(fileAcc, file) => ({
...fileAcc,
// Group annotations by name
[file.fileId]:
file.annotations?.reduce((annoAcc, annotation) => {
const name = annotationIdToAnnotationMap?.[annotation.annotationId]?.name;
if (!name) {
throw new Error(
"Failure mapping editable metadata response. " +
`Failed to find annotation name for annotation id ${annotation.annotationId}`
);
}
return {
...annoAcc,
[name]: annotation.values,
};
}, {} as AnnotationNameToValuesMap) || {},
}),
{} as { [fileId: string]: AnnotationNameToValuesMap }
);
}
}
Loading
Loading