-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Folder And Dataset Metadata #7886
Merged
Merged
Changes from 51 commits
Commits
Show all changes
107 commits
Select commit
Hold shift + click to select a range
dbe80eb
Add dataset metadata input to details view in dashboard
MichaelBuessemeyer 4ab38dc
update docs
hotzenklotz 242cf1d
Improve mobile support of views (#7876)
MichaelBuessemeyer 3c139fe
Mapping in default view config (#7858)
MichaelBuessemeyer 5bb3997
Fix animation modal color layer validation (#7882)
hotzenklotz 1797a4e
Validate animation job bounding box (#7883)
hotzenklotz 55c3d5f
Merge branch 'master' of github.com:scalableminds/webknossos into fo…
MichaelBuessemeyer 5cf5bfd
force serialization of dataset details prop in compact version of dat…
MichaelBuessemeyer 70b185e
WIP adding details to folder
MichaelBuessemeyer 3c4ae12
First version metadata support for folders
MichaelBuessemeyer 434b875
fix folder listing query
MichaelBuessemeyer 47daad4
finish frontend support for folder metadata
MichaelBuessemeyer 9a8a5cd
default to undefined columnKey and not null to potentially prevent fr…
MichaelBuessemeyer 0bd1c29
add evolution
MichaelBuessemeyer 51cd0e4
update test db init data & folder snapshot test & bump schema version
MichaelBuessemeyer 488f5bd
Merge branch 'master' of github.com:scalableminds/webknossos into fol…
MichaelBuessemeyer b3294c4
remove unused import
MichaelBuessemeyer 8051729
also blur on property input
MichaelBuessemeyer 3fb9613
only support / show current error
MichaelBuessemeyer 1f0db40
improve css
MichaelBuessemeyer eaaaaea
WIP: use new json schema to save dataset & folder details and re-styl…
MichaelBuessemeyer 4ef393a
Merge branch 'master' of github.com:scalableminds/webknossos into fol…
MichaelBuessemeyer 846b500
fix backend to work with new type of details dataset / folder field &…
MichaelBuessemeyer e7ed7f8
WIP: improve dataset details table styling and add support for differ…
MichaelBuessemeyer 5e2912b
details table for folders and datasets version 2
MichaelBuessemeyer 40e60c1
WIP: adapt migration to new details format
MichaelBuessemeyer c091c0e
uncomment ci tests
MichaelBuessemeyer 98892ca
mini migration fix
MichaelBuessemeyer d4ebd9f
- rename details to metadata
MichaelBuessemeyer b305e7c
fix schema
MichaelBuessemeyer a853611
remove unused backend imports
MichaelBuessemeyer 113bb6e
only update metadata set of metadata table when folder / dataset changes
MichaelBuessemeyer e1bb660
ensure flushing updates on unmount of metadata table & increase debou…
MichaelBuessemeyer eee0471
remove accidental change
MichaelBuessemeyer 48665a7
WIP apply styling feedback & refactor handling metadataentry type
MichaelBuessemeyer 9a8c6bb
remove unused css
MichaelBuessemeyer 6cdfa74
keep old dataset while updating & refetching in the dataset details view
MichaelBuessemeyer 128f84c
remove unused code as search support for metadata entries is currentl…
MichaelBuessemeyer ee5865d
do not include metadata in dataset compact version
MichaelBuessemeyer 050129b
Fix Dataset refetching
MichaelBuessemeyer 539fc5d
have fix width for metadata table cell contents to ensure consistent…
MichaelBuessemeyer 51b1286
enable selecting current folder of folder tree view as active details…
MichaelBuessemeyer 5d2bd7d
do not have initial empty metadata row
MichaelBuessemeyer f235cd4
Merge branches 'folder-dataset-properties' and 'master' of github.com…
MichaelBuessemeyer 5ea7a34
WIP: Apply styling update
MichaelBuessemeyer df40a6c
Finish next version
MichaelBuessemeyer 0d1fe66
hopefully fix flickering bug in when two different metadata updates h…
MichaelBuessemeyer 6728c71
Only have default metadata on species, brainRegion & acquisition when…
MichaelBuessemeyer 0382aba
Merge branch 'master' of github.com:scalableminds/webknossos into fol…
MichaelBuessemeyer 06cbd2f
also fill full metadata table width in case of an empty table
MichaelBuessemeyer 050c3ac
allow changing prop name although it is a duplicate; do not save meta…
MichaelBuessemeyer b465021
Merge branch 'master' of github.com:scalableminds/webknossos into fol…
MichaelBuessemeyer 6678725
clean up code for review
MichaelBuessemeyer 47f96fc
add changelog & migration entry; rename evolution
MichaelBuessemeyer 1459f31
also rename revision; add comments to revision; remove dev logging
MichaelBuessemeyer 0e00f7b
Merge branch 'master' of github.com:scalableminds/webknossos into fol…
MichaelBuessemeyer aeba1f9
Apply PR Feedback
MichaelBuessemeyer e9fcb24
move metadata table to own file
MichaelBuessemeyer b14a4d9
only update when metadata changed & refactor code
MichaelBuessemeyer ebf16b7
fix updating the wrong dataset or folder with the newest metadata ver…
MichaelBuessemeyer d998e7f
allow multiple error rows & only update local metadata set if new fol…
MichaelBuessemeyer 4a75782
update preview image
MichaelBuessemeyer a748164
fix color layer / segmentation layer switchero bug
MichaelBuessemeyer 23cab35
remove periodic autosave an replace with explicit save via button or …
MichaelBuessemeyer 2dafad4
Merge branch 'master' of github.com:scalableminds/webknossos into fol…
MichaelBuessemeyer f90d669
re add auto saving mechanism and do not have nested components to avo…
MichaelBuessemeyer 9fd95dd
update snapshots
MichaelBuessemeyer 1793ebf
Merge branch 'master' of github.com:scalableminds/webknossos into fol…
MichaelBuessemeyer 93d9b6f
refactor code & remove `useWillUnmount` which sent outdated metadata …
MichaelBuessemeyer 3c7b291
Merge branch 'master' of github.com:scalableminds/webknossos into fol…
MichaelBuessemeyer 19adc1b
remove unnecessary dependency from useEffect accidentally added in a …
MichaelBuessemeyer c0827bd
include metadata in full dataset update route
MichaelBuessemeyer d624a27
fix full update dataset route for metadata support
MichaelBuessemeyer dd3c3da
remove option to update tags (as they will no longer be rendered in t…
MichaelBuessemeyer df5af16
Merge branch 'master' into folder-dataset-properties
MichaelBuessemeyer 0a607e8
remove index from initial metdata added to datasets with publication
MichaelBuessemeyer c0c8233
apply pr feedback (testing pending)
MichaelBuessemeyer 6932f71
lint frontend
MichaelBuessemeyer 2eba0c9
Merge branch 'master' into folder-dataset-properties
MichaelBuessemeyer 0e5b29d
do not send updates while a row of the metadata table is focused
MichaelBuessemeyer 62b67e5
avoid lost isSaving updates due to changes to `focusedRow` state whil…
MichaelBuessemeyer bfc14c3
fix sending double updates on unmount
MichaelBuessemeyer d8be922
Merge branch 'master' of github.com:scalableminds/webknossos into fol…
MichaelBuessemeyer 57dad17
apply pr feedback
MichaelBuessemeyer aa08135
Merge branch 'master' of github.com:scalableminds/webknossos into fol…
MichaelBuessemeyer 94a2dea
migrate existing tags into metadata
MichaelBuessemeyer cacb0dd
remove unwanted additional wrapping array around migrated tags in met…
MichaelBuessemeyer 2d4ebfa
add undo migrating tags to metadata to revision
MichaelBuessemeyer ff72b1f
fix revision schema
MichaelBuessemeyer aa3fdeb
give delete metadata entry button some more space; align key input to…
MichaelBuessemeyer a3a92d2
Merge branch 'master' into folder-dataset-properties
fm3 e9fccae
Merge branch 'master' of github.com:scalableminds/webknossos into fol…
MichaelBuessemeyer 984b889
update schema migration version
MichaelBuessemeyer a5184ee
apply feedback
MichaelBuessemeyer 3ea1f4d
readd tag support in frontend (and backend)
MichaelBuessemeyer 30bb3e9
fix backend
MichaelBuessemeyer 6080226
make metadata non nullable
MichaelBuessemeyer bc6739d
update snapshots tests
MichaelBuessemeyer 32f84d6
Merge branch 'master' of github.com:scalableminds/webknossos into fol…
MichaelBuessemeyer 6508c59
fix metadatatype type definition
MichaelBuessemeyer de0768a
Merge branch 'master' into folder-dataset-properties
MichaelBuessemeyer 6fffde2
Merge branch 'master' into folder-dataset-properties
philippotto 46224ba
fix reversion against default null values of re-added details column
MichaelBuessemeyer c505be0
Merge branch 'master' into folder-dataset-properties
MichaelBuessemeyer 5dd4872
Merge branch 'master' into folder-dataset-properties
philippotto 57311df
Merge branch 'master' into folder-dataset-properties
fm3 76c2a97
Merge branch 'master' into folder-dataset-properties
MichaelBuessemeyer File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -1,14 +1,14 @@ | ||||||
package models.folder | ||||||
|
||||||
import com.scalableminds.util.accesscontext.DBAccessContext | ||||||
import com.scalableminds.util.tools.Fox | ||||||
import com.scalableminds.util.tools.{Fox, JsonHelper} | ||||||
import com.scalableminds.util.tools.Fox.{bool2Fox, option2Fox} | ||||||
import com.scalableminds.webknossos.schema.Tables._ | ||||||
import com.typesafe.scalalogging.LazyLogging | ||||||
import models.organization.{Organization, OrganizationDAO} | ||||||
import models.team.{TeamDAO, TeamService} | ||||||
import models.user.User | ||||||
import play.api.libs.json.{JsObject, Json, OFormat} | ||||||
import play.api.libs.json.{JsArray, JsObject, Json, OFormat} | ||||||
import slick.jdbc.PostgresProfile.api._ | ||||||
import slick.lifted.Rep | ||||||
import slick.sql.SqlAction | ||||||
|
@@ -19,11 +19,11 @@ import javax.inject.Inject | |||||
import scala.annotation.tailrec | ||||||
import scala.concurrent.ExecutionContext | ||||||
|
||||||
case class Folder(_id: ObjectId, name: String) | ||||||
case class Folder(_id: ObjectId, name: String, metadata: JsArray) | ||||||
|
||||||
case class FolderWithParent(_id: ObjectId, name: String, _parent: Option[ObjectId]) | ||||||
case class FolderWithParent(_id: ObjectId, name: String, metadata: JsArray, _parent: Option[ObjectId]) | ||||||
|
||||||
case class FolderParameters(name: String, allowedTeams: List[ObjectId]) | ||||||
case class FolderParameters(name: String, allowedTeams: List[ObjectId], metadata: JsArray) | ||||||
object FolderParameters { | ||||||
implicit val jsonFormat: OFormat[FolderParameters] = Json.format[FolderParameters] | ||||||
} | ||||||
|
@@ -48,17 +48,21 @@ class FolderService @Inject()(teamDAO: TeamDAO, | |||||
teamService.publicWrites(t, requestingUserOrganization)) ?~> "dataset.list.teamWritesFailed" | ||||||
isEditable <- folderDAO.isEditable(folder._id) | ||||||
} yield | ||||||
Json.obj("id" -> folder._id, | ||||||
"name" -> folder.name, | ||||||
"allowedTeams" -> teamsJs, | ||||||
"allowedTeamsCumulative" -> teamsCumulativeJs, | ||||||
"isEditable" -> isEditable) | ||||||
Json.obj( | ||||||
"id" -> folder._id, | ||||||
"name" -> folder.name, | ||||||
"metadata" -> folder.metadata, | ||||||
"allowedTeams" -> teamsJs, | ||||||
"allowedTeamsCumulative" -> teamsCumulativeJs, | ||||||
"isEditable" -> isEditable | ||||||
) | ||||||
|
||||||
def publicWritesWithParent(folderWithParent: FolderWithParent, allEditableIds: Set[ObjectId]): JsObject = | ||||||
Json.obj( | ||||||
"id" -> folderWithParent._id, | ||||||
"name" -> folderWithParent.name, | ||||||
"parent" -> folderWithParent._parent, | ||||||
"metadata" -> folderWithParent.metadata, | ||||||
"isEditable" -> allEditableIds.contains(folderWithParent._id) | ||||||
) | ||||||
|
||||||
|
@@ -117,7 +121,7 @@ class FolderService @Inject()(teamDAO: TeamDAO, | |||||
remainingPathNames match { | ||||||
case pathNamesHead :: pathNamesTail => | ||||||
for { | ||||||
newFolder <- Fox.successful(Folder(ObjectId.generate, pathNamesHead)) | ||||||
newFolder <- Fox.successful(Folder(ObjectId.generate, pathNamesHead, JsArray.empty)) | ||||||
_ <- folderDAO.insertAsChild(parentFolderId, newFolder) | ||||||
folderId <- createMissingFoldersForPathNames(newFolder._id, pathNamesTail) | ||||||
} yield folderId | ||||||
|
@@ -133,10 +137,16 @@ class FolderDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionContext) | |||||
protected def isDeletedColumn(x: Folders): Rep[Boolean] = x.isdeleted | ||||||
|
||||||
protected def parse(r: FoldersRow): Fox[Folder] = | ||||||
Fox.successful(Folder(ObjectId(r._Id), r.name)) | ||||||
for { | ||||||
metadata <- JsonHelper.parseAndValidateJson[JsArray](r.metadata.getOrElse("[]")).toFox | ||||||
MichaelBuessemeyer marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
folder <- Fox.successful(Folder(ObjectId(r._Id), r.name, metadata)) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
} yield folder | ||||||
|
||||||
private def parseWithParent(t: (String, String, Option[String])): Fox[FolderWithParent] = | ||||||
Fox.successful(FolderWithParent(ObjectId(t._1), t._2, t._3.map(ObjectId(_)))) | ||||||
private def parseWithParent(t: (String, String, Option[String], Option[String])): Fox[FolderWithParent] = | ||||||
for { | ||||||
metadata <- JsonHelper.parseAndValidateJson[JsArray](t._3.getOrElse("[]")).toFox | ||||||
folderWithParent <- Fox.successful(FolderWithParent(ObjectId(t._1), t._2, metadata, t._4.map(ObjectId(_)))) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
} yield folderWithParent | ||||||
|
||||||
override protected def readAccessQ(requestingUserId: ObjectId): SqlToken = | ||||||
readAccessQWithPrefix(requestingUserId, q"") | ||||||
|
@@ -246,6 +256,12 @@ class FolderDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionContext) | |||||
_ <- run(q"UPDATE webknossos.folders SET name = $name WHERE _id = $folderId".asUpdate) | ||||||
} yield () | ||||||
|
||||||
def updateMetadata(folderId: ObjectId, metadata: JsArray)(implicit ctx: DBAccessContext): Fox[Unit] = | ||||||
for { | ||||||
_ <- assertUpdateAccess(folderId) | ||||||
_ <- run(q"UPDATE webknossos.folders SET metadata = $metadata WHERE _id = $folderId".asUpdate) | ||||||
} yield () | ||||||
|
||||||
def findAllEditableIds(implicit ctx: DBAccessContext): Fox[List[ObjectId]] = | ||||||
for { | ||||||
updateAccessQuery <- accessQueryFromAccessQ(updateAccessQ) | ||||||
|
@@ -269,7 +285,7 @@ class FolderDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionContext) | |||||
for { | ||||||
accessQueryWithPrefix <- accessQueryFromAccessQWithPrefix(readAccessQWithPrefix, prefix = q"f.") | ||||||
accessQuery <- readAccessQuery | ||||||
rows <- run(q"""SELECT f._id, f.name, fp._ancestor | ||||||
rows <- run(q"""SELECT f._id, f.name, f.metadata, fp._ancestor | ||||||
FROM webknossos.folders_ f | ||||||
JOIN webknossos.folder_paths fp -- join to find immediate parent, this will also kick out self | ||||||
ON f._id = fp._descendant | ||||||
|
@@ -278,11 +294,11 @@ class FolderDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionContext) | |||||
FROM webknossos.folder_paths | ||||||
WHERE _ancestor = $folderId) | ||||||
AND $accessQueryWithPrefix | ||||||
UNION ALL SELECT _id, name, NULL -- find self again, with no parent | ||||||
UNION ALL SELECT _id, name, metadata, NULL -- find self again, with no parent | ||||||
FROM webknossos.folders_ | ||||||
WHERE _id = $folderId | ||||||
AND $accessQuery | ||||||
""".as[(String, String, Option[String])]) | ||||||
""".as[(String, String, Option[String], Option[String])]) | ||||||
parsed <- Fox.combined(rows.toList.map(parseWithParent)) | ||||||
} yield parsed | ||||||
|
||||||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,69 @@ | ||
START TRANSACTION; | ||
|
||
do $$ begin ASSERT (select schemaVersion from webknossos.releaseInformation) = 117, 'Previous schema version mismatch'; end; $$ LANGUAGE plpgsql; | ||
|
||
DROP VIEW webknossos.folders_; | ||
DROP VIEW webknossos.datasets_; | ||
|
||
-- Folder part | ||
ALTER TABLE webknossos.folders ADD COLUMN metadata JSONB DEFAULT '[]'; | ||
ALTER TABLE webknossos.folders ADD CONSTRAINT metadataIsJsonArray CHECK(jsonb_typeof(metadata) = 'array'); | ||
|
||
-- Dataset part | ||
ALTER TABLE webknossos.datasets ADD COLUMN metadata JSONB DEFAULT '[]'; | ||
ALTER TABLE webknossos.datasets ADD CONSTRAINT metadataIsJsonArray CHECK(jsonb_typeof(metadata) = 'array'); | ||
-- Add existing details on species to metadata | ||
UPDATE webknossos.datasets | ||
SET metadata = CASE | ||
WHEN details->>'species' IS NOT NULL THEN | ||
metadata || jsonb_build_array( | ||
jsonb_build_object( | ||
'type', 'string', | ||
'index', 2, | ||
'key', 'species', | ||
'value', details->>'species' | ||
) | ||
) | ||
ELSE | ||
metadata | ||
END; | ||
-- Add existing details on brain region to metadata | ||
UPDATE webknossos.datasets | ||
SET metadata = CASE | ||
WHEN details->>'brainRegion' IS NOT NULL THEN | ||
metadata || jsonb_build_array( | ||
jsonb_build_object( | ||
'type', 'string', | ||
'index', 2, | ||
'key', 'brainRegion', | ||
'value', details->>'brainRegion' | ||
) | ||
) | ||
ELSE | ||
metadata | ||
END; | ||
|
||
-- Add existing details on acquisition to metadata | ||
UPDATE webknossos.datasets | ||
SET metadata = CASE | ||
WHEN details->>'acquisition' IS NOT NULL THEN | ||
metadata || jsonb_build_array( | ||
jsonb_build_object( | ||
'type', 'string', | ||
'index', 2, | ||
'key', 'acquisition', | ||
'value', details->>'acquisition' | ||
) | ||
) | ||
ELSE | ||
metadata | ||
END; | ||
|
||
-- Drop details | ||
ALTER TABLE webknossos.datasets DROP COLUMN details; | ||
|
||
CREATE VIEW webknossos.folders_ as SELECT * FROM webknossos.folders WHERE NOT isDeleted; | ||
CREATE VIEW webknossos.datasets_ as SELECT * FROM webknossos.datasets WHERE NOT isDeleted; | ||
UPDATE webknossos.releaseInformation SET schemaVersion = 118; | ||
|
||
COMMIT TRANSACTION; |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is the "index" needed? If the data model is an array, the ordering should already be fixed, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, you are right. I added it to support the potential free rearrangement of the metadata entries.
But you are right, the entry order can be stored via the array index. In the frontend the entries have an index to make the code easier. -> I could remove the index and always re-add it in the frontend 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Its now implemented that way