Skip to content

Latest commit

 

History

History
462 lines (338 loc) · 28.9 KB

revision-history.md

File metadata and controls

462 lines (338 loc) · 28.9 KB
Raw

Revision History

This file was last reviewed on 2023-11-20 Please, keep this note updated


5.7-SNAPSHOT

User Interface

  • Gene table includes new filtering options, including graph evidence distance and KnetScore (#692).
  • Gene and Evidence table, infinite scrolling replaced old pagination.
  • User queries and sample queries tailored to user plan/permissions.
  • Sample queries visualisation based on currently selected species and user role.
  • Sample queries box now has improved scaling.
  • Sample query file format extended to support user role (which is required to access a sample, #774).
  • Some improvements to the genemap visualisation (Map View) for better container size support on larger resolutions (#723).
  • Added several additional centromere locations of various species for displaying as grey lines in genemap (Map View).
  • The input box for Gene List Search now displays a visual indicator reflecting user role limits: 20 for free users, 100 for registered users, and unlimited for Pro users.
  • Updated Concept Selector icon to better fit the interface.

UI internals

  • Evidence table, matching genes download improved for performance (using browser HTTP cache).
  • Refactoring of chromosome region selection code (moved to its own .js files).
  • Library for user management (#768).
  • Js that processes gene/evidence table from the API upgraded to comply with the new pure-JSON format (see #655).
  • Old release-notes.html removed (#657).
  • Added additional user role for Registered users which increases Gene List Search limit from 20 to 100.

UI bugfixes

  • #773 Incorrect way to select rows during gene table filtering.
  • #777 Detail pop-up in the gene table with incorrect click event handling.
  • #780 Evidence table, pop-up gene list duplicated header.
  • #771 Gene list links in evidence view not working.
  • #778 Network view with too many genes for the current user role breaks the UI.

API

  • Gene table and Evidence table outputs now are pure JSON (cleaning and improving the "CSV inside JSON" old format). This improves the coding of API clients (#655).
  • Gene table output now includes the graph distance between genes and their evidences (based on semantic motifs, #743).
  • Deprecated API calls removed, /dataSource (#653), /ksHost (#792), /latestNetworkStats (#657).

API internals

  • More logging added to the the API client used for tests (#eebed3e).

Miscellanea

  • Migration to Java 17, new Java features available to developers.
  • Migration to recent versions of Tomcat and Jetty (used for building), with consequent migration of old J2EE libraries (eg, java.servlet to Jakarta, #74f27f0)
  • Neo4j mode migrated to Neo4 5.10.x
  • Various dependencies upgraded.
  • Minor enhancements to demo/aratiny launching scripts and continuous integration scripts.
  • Maven build, option to run aratiny in single-specie mode added (#a387ee9, #cf23443).
  • Several new example queries added to better showcase updated user limits.
  • Tools added (in knetminer-initializer) to start the knetminer-nova migration.

5.6.0-SNAPSHOT

  • Current snapshot (on 5.6 branch).

5.6

What's new?

KnetMiner 5.6 brings multispecies functionality, performance improvements, bug fixes, UI enhancements, website improvements, new tutorial and several new species and updated datasets.

Find the multispecies 5.6 (Beta) instances here:

  • Poaceae 3-Species containing: Arabidopsis Thaliana, Triticum aestivum and Oryza sativa Japonica.
  • Ascomycota 10-Species containing: Fusarium culmorum CS7071, Fusarium graminearum PH-1, Zymoseptoria tritici, Aspergillus fumigatus Af293, Aspergillus nidulans, Candida albicans, Magnaporthe oryzae, Neurospora crassa, Saccharomyces cerevisiae and Schizosaccharomyces pombe.

Key Features

1. Multispecies functionality

Multispecies functionality brings a major change to how KnetMiner functions and can be queried.

  • KnetMiner can now be configured (see "New YAML based Configuration" below) to serve a knowledge graph with multiple species (taxonomy ids).
  • Added a Species selector in the header which shows the list of species available in the config.yml.
  • Search messages and sample queries have been made species specific.
  • Genome Region Search has been updated to show chromosome names and gene numbers for selected species.
  • Gene View will only show genes belonging to the selected species.
  • Evidence View columns (p-value, total genes) are still based on all species data.
  • Map View automatically picks the chromosome map for the selected species.
  • Network View still shows nodes and relations across all species present in the KG.

Multispecies functionality (top left of new KnetMiner interface)

2. New YAML based Configuration

KnetMiner is now easier than ever to configure. We have introduced a new YAML-based configuration system, which enables more efficient customisation and extension of default settings through the use of inclusions, overrides and merges. This system supports advanced features such as special markers, rules and property interpolation.

For a comprehensive guide on creating and configuring a KnetMiner instance, please refer to our documentation available on GitHub.

Gene Synonyms dropdown

3. Gene names and synonyms

Show all synonyms in Gene View: allows users to view all synonyms for a gene, in Gene View.

  • Gene preferred names are now correctly selected by default.

Gene Synonyms dropdown

We now append the species initials to the front of genes in Network View.

  • Helps identify genes from various species in multi-species networks.
  • Stops users having to individually click on nodes to determine their species.

4. Network data exporter

Users can now export their network data (from Network view).

  • Both exporting methods will export the currently visible graph with node and edge information but without all details (which would normally be visible in the Info box).
  • Allows for exporting in Tabular format and Cytoscape desktop (json) formats.

Gene Synonyms dropdown

5. KnetScore visibility

Knetscore is now visible in Gene View

6. Evidence View Gene List functionality

Quickly return all the genes associated to a concept via semantic motifs in the current Knowledge Graph.

  • Allows users to quickly view all genes related to a particular Evidence
  • Quick copying of gene accessions for KnetMiner Gene List Search
  • Quick downloading of entire table

Gene list popup

7. KnetMiner (KnetGraph) programmatic endpoints homepage update

Available here.

  • Modernised visuals
  • Updated links
  • Improved key features (logo, key texts)

8. New Tutorial

Available here

The new and improved KnetMiner tutorial offers users a user-friendly platform to quickly troubleshoot problems or learn about features. We now also link to the tutorial page from within the KnetMiner header.

  • Improved navigation
  • Google indexed


Other new features

More meaningful P-values in Evidence View

  • Exponential values are now displayed correctly.
  • Decimal place count has been increased.

Improved evidence filtering

  • Added individual evidence names to key.
  • Users can now multi-select evidences to filter by, both in Gene and Evidence views.
  • New UI is translated into the evidence column too, with correct overflow and table scaling.

Added additional RefSeq gene IDs to Poaceae KG (Release 55+)

Improved gene identification by allowing users to query KnetMiner using a combination of gene ID formats. In 5.6, we've added:


Visual improvements

Added information ("i") tooltips

  • Several column headers in Gene and Evidence views now contain helpful tooltips which describe the column.

Evidence View headers with 'i'

Improved Evidence View handling of empty user Gene Lists

  • N/A for P-values column.
  • "0" for Gene List column.

New KnetMiner Gene page

  • Updated to visually match new KnetMiner interface.
  • Correctly retains gene list and keywords via URL.

Refined knowledge graph release notes

Available via an animated button beside the multispecies selector (top left).

Contains:

  • KG information: title, explainer, versioning and size details.
  • A detailed tabulated breakdown of which data is available in the graph.

Evidence view Node Label max character length increased

Evidence view Description/Node label can now display longer lengths of text.

New loading animation

We've removed the double-helix loading animation and simplified the animation into the search button itself.

image

KnetMiner is rebooting page

On the odd chance that a user finds us rebooting or updating the instance they selected, they are prompted about ongoing updates.

  • Contact us directs to /contact.
  • Reload button refreshes the page and tries to access KnetMiner again.
  • Home page returns to KnetMiner.com.

Concept Selector updated

Query suggestor has become a “Concept selector”.

  • Quickly map keywords to concepts from the knowledge graph.

Backend updates

  • Overhauled KnetMiner configuration.
  • New /graphinfo/concept-info endpoint to retrieve node properties.
  • The text mining tool (from KnetBuilder) has been extended to allow for stop words which should be excluded from the text mining search.
  • Bugfixes to Google Analytics for KnetMiner (with new configuration options for deployers).
  • Cleaned up dataset data and configuration throughout KnetMiner and KnetMiner dependencies.

APIs

  • API URL auto-discovery in the UI code.
  • New API to help the client get information on the available species and related information.
  • KnetMiner now uses the same labels as the CytoscapeJS plugin (from KnetBuilder).
  • Cleaning of JSON formats for the split genome API.
  • API exception logging has been improved.
  • Network API can now export plainJSON format.

ETL updates

  • New workflow for text mining, including (1) more gene name stop words and (2) expanded gene synonym searches
  • Updates to obviate NCBI PubMed Abstract download restrictions
  • Updates to use new UniProt REST interface
  • Collections of names (chemical and brand) of both fungicides and insecticides for text-mining training

New species in KnetMiner

  • Ascomycota:

    • Aspergillus Fumigatus
    • Fusarium Culmorum
    • Neurospora Crassa
    • Zymoseptoria Tritici
    • Aspergillus Nidulans
    • Fusarium Graminearum
    • Saccharomyces Cerevisiae
    • Candida Albicans
    • Magnaporthe Oryzae
    • Schizosaccharomyces Pombe

  • Brassica:

    • Brassica Napus
    • Brassica Oleracea
    • Brassica Rapa
    • Camelina Sativa
    • Arabidopsis Thaliana

  • Tropicana:

    • Arabidopsis Thaliana
    • Musa Acuminata
    • Coffea Canephora

New configurations

  • Brassica Camelina.

Minor visual updates

  • Display number of genes inputted in Gene List box.
  • "Clear Search Fields" button.
  • Added "Help" popup.
  • Removed background image.
  • Updated top banner color to white.
  • Added "Tutorial" and "Cite Us" buttons to banner.
  • Rounded buttons, boxes and sections.
  • Improved clarity of text pop ups, errors and notifications.
  • Added "Revert all filtering changes" button to Gene and Evidence View keys.
  • Moved positions of several UI features (amount of genes to show in gene view, unlinked/linked genes selectors, "X genes selected").
  • Swapped Evidence and Map view positions.
  • Network View more clearly separated from other views.
  • Network View button is now greyed out/disabled when no graph has been created.
  • Greyed out "Genes" box in Genome Region Search (as the value isn't user-adjustable).
  • Revamped Region Search "Add region" UI/UX.
  • Evidence view now correctly displays amount of "terms" selected.
  • Added meaningful icons to views buttons.
  • KnetMiner logo in banner now redirects to /products page.
  • Example queries auto-changes with species.
  • Now only show X on second (and beyond) Genome Region Search boxes.
  • Renamed Evidence View's "Description" column to "Node label".

Fixed bugs and internal improvements

  • Multiple concept names having preferred flags mistakenly set to true.
  • White text in several dropdowns displaying as white on white for users with darkmode active in their browser.
  • Page crashing when clicking example queries while having multiple open Genome Region Search boxes.
  • Removed incorrect numbering in Evidence View key.
  • Interactive legend not scaling correctly with many evidence types.
  • Methods to harmonise the gene and concept label displaying (#604).
  • Compatibility with Neo4j 4.4.
  • Heavy code refactoring, cleaning, etc for KnetMinerServer, KnetMinerDataSource and OndexLocalDataSource.
  • UI's Javascript refactoring (#621).

KnetMiner.com website updates

Misc


5.0

What's new?

KnetMiner 5.0 comes with a number of new features, speed improvements and a new freemium/premium version.

Key Features

  1. KnetMiner (https://knetminer.com) now also offers a range of Free and Pro features. The free version is a perfect starting point for using or testing the platform with a small number of genes. The Pro version enables searching the knowledge graph with any number of genes, creating very large gene networks and storing as many networks as you wish.

  2. The gene list search feature has received a major performance enhancement. It now supports wildcards (eg. search for all TPS* genes) and can perform a novel KG enrichment analysis. wheat_genelist_wildcard_search

  3. The Evidence View has seen major upgrades to allow users download the content and to generate gene networks for multiple enriched terms. wheat_evidenceview_new3

  4. The Network View has been vastly improved as well, with auto-filtering when the knetworks are too big and a new feature to export the content of the network (all visible genes) as a table. wheat_knetmaps2_new

  5. The network visualisation (knetmaps.js) has been enhanced with better node labelling to highlight important parts of the graph and to distinguish genes from different species by taxID. A new type of edge (dashed lines) has also been introduced to flag relationships which need to be treated with caution. wheat_knetmaps_alsonew

  6. KnetMiner generated gene networks can be easily saved to KnetSpace (https://knetminer.com/beta/knetspace/) and shared with colleagues and published online. wheat_knetspace

References and further details:

  1. Latest KnetMiner publication: https://onlinelibrary.wiley.com/doi/10.1111/pbi.13583
  2. KnetMiner covid19 data note: https://f1000research.com/articles/10-703/v1

Software improvements and bug fixes

Search and analysis

  • Free and Pro KnetMiner features added. Free users are those without a KnetSpace account or those on a KnetSpace Free Plan wile Pro users are subscribers of the KnetSpace Pro plan. Under the free version, users can search KnetMiner with upto 20 genes in their gene list and visualize a knetwork of max. 10 genes. Pro users can search KnetMiner with no limits on gene list size and can render knetworks to analyse upto 200 genes at once.
  • Search results messages now show detailed information such as number of linked, unlinked and not-found user genes based on user query and gene list, and also the time taken for the query to run.
  • Error messages now return more detailed and user-friendly information.
  • Users can now directly add/omit evidence terms to their search box from Evidence View.
  • Evidence view table can now be downloaded by users as a tab-delimited file.
  • Users can render a combined knetwork of multiple evidence terms from Evidence View.

KnetMiner API

  • New explainer implemented for KnetMiner genepage API that analyses the user query to generate a word summary of what the resulting knetwork contains.
  • KnetMiner knowledge graph (dataset) version now added to latestNetworkStats API to have it displayed in release_notes html page.

KnetMiner back-end

  • Migration to Java 11. Now you do need Java >= 11 to make knetminer work If you use Docker, our images use the right Java version automatically.
  • Big refactoring of the OndexServiceProvider component.
  • Wildcards now supported in the Gene List Search box (eg, 'MYB*', 'TT?')
  • knetminer-base image removed, now there are only two docker image levels.
  • Migration from Travis to GitHub Actions for CI builds.
  • KnetMiner now allows developer users to enable/disable knetspace login in their KnetMiners, and to enable/disable or re-route google analytics (keywords, gene lists, QTLs) to an Amazon S3 bucket as a set of useful metrics for internal analysis.
  • Extensive review of the codebase file structure and names.
  • Third-party dependency updates.

Network View (KnetMaps.js)

  • Users can now export all visible genes and associated information as a tab-delimited file
  • Network View now auto-filters large knetworks to let users visualize subsets and avoid browser issues.
  • The network visualization now has more advanced node labelling to highlight important parts of the graph and to distinguish genes by taxID as well, by coloring gene node labels appropriately, which is highly useful for multi-species KGs.
  • Network visualization now uses "dashed" edges to flag relationships that are not abdolute and need to be treated with caution.
  • Multiple new data sources hyperlinks added/updated to enable users to explore ARA-GWAS, ENSEMBL_PLANTS, wheat-expression, ARA-PHENO and other sources.

Other updates

Poaceae multi-species knowledge graph

A new multi-species poaceae knowledge graph (KG) integrates a vast amount of: genetic, genomic and literature data from the four most important global cereal crops: wheat, rice, maize, barley, combined with the model plant species Arabidopsis and Brachypodium. We have cleaned, processed, and integrated over 100 datasets which has resulted in a KG of 2 million nodes and 10 million relationships between them. This KG is available through through a paid KnetMiner resource subscribtion (request quote) and our free RDF-SPARQL and Neo4j-Cypher graph query endpoints (https://knetminer.com/data).

4.0

What's new?

  1. New KnetMiner branding

    • KnetMiner 4.0 sees a major revamp with an all-new website and new branding, logo, fonts, and separated sub- products and services with their own logo, content pages and demos.
    • KnetMiner now on wikipedia.

  2. New COVID-19 KnetMiner

  3. Save your knetworks in the all-new KnetSpace

    • KnetMiner users can now save their knowledge networks on KnetSpace (beta), our new collaborative knowledge discovery platform to manage KnetMiner knetworks.
    • New login feature added to KnetMiner with a “Sign in '' button in the header. Users can now sign-in using their KnetSpace username and password (sign-up is very quick). In future, we will add Google login and similar.
    • Previously, users could perform a keyword or gene list search in KnetMiner, generate an initial network for some genes and export a PNG image. As of KnetMiner 4.0, you can save it to your KnetSpace and can edit it anytime in future and share it with your students, colleagues or collaborators to help you find interesting links or curate the network collaboratively (eg hide certain nodes/edges, add labels, change layout).
    • You can give people permissions to either edit your original network or clone it to their own KnetSpace. Once you are happy with a network you can tweet the link or share the interactive network on your own website.
    • To save networks, simply click the new “Save network” button on the KnetMiner Network View tab and provide a name and description. This will quickly save the visible network, its layout and metadata information such as owner, date, version, source, genes, keywords used, etc. to your KnetSpace account. knetspace_git

  4. Free and Pro user subscription plans

    • We have added Free and Pro subscription plans. Sign-up to KnetMiner and KnetSpace is for free. The Free plan provides all standard KnetMiner features, plus allows users to store a limited number of user knetworks to KnetSpace. The Pro plan provides greater storage and allows users to make their saved Knetworks private. Choose a plan that’s right for you.

  5. Neo4j-Cypher integration (beta)

    • The graph traversal code was re-engineered to support the Cypher graph query language. This is the first major milestone to transition towards an alternative graph database backend. Cypher is a more expressive and more standardised query language. This feature is still experimental and requires a separate Neo4j database installation, based on our RDF and Neo4j export tools. The default KnetMiner 4.0 option is to use the old semantic motif search.

References and further details:

Bug fixes and software improvements

Search

  • Searches pertaining to a gene identifier, using the Gene List, are now exact.

API

Several new APIs have been implemented now, including the following:

  • API for KnetSpace host url address, as defined in the POM by variable knetminer.knetSpaceHost with the API URL given as /ws/{$speciesName}/ksHost
  • API for geneCount added which provides the size of the subgraph (number of nodes, edges) generated by KnetMiner for a given gene list. Given as /ws/{$speciesName}/geneCount?list=
  • DataSource information of the knowledge graph including source organisation, database version, provider, species tax ID, date of creation, and species name, is accessible via the /ws/{$speciesName}/dataSource API.

KnetMiner back-end

  • KnetMiner can now be customised and deployed with a subset of genes. For example, we limited the COVID-19 KnetMiner to call 8,000 human genes as the seed. The path to the seed gene list can be defined in the main KnetMiner POM file variable knetminer.backend.seedGenesFile. This means only the seed genes will appear in the KnetMiner search index. The default is to index all genes. Refer to this commit.
  • Users may now define the number of Publications exported in knowledge subgraphs by altering the main POM configuration variable defaultExportedPublicationCount. The default is set to 20 most recent publications.
  • The unconnected filter prior to JSON export was extended with additional concept classes
  • Metadata information for the knowledge graph can now be defined in the KnetMiner dataset/species POM.

Network view

  • Updated to KnetMaps.js v2.0.2.
  • Save PNG and JSON have been replaced with a ‘Save Knetwork’ button, as described in the KnetSpace integration section.
  • More data sources URLs added to support rendering of new data types.
  • Custom branding and fonts packaged for KnetMaps as well via gulp-configs.
  • Keyword highlighting logic updated and limited only to primary colours and some shades.

3.2

New features/enhancements

  • New genepage design using KnetMiner look & feel
  • New analytics logger for genome, network and genepage modes
  • Network View legend was enhanced
  • Release Notes stats table was expanded with further graph analytics
  • New search button
  • Updated footer