Skip to content

Stateless Portal Design Documentation

Jump to bottom Edit New page
David Nahodil edited this page Jun 13, 2014 · 2 revisions

Reference Internal Discussion 37, Internal Discussion 37.

Stateless portal stack objective - removing the attached portal database and associated internal configuration. Migrate to metadata record within GeoNetwork MEST, portal server (grails app) and GeoServer (through plugins).

Advantages of moving to a stateless portal

  • Portal will become scalable - general performance of the system improves proportional to new resources added.
  • Implicit implementation of multi-WFS records and collections that consist of multiple WMS layers (eg. ALA).
  • Development is easier and the codebase is cleaner.
  • Configuration is fire and forget: instead of duplicated configuration in 3 different locations (portal database, GeoNetwork MEST, GeoServer), configuration is required only once in each location (which also makes logical sense from a design perspective).
  • A significant reduction in the number of steps required by project officers to add and configure layers within the portal.
  • Web apps such as WFS scanner and WMS scanner can be retired and replaced with GeoServer plugins. Associated increase in developer efficiency by not having to manage and fix bugs in these apps.
  • Key component of the design is moving javascript portal logic to the portal server (grails). The grails server provides a centralised hub to manage requests from steps 1, 2 and 3 in the portal. Again, simplifies the javascript code base and consolidates portal logic to one location.
  • Security and architectural enhancements as the grails portal server acts as middleware between javascript app and backend applications (GeoServer, GeoNetwork, Gogoduck, NcWMS).
  • Middleware grails portal server strictly enforces differentiation between server and client controls – enforces good security practice and permits system throttling where required.

Stateless Portal Functionality

Stateless portal functionality in steps 1, 2 and 3 of the portal can be broken down to the following 9 functions. We propose that a GeoServer plugin be constructed to handle filter and style queries:

  • Step 1:

    • search(query_terms): returns the results of a layer search in step 1 based on the terms in the query (not significantly changed compared to current portal implementation). However, search can decide whether to query spatial search or GeoNetwork depending on the query terms provided.
    • getMotd(): returns the message of the day from config file (subject to implementation decisions).

  • Step 2:

    • getMetadataRecord(record_id): returns the entire metadata record, maintaining the XML hierarchy in a javascript object.
    • getFilters(geoserver_address, layer_name): returns the filter attributes from GeoServer based on a specified GeoServer layer name (requires GeoServer plugin).
    • getFilterValues(geoserver_address, layer_name, filter_name): returns the filter values from GeoServer for a specified filter (requires GeoServer plugin).
    • getTile(geoserver_address, layer_name, params): returns a tile from GeoServer for the specified layer name (link with squid caching) - AKA openLayersProxy.
    • getStyles(geoserver_address, layer_name, version): returns a style from GeoServer corresponding to the GeoServer layer name and specified style version (essentially treat this as another filter option, can be built into GeoServer plugin).

  • Step 3:

    • downloadSync(format, geoserver_address, layer_name, subset_parameters): returns download options for non-aggregated data (CSV, list of urls, un-subsetted NetCDF) given a required format, GeoServer, layer name and parameters for the subset.
    • downloadAsync(format, geoserver_address, layer_name, subset_parameters, aggregator_address, email_address, extra_params): returns download options for aggregated data (subsetted NetCDF). Requires an address for the aggregation service used, email address provided by the user, and extra parameters (such as depth for CARS layers).

Use Cases

Use case 1: User performs a basic search, selects Argo layer, subsets using spatial constraint bounding box filter, downloads as CSV. Note that gs address == the address of the GeoServer instance for all use cases.

  1. Search(''): basic search with no query parameters, directly query GeoNetwork.
  2. getMetadataRecord(argo record id): return metadata record for argo and convert into Javascript object.
  3. getFilters(gs address, argo layer name): return filter attributes from GeoServer based on the name of the Argo layer.
  4. getFilterValues(gs address, argo layer name, spatial filter=bounding box): return filter values for a particular attribute for Argo layer.
  5. downloadSync(CSV, gs address, argo layer name, spatial constraint bounding box): returns a CSV download filtered using bounding box.

Use case 2: User performs a search, selecting platform = mooring, selects ANMN passive acoustics, subsets using spatial constraint bounding box filter, downloads as un-subsetted netCDF.

  1. Search(platform=mooring): basic search with platform parameter, directly query GeoNetwork.
  2. getMetadataRecord(ANMN record id): return metadata record for ANMN and convert into Javascript object.
  3. getFilters(gs address, ANMN layer name): return filter attributes from GeoServer based on the name of the ANMN layer.
  4. getFilterValues(gs address, ANMN layer name, spatial filter=bounding box): return filter values for a particular attribute for ANMN layer.
  5. downloadSync(netCDF, gs address, ANMN layer name, spatial constraint bounding box): returns a netCDF download filtered using bounding box.

Use case 3: User performs a search, selecting platform = mooring and spatial filter over Australia, selects ANMN velocity time series, subsets using temporal filter between 20/05/2013 and 25/06/2013, downloads as list of URLs.

  1. Search(platform=mooring, spatial search Australia): spatial search with platform parameter, query spatial search.
  2. getMetadataRecord(ANMN velocity record id): return metadata record for ANMN velocity layer and convert into Javascript object.
  3. getFilters(gs address, ANMN velocity layer name): return filter attributes from GeoServer based on the name of the ANMN velocity layer.
  4. getFilterValues(gs address, ANMN velocity layer name, temporal filter between 20/05/2013 and 25/06/2013): return filter values for temporal filter attribute for ANMN velocity layer.
  5. downloadSync(URL list, gs address, ANMN velocity layer name, temporal constraint between 20/05/2013 and 25/06/2013): returns a netCDF download filtered using temporal constraint.

Use case 4: User performs a basic search with no criteria, selects SRS GHRSST subskin layer, subsets using temporal filter between 18/05/2014 and 24/05/2014 and bounding box, selects a monochrome style, downloads as aggregated netCDF.

  1. Search(''): basic search with no query parameters, directly query GeoNetwork.
  2. getMetadataRecord(SRS record id): return metadata record for SRS and convert into Javascript object.
  3. getFilters(gs address, SRS layer name): return filter attributes from GeoServer based on the name of the SRS layer.
  4. getFilterValues(gs address, SRS layer name, temporal filter between 18/05/2014 and 24/05/2014): return filter values for temporal filter attribute for SRS layer.
  5. getFilterValues(gs address, SRS layer name, spatial filter=bounding box): return filter values for spatial filter attribute for SRS layer.
  6. getStyles(gs address, SRS layer name, monochrome): return a monochrome style from GeoServer for the SRS layer.
  7. downloadAsync(netCDF, gs address, SRS layer name, temporal constraint between 18/05/2014 and 24/05/2014; bounding box spatial constraint, gogoduck server address, email): returns an aggregated netCDF file from gogoduck and sends to the user's specified email address.

Use case 5: User performs a basic search with no criteria, selects CARS weekly temperature layer, subsets using temporal filter between 17/05/2013 and 24/05/2013, uses a monochrome style and specifies a depth of 1 metre, downloads as aggregated netCDF.

  1. Search(''): basic search with no query parameters, directly query GeoNetwork.
  2. getMetadataRecord(CARS record id): return metadata record for CARS and convert into Javascript object.
  3. getFilters(gs address, CARS layer name): return filter attributes from GeoServer based on the name of the CARS layer.
  4. getFilterValues(gs address, CARS layer name, temporal filter between between 17/05/2013 and 24/05/2013): return filter values for temporal filter attribute for CARS layer.
  5. getStyles(gs address, CARS layer name, monochrome): return a monochrome style from GeoServer for the CARS layer.
  6. downloadAsync(netCDF, gs address, CARS layer name, temporal constraint between 17/05/2013 and 24/05/2013, gogoduck server address, email, depth=1m): returns an aggregated netCDF file from gogoduck and sends to the user's specified email address. Note depth is added as an additional parameter.

Current Portal Configuration to be Migrated

List of current portal config (tables within current portal database) and migration information to move to a stateless portal stack:

  • aodaac_job: Contains a record of aodaac jobs run in the portal and can be removed.
  • aodaac_product_link: Contains the product ids of aodaac layers configured in the portal. Move to metadata record.
  • config: Single row configuration - move to separate config file.
  • config_layer: Empty table, delete.
  • database_changelog_lock: Liquibase injections no longer required without portal db, delete.
  • filter: contains filter attributes that can be collected using GeoServer plugin (contains redundant wms start and end date). Can be deleted.
  • filter_possible_values: Contains filer values that can be collected using GeoServer plugin (need to identify field filter_possible_values_idx). Can be deleted.
  • hibernate_sequence: legacy table, delete.
  • layer: Contains fundamental layer data (title, GeoServer layer name, default bbox, server id, etc). Move fundamental information to metadata record. Some info (such as default bbox) can be queried using GeoServer plugin.
  • layer_metadata_url: Need to confirm the use of this information.
  • layer_style: Not required, styles can be queried directly from GeoServer. Delete.
  • layer_view_parameters: Contains information about the starting viewport for each layer. This information must be migrated to metadata or GeoServer (best place is GeoServer if possible).
  • menu: baselayer menu information. Can be deleted.
  • menu_item: See above, delete.
  • metadata_url: Hyperlink references to GeoNetwork. Old records present here. Probably can be deleted, but need to confirm.
  • motd: Move to config file (subject to implementation decisions).
  • operation: Url construction, generate on server-side. Delete.
  • organisation_type: Legacy organisation information, delete.
  • portal_user, portal_user_permissions, portal_user_roles, user_role, user_role_permissions: All relate to portal administrator permissions. Delete, no longer required with stateless portal stack.
  • search: Empty table, delete.
  • search_filter: Empty table, delete.
  • search_filter_value: Empty table, delete.
  • server_operation: Legacy server operation ids, delete.
  • server_portal_user: Security hole here, migrate to IP access control list. Stateless portal fixes security issue.
  • snapshots, snapshot_layer: Snapshots are legacy, delete both tables.
  • wmsdimension: Empty, delete.

Suggested Task Breakdown

  1. Update metadata for WFS and NcWMS record to contain protocols in the correct format and GeoServer layer names.
  2. Modify javascript side of the portal to collect the complete metadata record as a javascript object (dependent on final implementation, could also use object hierarchy to access metadata elements within the portal). Modify step 2 and 3 code to use protocol and GeoServer layer name information within the metadata record for WFS, Gogoduck and aodaac downloads.
  3. Delete unused configuration options within the portal database and test functionality.
  4. Migrate configuration options listed above (single row config in config table) to separate configuration file and make configuration file accessible for the javascript portal client.
  5. Construct GeoServer plugin for querying filter attributes, values and style information from within the portal.
  6. Modify javascript portal and portal server to incorporate logic for using the GeoServer plugin to get filter and style information.
  7. Migrate step 2 and 3 javascript logic where possible from javascript portal client to grails portal server. Ideally, the javascript side should be as “dumb” as possible by this stage.
  8. Delete the portal's internal database, it's no longer needed!
Add a custom footer
Add a custom sidebar
Clone this wiki locally