Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CORE-1889 Tapis v3 Migration #270

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

CORE-1889 Tapis v3 Migration #270

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
15e46d8
CORE-1889 Initial update to mescal v4
psarando Mar 18, 2023
484a8ad
CORE-1889 Rename Agave references to Tapis
psarando May 24, 2023
844ca3e
CORE-1897 Update Tapis oauth2 token endpoint integration
psarando May 26, 2023
e2bb31c
CORE-1897 Remove apps.persistence.oauth/validate-token-type
psarando Jun 1, 2023
50b50ee
CORE-1900 Update Tapis app sharing calls
psarando May 24, 2024
8294722
CORE-1901 Remove unused Tapis prepare-step-submission function
psarando May 25, 2024
f4383de
CORE-1901 Update Tapis job type
psarando May 31, 2024
9511b0b
CORE-1997 Replace /callbacks/agave-job with /callbacks/tapis-job
psarando May 31, 2024
407a907
CORE-1901 Fix FileFolderInput relaunch param formatting
psarando Jun 27, 2024
9d6ada7
CORE-1901 Allow Agave jobs only in job listings
psarando Jun 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions project.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@
[medley "1.4.0"]
[metosin/compojure-api "1.1.14"]
[org.cyverse/async-tasks-client "0.0.5"]
[org.cyverse/authy "2.8.0"]
[org.cyverse/authy "3.0.1"]
[org.cyverse/clojure-commons "3.0.9"]
[org.cyverse/debug-utils "2.9.0"]
[org.cyverse/kameleon "3.0.10"]
[org.cyverse/mescal "3.1.12"]
[org.cyverse/mescal "4.0.0-SNAPSHOT"]
[org.cyverse/metadata-client "3.1.2"]
[org.cyverse/common-cli "2.8.2"]
[org.cyverse/common-cfg "2.8.3"]
[org.cyverse/common-swagger-api "3.4.5"]
[org.cyverse/common-swagger-api "3.4.6-SNAPSHOT"]
[org.cyverse/cyverse-groups-client "0.1.9"]
[org.cyverse/permissions-client "2.8.4"]
[org.cyverse/service-logging "2.8.4"]
Expand Down
2 changes: 1 addition & 1 deletion src/apps/constants.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
(ns apps.constants
(:require [mescal.agave-de-v2.constants :as c]))
(:require [mescal.tapis-de-v3.constants :as c]))

(def de-system-id "de")
(def hpc-system-id c/hpc-system-id)
Expand Down
6 changes: 3 additions & 3 deletions src/apps/persistence/app_groups.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@

(defn get-app-group-hierarchy
"Gets the app group hierarchy rooted at the node with the given identifier."
[root-id {:keys [agave-enabled app-ids] :or {agave-enabled "false"}}]
[root-id {:keys [tapis-enabled app-ids] :or {tapis-enabled "false"}}]
(if (seq app-ids)
(select (sqlfn :app_category_hierarchy root-id (Boolean/parseBoolean agave-enabled) (sql-array "uuid" app-ids)))
(select (sqlfn :app_category_hierarchy root-id (Boolean/parseBoolean agave-enabled)))))
(select (sqlfn :app_category_hierarchy root-id (Boolean/parseBoolean tapis-enabled) (sql-array "uuid" app-ids)))
(select (sqlfn :app_category_hierarchy root-id (Boolean/parseBoolean tapis-enabled)))))

(defn get-visible-workspaces
"Gets the list of workspaces that are visible to the user with the given workspace
Expand Down
14 changes: 7 additions & 7 deletions src/apps/persistence/app_listing.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,10 +85,10 @@
(where query {:id [not-in omitted-app-ids]})
query))

(defn- add-agave-pipeline-where-clause
[query {agave-enabled? :agave-enabled :or {agave-enaled? "false"}}]
(let [agave-enabled? (Boolean/parseBoolean agave-enabled?)]
(if-not agave-enabled?
(defn- add-tapis-pipeline-where-clause
[query {tapis-enabled? :tapis-enabled :or {tapis-enabled? "false"}}]
(let [tapis-enabled? (Boolean/parseBoolean tapis-enabled?)]
(if-not tapis-enabled?
(where query {:step_count :task_count})
query)))

Expand Down Expand Up @@ -151,7 +151,7 @@
(add-app-id-where-clause query-opts)
(add-app-type-where-clause query-opts)
(add-omitted-app-id-where-clause query-opts)
(add-agave-pipeline-where-clause query-opts)))
(add-tapis-pipeline-where-clause query-opts)))

(defn- get-app-count-base-query
"Adds a where clause to the get-all-apps-count-base-query, filtering out `deleted` apps."
Expand Down Expand Up @@ -275,7 +275,7 @@
(add-app-id-where-clause query_opts)
(add-app-type-where-clause query_opts)
(add-omitted-app-id-where-clause query_opts)
(add-agave-pipeline-where-clause query_opts)))
(add-tapis-pipeline-where-clause query_opts)))

(defn- get-app-listing-base-query
"Adds a where clause to the get-all-apps-listing-base-query, filtering out `deleted` apps."
Expand Down Expand Up @@ -453,7 +453,7 @@
(add-app-id-where-clause params)
(add-app-type-where-clause params)
(add-omitted-app-id-where-clause params)
(add-agave-pipeline-where-clause params)
(add-tapis-pipeline-where-clause params)
(select))))

(defn list-public-apps-by-user
Expand Down
10 changes: 5 additions & 5 deletions src/apps/persistence/app_search.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,10 @@
(where {:es.app_version_id :v.id
:es.task_id nil})))

(defn- add-agave-pipeline-where-clause
[q {agave-enabled? :agave-enabled :or {agave-enaled? "false"}}]
(let [agave-enabled? (Boolean/parseBoolean agave-enabled?)]
(if-not agave-enabled?
(defn- add-tapis-pipeline-where-clause
[q {tapis-enabled? :tapis-enabled :or {tapis-enabled? "false"}}]
(let [tapis-enabled? (Boolean/parseBoolean tapis-enabled?)]
(if-not tapis-enabled?
(where q (not (exists (external-step-subselect))))
q)))

Expand Down Expand Up @@ -88,7 +88,7 @@
(add-app-id-where-clause q query-opts)
(add-app-type-where-clause q query-opts)
(add-omitted-app-id-where-clause q query-opts)
(add-agave-pipeline-where-clause q query-opts)
(add-tapis-pipeline-where-clause q query-opts)
(add-search-term-where-clauses q search-term (:pre-matched-app-ids query-opts))
(add-non-admin-where-clauses q query-opts)))

Expand Down
3 changes: 2 additions & 1 deletion src/apps/persistence/jobs.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,12 @@

(def de-job-type "DE")
(def agave-job-type "Agave")
(def tapis-job-type "Tapis")
(def interactive-job-type "Interactive")
(def osg-job-type "OSG")

(def de-client-name c/de-system-id)
(def agave-client-name c/hpc-system-id)
(def tapis-client-name c/hpc-system-id)
(def interactive-client-name c/interactive-system-id)
(def osg-client-name c/osg-system-id)
(def combined-client-name "combined")
Expand Down
10 changes: 1 addition & 9 deletions src/apps/persistence/oauth.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,6 @@
(:import [java.sql Timestamp]
[java.util UUID]))

(defn- validate-token-type
"Verifies that the token type is supported."
[token-type]
(when-not (= "bearer" token-type)
(throw+ {:type :clojure-commons.exception/illegal-argument
:error (str "OAuth 2.0 token type, " token-type ", is not supported.")})))

(defn- user-id-subselect
"Returns a subselect statement to find a user ID."
[username]
Expand Down Expand Up @@ -73,8 +66,7 @@

(defn store-access-token
"Stores information about an OAuth access token in the database."
[api-name username {:keys [token-type expires-at refresh-token access-token]}]
(validate-token-type token-type)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tapis no longer returns a token-type field.

[api-name username {:keys [expires-at refresh-token access-token]}]
(if (has-access-token api-name username)
(replace-access-token api-name username expires-at refresh-token access-token)
(insert-access-token api-name username expires-at refresh-token access-token)))
Expand Down
26 changes: 12 additions & 14 deletions src/apps/routes/callbacks.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,20 @@
(:use [apps.routes.schemas.callback]
[common-swagger-api.schema]
[common-swagger-api.schema.analyses :only [AnalysisIdPathParam]]
[common-swagger-api.schema.callbacks :only [AgaveJobStatusUpdateParams AgaveJobStatusUpdate]]
[common-swagger-api.schema.callbacks :only [TapisJobStatusUpdate]]
[ring.util.http-response :only [ok]])
(:require [apps.service.callbacks :as callbacks]))

(defroutes callbacks
(POST "/de-job" []
:body [body (describe DeJobStatusUpdate "The updated job status information.")]
:summary "Update the status of of a DE analysis."
:description "The jex-events service calls this endpoint when the status of a DE analysis
changes"
(ok (callbacks/update-de-job-status body)))
(POST "/de-job" []
:body [body (describe DeJobStatusUpdate "The updated job status information.")]
:summary "Update the status of of a DE analysis."
:description "The jex-events service calls this endpoint when the status of a DE analysis changes"
(ok (callbacks/update-de-job-status body)))

(POST "/agave-job/:job-id" []
:path-params [job-id :- AnalysisIdPathParam]
:body [body (describe AgaveJobStatusUpdate "The updated job status information.")]
:query [params AgaveJobStatusUpdateParams]
:summary "Update the status of an Agave analysis."
:description "The DE registers this endpoint as a callback when it submts jobs to Agave."
(ok (callbacks/update-agave-job-status job-id (:lastUpdated body) params))))
(POST "/tapis-job/:job-id" []
:path-params [job-id :- AnalysisIdPathParam]
:body [body (describe TapisJobStatusUpdate "The updated job status information.")]
:summary "Update the status of an Tapis analysis."
:description "The DE registers this endpoint as a callback when it submts jobs to Tapis."
(ok (callbacks/update-tapis-job-status job-id body))))
5 changes: 3 additions & 2 deletions src/apps/service/apps/combined.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
jp/combined-client-name)

(getJobTypes [_]
(mapcat #(.getJobTypes %) clients))
(conj (mapcat #(.getJobTypes %) clients) jp/agave-job-type))

(listSystemIds [_]
(mapcat #(.listSystemIds %) clients))
Expand Down Expand Up @@ -365,4 +365,5 @@
(.hasAppPermission (util/get-apps-client clients system-id) username system-id app-id required-level))

(supportsJobSharing [_ job-step]
(.supportsJobSharing (util/apps-client-for-job-step clients job-step) job-step)))
(and (not= (:job_type job-step) jp/agave-job-type)
(.supportsJobSharing (util/apps-client-for-job-step clients job-step) job-step))))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this namespace are only for allowing users to still list their old Agave jobs, even if they can't relaunch them anymore.

Maybe it would be better to re-add an Agave type that handles the logic in these methods, and throws an error for the rest? 🤷

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems pretty clean to me, but this is also understandable, so I'm open to leaving it as is.

4 changes: 2 additions & 2 deletions src/apps/service/apps/combined/jobs.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@
[{app-version-id :app_version_id} {app-step-number :app_step_number}]
(nth (ap/load-app-steps app-version-id) (dec app-step-number)))

(defn- prepare-agave-job-step-submission
(defn- prepare-tapis-job-step-submission
[job-info job-step submission]
(let [app-step (get-current-app-step job-info job-step)
submission (prepare-common-job-step-submission job-info job-step submission)]
Expand All @@ -97,7 +97,7 @@
[job-info job-step submission]
(if (cu/is-de-job-step? job-step)
(prepare-de-job-step-submission job-info job-step submission)
(prepare-agave-job-step-submission job-info job-step submission)))
(prepare-tapis-job-step-submission job-info job-step submission)))

(defn- record-step-submission
[job-id step-number external-id]
Expand Down
4 changes: 2 additions & 2 deletions src/apps/service/apps/jobs/util.clj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
(:use [slingshot.slingshot :only [try+ throw+]])
(:require [apps.clients.data-info :as data-info]
[apps.persistence.jobs :as jp]
[apps.persistence.app-metadata :as am]
[apps.util.service :as service]
[clojure.string :as string]
[clojure.tools.logging :as log]
Expand Down Expand Up @@ -51,8 +52,7 @@
"FolderInput" "collection"
"MultiFileSelector" "many"})

(def input-types
(set (keys input-multiplicities)))
(def input-types am/param-ds-input-types)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The HPC FileFolderInput type was missing from this before, so inputs were not repopulating in the UI relaunch forms correctly.


(def output-multiplicities
{"FileOutput" "single"
Expand Down
Loading