bdre wf deployment logic
This feature will allow the deployment of BDRE generated oozie workflows along with user developed code to Hadoop cluster. The current deployment feature has a dependency on Jenkins which might not be possible in all implementations. This feature will remove the dependency from Jenkins and call the deployment scripts directly from BDRE via a REST interface.
Create a table called PROCESS_DEPLOYMENT_QUEUE with following Structure
| Column | Data Type | Property | Comments |
|---|---|---|---|
| deployment_id | Bigint | auto generated - not null | primary key |
| process_id | Int | FK of process table - not null | Parent process id |
| start_ts | timestamp | nullable | |
| insert_ts | timestamp | not null default current timestamp | |
| end_ts | timestamp | nullable | |
| deploy_status_id | small int | FK of DEPLOY_STATUS table. Default 'not started' | DEPLOY_STATUS is master table like process_status contains 3 status running, success, fail |
| user_id | int | not null | |
| bus_domain_id | smallint | not null | |
| process_type_id | smallint | not null | |
| deploy_script_location | varchar(1000) | nullable | Location of the deploy script to be delegated to. |
Stored CRUD procedures should be written for these table.
The insert proc that inserts a new record in PROCESS_DEPLOYMENT_QUEUE table should be passed with process_id and user_id only and it'll calculate the rest from defaults or by looking up from other tables(e.g. process table and deploy_script_location from properties of this process_id). If the property for deploy_script_location is not found then use null.
- deploy.sh script should call startDeploy and endDeploy APIs when it starts and completes.
- initDeploy(deployment_id) method should be written in the md_api project. It'll
- haltDeploy(deployment_id) method should be written in the md_api project. It would mark deployment for the deploy.sh as successfully complete.
- termDeploy(deployment_id) method should be written in the md_api project. It would mark deployment for the deploy.sh as failed.
- A master deploy.sh will be written to wrap the calls to the process type or process specific deploy scripts.
- For each bus domain and for each process type separate generic shell script will be used by default. It'll save the developers from mentioning the process type specific scripts again and again.
- If deploy_script_location is not null(i.e. a process specific script is specified in the properties table for that parent process) then the call would be delegated to that script.
- The deployment shell scripts can reside in Tomcat user's home directory. The deployment scripts base directory would be picked up from BDRE xml config file in md-commons project.
- Names of the shell scripts: deploy.sh will delegate the calls to <BDRE_DEPLOYMENT_SCRIPT_DIR>/<bus_domain>--deploy.sh
- The shell scripts written for deployment from Jenkins can be reused to some extent and are found in the Jack repo.
- Please have Cygwin installed to test the shell scripts if developing in Windows.
Regular CRUD REST API for PROCESS_DEPLOYMENT_QUEUE and PROCESS_DEPLOYMENT_STATUS tables. The PUT API that inserts a new record in PROCESS_DEPLOYMENT_QUEUE table should be passed with process_id only. user_id would be the logged in user and it can be obtained from Spring MVC API (already being used to obtain the user id during logging).
- The deploy buttons in the process page will call the rest API and pass the process id to the PUT method of ProcessDeploymentAPI.
- A new page called "Process Deployment Queue" would be added to the navigation which will contain the jTable to interact with the PROCESS_DEPLOYMENT_QUEUE table using the REST API.
- New inserts only take process id as the input.
- Edit allows to change status only.
- Delete should not allow deleting of running and completed deployment.
- Users are not allowed to create entries from the jTable. Set create=false(no + button for process_deployment_queue). They can only create it by clicking the deploy button in the Process page.
- User can delete or edit a deploy job. Users cannot edit following fields though deployment_id,process_id,start_ts,insert_ts,deploy_status_id.
- A standalone Java program will be written to call deploy.sh script with different parameters. It'll get a list of newly added non running deployment jobs and start them one by one. Note that the call to deploy.sh should not be blocked.