Skip to content

Latest commit

 

History

History
274 lines (184 loc) · 12.6 KB

README.md

File metadata and controls

274 lines (184 loc) · 12.6 KB
Raw

Kubeflow Pipelines Management Frontend

Tools you need

You need node v14 and npm. Recommend installing node and npm using https://github.com/nvm-sh/nvm. After installing nvm, you can install node v14 by nvm install 14.

Manage dev environment with npm

First time

  1. Clone this repo
  2. Navigate to frontend folder: cd $KFP_SRC/frontend.
  3. Install dependencies: npm ci.

npm ci makes sure your installed dependencies have the exact same version as others. (Usually, you just need to run this once, but after others installed extra dependencies, you need to run npm ci again to get package updates.)

Package management

Run npm install --save <package> (or npm i -S <package> for short) to install runtime dependencies and save them to package.json. Run npm install --save-dev <package> (or npm i -D <package> for short) to install dev dependencies and save them to package.json.

After adding a dependency, validate licenses are correctly added for all dependencies first by running npm run gen-licenses.

Daily workflow

You will see a lot of npm run xxx commands in instructions below, the actual script being run is defined in the "scripts" field of package.json. Development common scripts are maintained in package.json, and we use npm to call them conveniently.

npm next step

You can learn more about npm in https://docs.npmjs.com/about-npm/.

Start frontend development server

You can then do npm start to run a webpack dev server at port 3000 that watches the source files. It also redirects api requests to localhost:3001. For example, requesting the pipelines page sends a fetch request to http://localhost:3000/apis/v1beta1/pipelines, which is proxied by the webserver to http://localhost:3001/apis/v1beta1/pipelines, which should return the list of pipelines.

Follow the next section to start an api mock/proxy server to let localhost:3001 respond to api requests.

Start api mock/proxy server

Api mock server

This is the easiest way to start developing, but it does not support all apis during development.

Run npm run mock:api to start a mock backend api server handler so it can serve basic api calls with mock data.

If you want to port real MLMD store to be used for mock backend scenario, you can run the following command. Note that a mock MLMD store doesn't exist yet.

kubectl port-forward svc/metadata-envoy-service 9090:9090

Proxy to a real cluster

This requires you already have a real KFP cluster, you can proxy requests to it.

  1. Install Kubeflow Pipelines based on your use case and environment.
  2. Configure kubectl with access to your KFP cluster. (For GCP, follow Access to GCP cluster guide).
  3. Use the following table to determine which script to run.
What to develop? Script to run Extra notes
Client UI NAMESPACE=kubeflow npm run start:proxy
Client UI + Node server NAMESPACE=kubeflow npm run start:proxy-and-server You need to rerun the script every time you edit node server code.
Client UI + Node server (debug mode) NAMESPACE=kubeflow npm run start:proxy-and-server-inspect Same as above, and you can use chrome to debug the server.

Unit testing FAQ

There are a few typees of tests during presubmit:

  • formatting, refer to Code Style Section
  • linting, you can also run locally with npm run lint
  • client UI unit tests, you can run locally with npm test
  • UI node server unit tests, you can run locally with cd server && npm test

There is a special type of unit test called snapshot tests. When snapshot tests are failing, you can update them automatically with npm test -u and run all tests. Then commit the snapshot changes.

Production Build

You can do npm run build to build the frontend code for production, which creates a ./build directory with the minified bundle. You can test this bundle using server/server.js. Note you need to have an API server running, which you can then feed its address (host + port) as environment variables into server.js. See the usage instructions in that file for more.

Container Build

You can also do npm run docker if you have docker installed to build an image containing the production bundle and the server pieces. In order to run this image, you'll need to port forward 3000, and pass the environment variables ML_PIPELINE_SERVICE_HOST and ML_PIPELINE_SERVICE_PORT with the details of the API server.

Code Style

We use prettier for code formatting, our prettier config is here.

To understand more what prettier is: What is Prettier.

IDE Integration

  • For vscode, install the plugin "Prettier - Code formatter" and it will pick this project's config automatically. Recommend setting the following in settings.json for vscode to autoformat on save. 
    "[typescript]": {
  "editor.formatOnSave": true,
  "files.trimTrailingWhitespace": false,
},
"[typescriptreact]": {
  "editor.formatOnSave": true,
  "files.trimTrailingWhitespace": false,
},
    Also, vscode builtin trailing whitespace conflicts with jest inline snapshot, so recommend disabling it.
  • For others, refer to https://prettier.io/docs/en/editors.html.

Format Code Manually

Run npm run format.

Escape hatch

If there's some code that you don't want being formatted by prettier, follow guide here. (Most likely you don't need this.)

Api client code generation

If you made any changes to protos (see backend/README), you'll need to regenerate the Typescript client library from swagger. We use [email protected], which you can get here. Make sure to add the jar file to $PATH with the name swagger-codegen-cli.jar, then run npm run apis for v1 api or npm run apis:v2beta1 for v2 api.

// add jar file to $PATH
JAR_PATH=<folder-path-to-jar-file>
export PATH="$JAR_PATH:$PATH"

After code generation, you should run npm run format to format the output and avoid creating a large PR.

MLMD components

  • src/mlmd - components for visualizing data from an ml-metadata store. For more information see the google/ml-metadata repository.

This module previously lived in kubeflow/frontend repository. It contains tsx files for visualizing MLMD components.

MLMD protos lives in pipelines/third_party/ml-metadata/ml_metadata/, and the generated JS files live in pipelines/frontend/src/third_party/mlmd.

Building generated metadata Protocol Buffers

  • build:protos - for compiling Protocol Buffer definitions

This project contains a mix of natively defined classes and classes generated by the Protocol Buffer Compiler from definitions in the pipelines/third_party/ml-metadata/ml_metadata/ directory. Copies of the generated classes are included in the pipelines/frontend/src/third_party/mlmd directory to allow the build process to succeed without a dependency on the Protocol Buffer compiler, protoc, being in the system PATH.

If a file in pipelines/third_party/ml-metadata/ml_metadata/proto is modified or you need to manually re-generate the protos, you'll need to:

  • Add protoc (download) to your system PATH

    # Example:
apt install -y protobuf-compiler=3.15.8

  • Add protoc-gen-grpc-web (download) to your system PATH

    # Example:
curl -LO https://github.com/grpc/grpc-web/releases/download/1.4.2/protoc-gen-grpc-web-1.4.2-linux-x86_64
mv protoc-gen-grpc-web-1.4.2-linux-x86_64 /usr/local/bin/protoc-gen-grpc-web
chmod +x /usr/local/bin/protoc-gen-grpc-web

  • Replace metadata_store.proto and metadata_store_service.proto proto files with target mlmd version by running

    npm run build:replace -- {mlmd_versions}
// example:
// npm run build:replace -- 1.0.0

  • Generate new protos by running

    npm run build:protos

The script run by npm run build:replace can be found at scripts/replace_protos.js. The script run by npm run build:protos can be found at scripts/gen_grpc_web_protos.js.

The current TypeScript proto library was generated with protoc-gen-grpc-web version 1.2.1 with protoc version 3.17.3.

The Protocol Buffers in pipelines/third_party/ml-metadata/ml_metadata/proto are taken from the target version(v1.0.0 by default) of the ml_metadata proto package from google/ml-metadata.

Pipeline Spec (IR) API

For KFP v2, we use pipeline spec or IR(Intermediate Representation) to represent a Pipeline defintion. It is saved as json payload when transmitted. You can find the API in api/v2alpha1/pipeline_spec.proto. To take the latest of this file and compile it to Typescript classes, follow the below step:

npm run build:pipeline-spec

See explaination it does below:

Convert buffer to runtime object using protoc

Prerequisite: Add protoc (download) to your system PATH

Compile pipeline_spec.proto to Typed classes in TypeScript, so it can convert a payload stream to a PipelineSpec object during runtime.

You can check out the result like pipeline_spec_pb.js, pipeline_spec_pb.d.ts in frontend/src/generated/pipeline_spec.

The plugin tool for convertion we currently use is ts-proto. We previously use protobuf.js but it doesn't natively support Protobuf.Value processing.

You can checkout the generated TypeScript interfaces in frontend/src/generated/pipeline_spec/pipeline_spec.ts

Platform Spec API

For KFP v2, we use platform spec to represent a platform definition.

Kubernetes

The details of Kubernetes platform is in kubernetes_platform/proto/kubernetes_executor_config.proto. To take the latest of this file and compile it to Typescript classes, follow the below step:

npm run build:platform-spec:kubernetes-platform

Large features development

To accommodate KFP v2 development, we create a frontend feature flag capability which hides features under development behind a flag. Only when developer explicitly enables these flags, they can see those features. To control the visiblity of these features, check out a webpage similar to pattern http://localhost:3000/#/frontend_features.

To manage feature flags default values, visit frontend/src/feature.ts for const features. To apply the default feature flags locally in your browser, run localStorage.setItem('flags', "") in browser console.

Storybook

For component driven UI development, KFP UI integrates with Storybook to develop v2 features. To run Storybook locally, run npm run storybook and visit localhost:6006 in browser.