This repository has been archived by the owner on Jan 3, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 88
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
reorganization, refactoring, addition of pubsub example.
(The README for the pubsub example still needs further details).
- Loading branch information
Showing
23 changed files
with
986 additions
and
259 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
# How to become a contributor and submit your own code | ||
|
||
## Contributor License Agreements | ||
|
||
We'd love to accept your sample apps and patches! Before we can take them, we have to jump a couple of legal hurdles. | ||
|
||
Please fill out either the individual or corporate Contributor License Agreement (CLA). | ||
|
||
* If you are an individual writing original source code and you're sure you own the intellectual property, then you'll need to sign an [individual CLA](http://code.google.com/legal/individual-cla-v1.0.html). | ||
* If you work for a company that wants to allow you to contribute your work, then you'll need to sign a [corporate CLA](http://code.google.com/legal/corporate-cla-v1.0.html). | ||
|
||
Follow either of the two links above to access the appropriate CLA and instructions for how to sign and return it. Once we receive it, we'll be able to accept your pull requests. | ||
|
||
## Contributing A Patch | ||
|
||
1. Submit an issue describing your proposed change to the repo in question. | ||
1. The repo owner will respond to your issue promptly. | ||
1. If your proposed change is accepted, and you haven't already done so, sign a Contributor License Agreement (see details above). | ||
1. Fork the desired repo, develop and test your code changes. | ||
1. Ensure that your code adheres to the existing style in the sample to which you are contributing. Refer to the [Google Cloud Platform Samples Style Guide](https://github.com/GoogleCloudPlatform/Template/wiki/style.html) for the recommended coding standards for this organization. | ||
1. Ensure that your code has an appropriate set of unit tests which all pass. | ||
1. Submit a pull request. | ||
|
||
## Contributing A New Sample App | ||
|
||
1. Submit an issue to the GoogleCloudPlatform/Template repo describing your proposed sample app. | ||
1. The Template repo owner will respond to your enhancement issue promptly. Instructional value is the top priority when evaluating new app proposals for this collection of repos. | ||
1. If your proposal is accepted, and you haven't already done so, sign a Contributor License Agreement (see details above). | ||
1. Create your own repo for your app following this naming convention: | ||
* {product}-{app-name}-{language} | ||
* products: appengine, compute, storage, bigquery, prediction, cloudsql | ||
* example: appengine-guestbook-python | ||
* For multi-product apps, concatenate the primary products, like this: compute-appengine-demo-suite-python. | ||
* For multi-language apps, concatenate the primary languages like this: appengine-sockets-python-java-go. | ||
1. Clone the README.md, CONTRIB.md and LICENSE files from the GoogleCloudPlatform/Template repo. | ||
1. Ensure that your code adheres to the existing style in the sample to which you are contributing. Refer to the [Google Cloud Platform Samples Style Guide](https://github.com/GoogleCloudPlatform/Template/wiki/style.html) for the recommended coding standards for this organization. | ||
1. Ensure that your code has an appropriate set of unit tests which all pass. | ||
1. Submit a request to fork your repo in GoogleCloudPlatform organizationt via your proposal issue. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,9 +1,12 @@ | ||
|
||
Copyright (C) 2014 Google Inc. | ||
|
||
# Example app: Real-time data analysis using Kubernetes, Redis, and BigQuery | ||
# Example apps: Real-time data analysis using Kubernetes, Redis or PubSub, and BigQuery | ||
|
||
This is an example Kubernetes app that shows how to build a 'pipeline' to stream data into BigQuery. It uses Redis. | ||
This repository contains two related example [Kubernetes](https://github.com/GoogleCloudPlatform/kubernetes) apps that show how to build a 'pipeline' to stream data into BigQuery. | ||
|
||
The first app, in the `redis` subdirectory, uses [Redis](http://redis.io/). | ||
Documentation for this example can be found on the Google Cloud Platform site: | ||
https://cloud.google.com/solutions/real-time-analysis/kubernetes-redis-bigquery | ||
|
||
The second app, in the `pubsub` directory, uses [Google Cloud PubSub](https://cloud.google.com/pubsub/docs) instead of Redis. |
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
|
||
Copyright (C) 2014 Google Inc. | ||
|
||
# Example app: Real-time data analysis using Kubernetes, PubSub, and BigQuery | ||
|
||
This directory contains an example [Kubernetes](https://github.com/GoogleCloudPlatform/kubernetes) app that shows how to build a 'pipeline' to stream data into BigQuery. | ||
The pipeline uses uses [Google Cloud PubSub](https://cloud.google.com/pubsub/docs). | ||
|
||
See also a related app, in the `redis` directory, which uses [Redis](http://redis.io/) instead of PubSub. | ||
|
||
**More detailed documentation for this PubSub example is coming soon**. | ||
However, the structure of the example is similar to that of the Redis example, described [here](https://cloud.google.com/solutions/real-time-analysis/kubernetes-redis-bigquery), except that you will be configuring the app to use PubSub instead of Redis. | ||
|
||
The primary setup differences are listed below. | ||
|
||
## Configuring the example | ||
|
||
### Start up your Kubernetes cluster with the 'Cloud Platform' scope. | ||
|
||
As described in the Redis-based [tutorial](https://cloud.google.com/solutions | ||
/real-time-analysis/kubernetes-redis-bigquery), edit the MINION_SCOPES in | ||
`cluster/gce/config-default.sh` before starting up your cluster. For the | ||
PubSub example, you will need to add the `cloud-platform` scope as well as | ||
`bigquery`. | ||
|
||
``` | ||
MINION_SCOPES=("storage-ro" "compute-rw" "bigquery" "https://www.googleapis.com/auth/cloud-platform") | ||
``` | ||
|
||
(You don't need to make the startup script changes to `cluster/gce/util.sh` as | ||
described in the tutorial if you do not plan to run the Redis variant of this | ||
example). | ||
|
||
|
||
### Set up a PubSub topic in your project | ||
|
||
Before running the example, in addition to the Twitter and BigQuery | ||
configuration described in the tutorial, you will need to create a PubSub | ||
topic in your Cloud project. [More details TBA]. | ||
|
||
### Configure the Kubernetes .yaml files | ||
|
||
Configure the `bigquery-controller.yaml` and `twitter-stream.yaml` files with | ||
your information. You will need to specify the name of your PubSub topic as | ||
well as adding the other info described in the [Redis-based tutorial](https://cloud.google.com/solutions/real-time-analysis/kubernetes-redis-bigquery). Note that for this | ||
example, you don't need the redis config files mentioned in the tutorial. | ||
|
||
## Building your Docker images, and using the Google Container Registry | ||
|
||
As part of installation process, you will build a Docker image, containing the | ||
Python scripts, that is used by your Kubernetes app. You will then specify the | ||
image in the .yaml config files. | ||
|
||
The basic Docker image build process is described in the [tutorial](https://cloud.google.com/solutions/real-time-analysis/kubernetes-redis-bigquery); for this PubSub | ||
example, use the `Dockerfile` and assets in the `pubsub/pubsub-pipe-image` directory for your image. | ||
|
||
You can now use the new [Google Container | ||
Registry](https://cloud.google.com/tools/container-registry/) (GCR), which is in Beta, for | ||
your Docker images if you like. With GCR, your images are stored in your | ||
own project, using GCS. | ||
See the GCR [docs](https://cloud.google.com/tools/container-registry/) for | ||
details. | ||
|
Oops, something went wrong.