Update/overhaul of these 2 examples, including switching to use GKE (#21

)

* use GKE (rather than kubernetes)
* big readme updates for both examples
* update gcr image versions
* change redis-master to be a deployment, api updates
* updated top-level readme

README.md

@@ -1,13 +1,16 @@
 
 Copyright (C) 2014 Google Inc.
 
-# Example apps: Real-time data analysis using Kubernetes, Redis or PubSub, and BigQuery
+# Example apps: Real-time data analysis using Google Kubernetes Engine, Redis or PubSub, and BigQuery
 
-This repository contains two related example [Kubernetes](https://github.com/GoogleCloudPlatform/kubernetes) apps that show how to build a 'pipeline' to stream data into BigQuery.
+This repository contains two related example [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/)
+(GKE) apps that show how to build a 'pipeline' to stream data into BigQuery.
 
-The first app, in the `redis` subdirectory, uses [Redis](http://redis.io/).
-Documentation for this example can be found on the Google Cloud Platform site:
-https://cloud.google.com/solutions/real-time-analysis/kubernetes-redis-bigquery
 
-The second app, in the `pubsub` directory, uses [Google Cloud PubSub](https://cloud.google.com/pubsub/docs) instead of Redis. Documentation for this example can be found on the Google Cloud Platform site:
-https://cloud.google.com/solutions/real-time/kubernetes-pubsub-bigquery
+The app in the `pubsub` directory uses [Google Cloud PubSub](https://cloud.google.com/pubsub/docs).
+<!-- Documentation for this example can be found on the Google Cloud Platform site:
+https://cloud.google.com/solutions/real-time/kubernetes-pubsub-bigquery -->
+
+The app in the `redis` subdirectory uses [Redis](http://redis.io/).
+<!-- Documentation for this example can be found on the Google Cloud Platform site:
+https://cloud.google.com/solutions/real-time-analysis/kubernetes-redis-bigquery -->

pubsub/bigquery-controller.yaml

@@ -1,4 +1,4 @@
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1beta1
 kind: Deployment
 metadata:
   name: bigquery-controller
@@ -13,7 +13,7 @@ spec:
     spec:
       containers:
       - name: bigquery
-        image: gcr.io/google-samples/pubsub-bq-pipe:v3
+        image: gcr.io/google-samples/pubsub-bq-pipe:v4
         env:
         - name: PROCESSINGSCRIPT
           value: pubsub-to-bigquery

pubsub/pubsub-pipe-image/utils.py

@@ -80,7 +80,9 @@ def cleanup(data):
             # TODO: update BigQuery schema
             elif (k == 'video_info' or k == 'scopes' or k == 'withheld_in_countries'
                   or k == 'is_quote_status' or 'source_user_id' in k
-                  or 'quoted_status' in k or 'display_text_range' in k or 'extended_tweet' in k):
+                  or k == ''
+                  or 'quoted_status' in k or 'display_text_range' in k or 'extended_tweet' in k
+                  or 'media' in k):
                 pass
             elif v is False:
                 newdict[k] = v

pubsub/twitter-stream.yaml

@@ -1,4 +1,4 @@
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1beta1
 kind: Deployment
 metadata:
   name: twitter-stream
@@ -13,7 +13,7 @@ spec:
     spec:
       containers:
       - name: twitter-to-pubsub
-        image: gcr.io/google-samples/pubsub-bq-pipe:v3
+        image: gcr.io/google-samples/pubsub-bq-pipe:v4
         env:
         - name: PROCESSINGSCRIPT
           value: twitter-to-pubsub

redis/bigquery-controller.yaml

@@ -1,4 +1,4 @@
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1beta1
 kind: Deployment
 metadata:
   name: bigquery-controller
@@ -13,7 +13,7 @@ spec:
     spec:
       containers:
       - name: bigquery
-        image: gcr.io/google-samples/redis-bq-pipe:v3
+        image: gcr.io/google-samples/redis-bq-pipe:v4
         env:
         - name: PROCESSINGSCRIPT
           value: redis-to-bigquery

redis/redis-master.yaml

@@ -1,5 +1,5 @@
-apiVersion: v1
-kind: ReplicationController
+apiVersion: apps/v1beta1
+kind: Deployment
 metadata:
   name: redis-master
   labels:

redis/redis-pipe-image/utils.py

@@ -66,7 +66,9 @@ def cleanup(data):
             # TODO: update BigQuery schema
             elif (k == 'video_info' or k == 'scopes' or k == 'withheld_in_countries'
                   or k == 'is_quote_status' or 'source_user_id' in k
-                  or 'quoted_status' in k or 'display_text_range' in k or 'extended_tweet' in k):
+                  or k == ''
+                  or 'quoted_status' in k or 'display_text_range' in k or 'extended_tweet' in k
+                  or 'media' in k):
                 pass
             elif v is False:
                 newdict[k] = v

redis/twitter-stream.yaml

@@ -1,4 +1,4 @@
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1beta1
 kind: Deployment
 metadata:
   name: twitter-stream
@@ -13,7 +13,7 @@ spec:
     spec:
       containers:
       - name: twitter-to-redis
-        image: gcr.io/google-samples/redis-bq-pipe:v3
+        image: gcr.io/google-samples/redis-bq-pipe:v4
         env:
         - name: PROCESSINGSCRIPT
           value: twitter-to-redis