-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
pimg
committed
Apr 29, 2019
1 parent
6a89e4c
commit dd8c984
Showing
16 changed files
with
711 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,96 @@ | ||
# Setting up the development environment | ||
|
||
As was clear from the introduction, APIcast policies are created in the Lua programming language. So we need to setup an environment to do some Lua programming. Also, an actual APIcast server would be very nice to perform some local tests. | ||
|
||
Luckily the guys from 3scale made it very easy to setup a development environment for APIcast using Docker and Docker Compose. | ||
|
||
### prerequisites | ||
This means both Docker and Docker compose must be installed. | ||
|
||
The version of Docker I currently use is: | ||
|
||
Docker version 18.09.2, build 6247962 | ||
|
||
Instructions for installing Docker can be found on the Docker [website](https://docs.docker.com/install/). | ||
|
||
With Docker compose version: | ||
|
||
docker-compose version 1.23.1, build b02f1306 | ||
|
||
Instructions for installing Docker-compose can also be found on the Docker [website](https://docs.docker.com/compose/install/). | ||
|
||
### Setting up the development image | ||
Now that we have both Docker and Docker-compose installed we an setup the APIcast development image. | ||
|
||
Firstly the APIcast git repostitory must be cloned so we can start the development of our policy. Since we are going to base our policy on the latest 3scale release we are switching to the stable branch of APIcast. | ||
|
||
```shell | ||
git clone https://github.com/3scale/apicast.git | ||
``` | ||
|
||
when done switch to a stable branch, I am using 3.3 | ||
```shell | ||
cd apicast/ | ||
git checkout 3.3-stable | ||
``` | ||
|
||
To start the APIcast containers using Docker-compose we can use the Make file provided by 3scale. In the APIcast directory simply execute the command: | ||
```shell | ||
make development | ||
``` | ||
|
||
![make-development](img/make-development.jpg) | ||
|
||
The Docker container starts in the foreground with a bash session. The first thing we need to do inside the container is installing all the dependencies. | ||
|
||
This can also be done using a Make command, which again must be issued **inside** the container. | ||
```shell | ||
make dependencies | ||
``` | ||
It will now download and install a plethora of dependencies inside the container. | ||
|
||
The output will be very long, but if everything went well you should be greeted with an output that looks something like this: | ||
|
||
![make-dependencies](img/make-dependencies.png) | ||
|
||
Now as a final verification we can run some APIcast unit tests to see if we are up and running and ready to start the development of our policy. | ||
|
||
To run the Lua unit tests run the following command **inside** the container: | ||
|
||
```shell | ||
make busted | ||
``` | ||
![make-busted](img/make-busted.png) | ||
|
||
Now that we can successfully run unit tests we can start our policy development! | ||
|
||
The project’s source code will be available in the container and sync’ed with your local apicast directory, so you can edit files in your preferred environment and still be able to run whatever you need inside the Docker container. | ||
|
||
The development container for APIcast uses a Docker volume mount to mount the local apicast directory inside the container. This means all files changed locally in the repository are synced with the container and used in the tests and runtime of the development container. | ||
|
||
![3scale-dev-container-mount](img/3scale-dev-container-mount.png) | ||
|
||
It also means you can use your favorite IDE or editor develop your 3scale policy. | ||
|
||
### Optional: setup an IDE for policy development | ||
The use of an IDE or text editor and more specifically which one is very personal so there is definitely no one size fits all here. But for those looking for a dedicated Lua IDE [ZeroBraneStudio](https://studio.zerobrane.com/) is a good choice. | ||
|
||
Since I come from a Java background I am very used to working with [IntelliJ IDEA](https://www.jetbrains.com/idea/), and luckily there are some plugins available that make Lua development a little bit nicer. | ||
|
||
These are the plugins I installed for developing Lua code and 3scale policies in particular: | ||
|
||
![idea-lua-plugin](img/IDEA-Lua-plugin.png) | ||
|
||
And for Openresty/Nginx there is also a plugin: | ||
|
||
![idea-openresty-plugin](img/IDEA-Openresty-plugin.png) | ||
|
||
As a final step, but this is more relevant if you are also planning on developing some Openresty based applications locally (outside the APIcast development container), you can install Openresty, based on the instructions on their [website](http://openresty.org/en/installation.html). | ||
|
||
What I did was I linked the Lua runtime engine of Openresty, which is LuaJIT, to the SDK of my IntelliJ IDEA so that I am developing code against the LuaJIT engine of Openresty. | ||
|
||
![idea-luajit](img/IDEA-LuaJit-SDK.png) | ||
|
||
As I already mentioned these steps are not required for developing policies in APIcast, and you definitely do not need to use IntelliJ IDEA. But having a good IDE or Text editor, whatever your choice, can make your development life a little bit easier. | ||
|
||
Now we are ready to create a 3scale APIcast policy. |
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,149 @@ | ||
# Run the policy locally | ||
|
||
Now that the development and unit testing of the policy is completed it is time to actually start APIcast with the newly created policy. This way we can verify the policy actually works inside the Nginx process. We are again going to use the APIcast development Docker image to run APIcast with our newly created policy. | ||
|
||
## Create APIcast configuration | ||
|
||
APIcast can be provisioned in two ways, either automatically from the 3scale API Manager or via a json configuration file. Since the custom policy is not available in the 3scale API Manager we need to configure APIcast with the json configuration file. In this configuration file we are also going to leverage the build-in echo service of APIcast so we can test the policy and receive an upstream response without running a full 3scale API Manager configuration. | ||
|
||
The part of the json configuration file detailing the configuration of our policy need to adhere to the json schema we defined earlier. | ||
|
||
The location of the config file is somewhat arbitrary as long as the APIcast process can access the file. In this example we are going to create the following file **‘hello_world_config.json’** in the **apicast/examples/configuration** directory. | ||
|
||
The contents of the configuration file is: | ||
|
||
```json | ||
{ | ||
"services": [ | ||
{ | ||
"proxy": { | ||
"policy_chain": [ | ||
{ | ||
"name": "hello_world", | ||
"version": "builtin", | ||
"configuration": { | ||
"overwrite": true, | ||
"secret": "mysecret" | ||
} | ||
}, | ||
{ | ||
"name": "apicast.policy.upstream", | ||
"configuration": { | ||
"rules": [ | ||
{ | ||
"regex": "/", | ||
"url": "http://echo:8081" | ||
} | ||
] | ||
} | ||
} | ||
] | ||
} | ||
} | ||
] | ||
} | ||
``` | ||
|
||
Inside the policy chain we first configure our hello_world policy with the overwrite and secret properties. Second in the policy chain is the upstream policy which acts as an echo mock service so we can actually receive a response. | ||
|
||
### Starting the APIcast server | ||
Now in order to start the APIcast server with the hello_world_configuration.json file inside the development container issue the following command: | ||
|
||
```shell | ||
bash-4.2$ bin/apicast --log-level=debug --dev -c examples/configuration/hello_world_config.json | ||
``` | ||
|
||
the bin/apicast executable starts the apicast server, we set the log-level to debug which is going to result in an incredible amount of debug logging. | ||
If the amount of debug logging is a bit too much for you the log-level can also be set to **notice** this results in a lot fewer log lines, but still the custom log entries in the policy are logged. | ||
|
||
### Executing test requests | ||
Now that we have our APIcast server up and running let’s test if the hello_world policy actually works. | ||
|
||
To test this we need to issue an HTTP request to the APIcast server. However the development Docker container does not expose any ports. We could alter either the makefile or the Docker compose file to expose the ports. But in this example we are simply going to create another bash session the the development container and issue a curl request from inside the container. | ||
|
||
To create a second bash session open a new terminal window and find the Docker container id of the APIcast development image using the following command: | ||
|
||
```shell | ||
$ docker ps | ||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES | ||
5a72c49671c5 quay.io/3scale/s2i-openresty-centos7:master "container-entrypoin…" 2 hours ago Up 2 hours 8080/tcp apicast_build_0_development_1_802efce654d5 | ||
366c62d0bccf redis "docker-entrypoint.s…" 2 hours ago Up 2 hours 6379/tcp apicast_build_0_redis_1_469bce65a85a | ||
``` | ||
|
||
The make development command we used to start the APIcast development container actually starts two containers one with the APIcast development environment, the second with a Redis cache. The container we are interested is the container with the image: **quay.io/3scale/s2i-openresty-centos7:master** | ||
|
||
In the above example it has the id of **5a72c49671c5** off course yours will be different. Now that we know the ID of the container let’s create a new bash session using the following command: | ||
|
||
```shell | ||
$ docker exec -it 5a72c49671c5 /bin/bash | ||
``` | ||
|
||
Now we have another interactive bash shell in the APIcast development container and we can issue the HTTP request to test the policy from here. | ||
|
||
In the container issue the following HTTP request: | ||
|
||
```shell | ||
$ curl localhost:8080 | ||
<html> | ||
<head><title>403 Forbidden</title></head> | ||
<body bgcolor="white"> | ||
<center><h1>403 Forbidden</h1></center> | ||
<hr><center>openresty/1.13.6.2</center> | ||
</body> | ||
</html> | ||
``` | ||
|
||
The response will be a 403 Forbidden. Let’s look at the logs to see what has happened. | ||
|
||
```shell | ||
$ bin/apicast --log-level=notice --dev -c examples/configuration/hello_world_config.json | ||
loading production environment configuration: /home/centos/gateway/config/production.lua | ||
loading development environment configuration: /home/centos/gateway/config/development.lua | ||
2019/04/29 09:32:33 [notice] 257#257: [lua] environment.lua:194: add(): loading environment configuration: /home/centos/gateway/config/production.lua | ||
2019/04/29 09:32:33 [notice] 257#257: [lua] environment.lua:194: add(): loading environment configuration: /home/centos/gateway/config/development.lua | ||
2019/04/29 09:32:33 [notice] 257#257: using the "epoll" event method | ||
2019/04/29 09:32:33 [notice] 257#257: openresty/1.13.6.2 | ||
2019/04/29 09:32:33 [notice] 257#257: built by gcc 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) | ||
2019/04/29 09:32:33 [notice] 257#257: OS: Linux 4.4.0-146-generic | ||
2019/04/29 09:32:33 [notice] 257#257: getrlimit(RLIMIT_NOFILE): 1048576:1048576 | ||
2019/04/29 09:43:17 [notice] 257#257: *6 [lua] hello_world.lua:53: request is not authorized, secrets do not match, client: 127.0.0.1, server: _, request: "GET / HTTP/1.1", host: "localhost:8080" | ||
[29/Apr/2019:09:43:17 +0000] localhost:8080 127.0.0.1:40946 "GET / HTTP/1.1" 403 175 (0.000) 0 | ||
``` | ||
|
||
The APIcast is running with the notice log-level, when started with the debug log-level considerable more log events will be present. But the one we are interested in is the second from the bottom. | ||
|
||
Which states: **‘request is not authorized, secrets do not match’** which is put in the log by the following line in the policy code: | ||
|
||
```lua | ||
if secret_header ~= self.secret then | ||
ngx.log(ngx.NOTICE, "request is not authorized, secrets do not match") | ||
ngx.status = 403 | ||
return ngx.exit(ngx.status) | ||
``` | ||
|
||
So we now our policy is executing. Let’s provide the secret header in our request in order to pass the validation. Issue the following HTTP request: | ||
|
||
```shell | ||
$ curl localhost:8080 -H 'secret: mysecret' | ||
GET / HTTP/1.1 | ||
X-Real-IP: 127.0.0.1 | ||
Host: echo | ||
User-Agent: curl/7.29.0 | ||
Accept: */* | ||
secret: mysecret | ||
``` | ||
|
||
Now we received a valid 200 response from the echo server. But the actual rewrite of query parameters to header is not tested, since the request did not contain any query parameters. So issue a new request with a query parameter to see the transformation at work. Issue the following request: | ||
|
||
```shell | ||
$ curl localhost:8080?myparam=myvalue -H 'secret: mysecret' | ||
GET /?myparam=myvalue HTTP/1.1 | ||
X-Real-IP: 127.0.0.1 | ||
Host: echo | ||
User-Agent: curl/7.29.0 | ||
Accept: */* | ||
secret: mysecret | ||
myparam: myvalue | ||
``` | ||
|
||
Now we see in the response the header myparam:myheader |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,93 @@ | ||
# Generate a policy scaffold | ||
In first part of th workshop about 3scale policy development we looked into the setup of a development environment. Now we have a functioning development environment we can start the actual development of the 3scale policy. In this part we will take a look and use the scaffolding utility provided by APIcast to generate a policy scaffold. | ||
|
||
The first thing we are going to do is create a new git branch of the APIcast source we have cloned in the previous part. This is an optional step, but developing a new feature or changing code in general in a new branch is a good habit to get into. So create a new branch and start up our development container. | ||
|
||
```shell | ||
$ git checkout -b policy-development-blog | ||
Switched to a new branch 'policy-development-blog' | ||
$ make development | ||
``` | ||
|
||
To generate the scaffold of our policy we can use the apicast utility located in the bin/ directory of our development container. | ||
So in the development container issue the following command: | ||
|
||
```shell | ||
$ bin/apicast generate policy hello_world | ||
``` | ||
|
||
where hello_world is the name of the policy. | ||
|
||
```shell | ||
bash-4.2$ bin/apicast generate policy hello_world | ||
source: /home/centos/examples/scaffold/policy | ||
destination: /home/centos | ||
|
||
exists: t | ||
created: t/apicast-policy-hello_world.t | ||
exists: gateway | ||
exists: gateway/src | ||
exists: gateway/src/apicast | ||
exists: gateway/src/apicast/policy | ||
created: gateway/src/apicast/policy/hello_world | ||
created: gateway/src/apicast/policy/hello_world/hello_world.lua | ||
created: gateway/src/apicast/policy/hello_world/init.lua | ||
created: gateway/src/apicast/policy/hello_world/apicast-policy.json | ||
exists: spec | ||
exists: spec/policy | ||
created: spec/policy/hello_world | ||
created: spec/policy/hello_world/hello_world_spec.lua | ||
bash-4.2$ | ||
``` | ||
|
||
As you can see from the output of the generate policy command a few files have been created. These artifacts related to our policy are located in three different directories: | ||
|
||
* t/ – this directory contains all Nginx integration tests | ||
* src/gateway/apicast/policy – this directory contains the source code and configuration schemas of all policies. Our policy resides in the subdirectory of hello_world | ||
* spec/policy – this directory contains the unit tests of all policies. The unit tests for our policy resides in the subdirectory of hello_world | ||
|
||
So the policy scaffolding utility not only generates a scaffold for our policy, but also the files for a configuration schema, unit tests and integration tests. Let’s have a look at these files. | ||
|
||
The source code of our policy residing in the directory src/gateway/apicast/policy/hello_world contains three files. | ||
|
||
* init.lua - all policies contain this init.lua file. It contains 1 line importing (require in Lua) our policy. It should not be modified. | ||
* aplicast-policy.json - The APIcast gateway is configured using a json document. Policies requiring configuration also use this json document. The apicast-policy.json file is a json schema file were configuration properties for the policy can be defined. We will look into configuration properties and this file in more detail in our next part of the workshop. | ||
```json | ||
{ | ||
"$schema": "http://apicast.io/policy-v1/schema#manifest#", | ||
"name": "hello_world", | ||
"summary": "TODO: write policy summary", | ||
"description": [ | ||
"TODO: Write policy description" | ||
], | ||
"version": "builtin", | ||
"configuration": { | ||
"type": "object", | ||
"properties": { } | ||
} | ||
} | ||
``` | ||
* hello_world.lua - This is the actual source code of our policy, which at the moment does not contain much. | ||
```lua | ||
-- This is a hello_world description. | ||
local policy = require('apicast.policy') | ||
local _M = policy.new('hello_world') | ||
local new = _M.new | ||
--- Initialize a hello_world | ||
-- @tparam[opt] table config Policy configuration. | ||
function _M.new(config) | ||
local self = new(config) | ||
return self | ||
end | ||
return _M | ||
``` | ||
|
||
The first two lines import the APIcast policy module an instantiate a new policy with hello_world as an argument. This returns a module itself which is implemented using a Lua table. Lua is not an Object Oriented language from itself but tables (and especially metatables) can mimic objects. The third line stores a reference to a function new which is defined below. The new function takes a config variable as argument, but as of now nothing is done with is. The new method simply returns itself. Finally the module representing our policy is returned. This is done so other components importing this policy module retrieve the table and can invoke all functions and variables stored in the policy. | ||
We won’t cover all the files in details here since we are going to touch these in upcoming series when we flesh out our policy with functionality. | ||
But as a final verification to see if we have something working let’s run the unit tests again. | ||
|
||
![busted-after-scaffold](img/make-busted-after-scaffold.png) | ||
|
||
The keen observer can see the number of successes in the unit test outcome has increased from 749 to 751 after we generated the scaffold for our policy. | ||
|
||
In the next part we actually create the implementation of the policy. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,32 @@ | ||
# 3scale policy development | ||
# 3scale APIcast policy development workshop | ||
This repository contains the code and configuration of a 3scale APIcast policy used in the workshop described in this README. | ||
It's purposes are to provide a first introduction to the world of 3scale policy development. | ||
|
||
In this workshop we are going to dive into the development, testing and deployment of a custom 3scale APIcast policy. In this initial part we are going to setup a development environment so we can actually start the development of our policy. | ||
|
||
But before we begin, let’s first take a look what a 3scale APIcast policy is. We are not going into too much detail here, since better and more detailed descriptions about 3scale APIcast policies already exist. | ||
|
||
For those unfamiliar, [3scale](www.3scale.net) is a full API Management solution of Red Hat. It exists of an API Manager used for account management, analytics and overall configuration. A developer portal used for outside developers for gaining access to API’s and viewing the documentation. And the API gateway named APIcast. The APIcast gateway is based on [Nginx](https://www.nginx.com/) and more specifically [Openresty](http://openresty.org/en/), which is a distribution of Nginx compiled with various modules, most notable the [lua-nginx-module](https://github.com/openresty/lua-nginx-module). | ||
|
||
The lua-nginx-module provides the ability to enhance a Nginx server by executing scripts using the [Lua programming language](https://www.lua.org/). This is done by providing a Lua hook for each of the Nginx phases. Nginx works using an event loop and a state model where every request (as well as the starting of the server and its worker processes) goes through various phases. Each phase can execute a specific Lua function. | ||
|
||
An overview of the various phases and corresponding Lua hooks was kindly in the README of the lua-nginx-module: https://github.com/openresty/lua-nginx-module#directives | ||
|
||
![Nginx phases](img/nginx-phases.png) | ||
|
||
Since the APIcast gateway uses Openresty 3scale provided a way to leverage these Lua hooks in the Nginx server using something called policies. As described in the APIcast README: | ||
|
||
**“The behaviour of APIcast is customizable via policies. A policy basically tells APIcast what it should do in each of the nginx phases.”** | ||
|
||
A detailed explanation of policies can be found in the same README: https://github.com/3scale/apicast/blob/master/doc/policies.md | ||
|
||
|
||
The code in this repo follows the APIcast directory structure. | ||
In order to use this code it has to be integrated in the APIcast code. | ||
Therefore the code and configuration in this repository act for reference purposes only. | ||
|
||
The workshop was 4 distinct sections: | ||
1. [setup the development environment](DEV_ENV_SETUP.md) | ||
2. [generate a policy scaffold](POLICY_SCAFFOLD.md) | ||
3. [create and test the policy](POLICY_IMPLEMENTATION.md) | ||
4. [run the policy locally](POLICY_RUN_LOCALLY.md) |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.