Indexer service load balancing with nginx

Currently, indexer-service can only be scaled effectively with multiple indexer-service instances. You then need to loadbalance those services, and nginx is ideal for this. Below is a guide on how to use nginx to achieve this on Ubuntu 20.04

nginx from scratch

#install nginx
sudo apt-get install nginx

#main config folder
cd /etc/nginx/

# Ubuntu and Debian follow a rule for storing virtual host files in 
# /etc/nginx/sites-available/, which are enabled through symbolic links to 
# /etc/nginx/sites-enabled/. You can use the command below to enable any new 
# virtual host files.

sudo ln -s /etc/nginx/sites-available/vhost /etc/nginx/sites-enabled/vhost

Configuring nginx as a load balancer for Graph indexer-services

When nginx is installed and tested, start to configure it for load balancing. In essence, all you need to do is set up nginx with instructions for which type of connections to listen to and where to redirect them. Create a new configuration file using whichever text editor you prefer. For example with nano:

sudo nano /etc/nginx/conf.d/graph-load-balancer.conf

In the graph-load-balancer.conf you’ll need to define the following two segments, upstream and server, see the example below.

# Define which servers to include in the load balancing scheme. 
# It's best to use the servers' private IPs for better performance and security.

   upstream graphservice {
      server graph-test-service-0:7600;
      server graph-test-service-1:7600;
   }

   # This server accepts all traffic to port 7600 and passes it to the upstream. 
   # Notice that the upstream name and the proxy_pass need to match.

   server {
      listen 7600;

      location / {
          proxy_pass http://graphservice;
      }
   }

Next, disable the default server configuration you earlier tested was working after the installation. Again depending on your OS, this part differs slightly.

On Debian and Ubuntu systems you’ll need to remove the default symbolic link from the sites-enabled folder:

sudo rm /etc/nginx/sites-enabled/default

Then use the following to restart nginx.

sudo systemctl restart nginx

Other notes

On a default nginx install on ubuntu 20.04, the http directive is already included in /etc/nginx/nginx.conf this means its not required in graphservice-load-balancer.conf.

METRICS

If you are actively monitoring your infrastructure, do not forget that by adding more service nodes you need to instrument their metric endpoints individually i.e. scrape their metric endpoints into Prometheus (so thats coming from port 7300 on each service node)

Testing that your loadbalancer configuration setup is working is easiest by monitoring the prometheus metrics for the service on 7300 to see if messages are incoming from the gateway.