Skip to content

DevOps: Diffusion Marketplace Server Setup from Scratch (for VAEC)

Jump to bottom
mutsuoara edited this page Aug 2, 2021 · 15 revisions

Diffusion Marketplace Server Setup from Scratch

Note: the struck out instructions are for historical and debugging purposes. You do not have to run those commands to set up the server.

Note: original instructions here: https://docs.google.com/document/d/1DhaWb1eTRSPMYWpaOJj2sRnN1DmEpjpvYLgkA0fXq0w/edit?usp=sharing

Get into VAEC

https://prod.adfs.federation.va.gov/adfs/ls/idpinitiatedsignon.aspx

Get contents of .env file

  • Ask dm-devops team for contents of .env file, or figure it out from the README on Github

Create Redhat instance.

  • Choose vaec-ami-dmva ami-0b06502d141079259
  • Choose DMA VAEC RHEL - ami-06470d67 this has the RHEL license and updates installed already
    • If you do not have the above, choose RH 7.6 ITOPS-RHEL HVM Encrypted 2019-03-19 - ami-7b76341a
      • this is the VA flavored AMI. You need to request and install a RHEL license to be able to install anything with yum.
      • AWS Marketplace should be fine to use if you do not have a RHEL license, though it may not pass nessus scans (RHEL-7.6_HVM*)
  • Save the key as a filename that is memorable and remember where you put it. It is important to have it.
  • Add the server to the default security group. Otherwise, it cannot communicate with the database

## Register the server instance https://vaww.vashare.oit.va.gov/sites/euo/EUOWiki/Wiki/Enterprise%20Satellite%20Server%20Registration.aspx

To perform the registration, download the basic bootstrap script from the Enterprise Satellite server. You will then edit the file to include the activation keys that you were given and execute the script.

Example:

cd ~
wget http://vasat.aac.va.gov/pub/bootstrap/bootstrap.sh
vim bootstrap.sh
### see below for edits
bash bootstrap.sh

Edit line 75 to include your activation keys. For example if you were given the keys 3-example-app, 3-loc-wdc, and 3-v2s: Change this:

ACTIVATION_KEYS=

To this:

ACTIVATION_KEYS=3-example-app,3-loc-wdc,3-v2s

Run the bootstrap.sh script

chmod +x bootstrap.sh
sudo bash bootstrap.sh

Create S3 bucket

  • Remember the name of the bucket
  • Create a policy and role to use the bucket via CloudFormation
    • Sample of role to upload:
---
AWSTemplateFormatVersion: 2010-09-09
Description: Example role, policy and instance profile for project admin using IAM boundary
  # MUST HAVE for mult-region - Stack name starts with "project-"

Resources:
  rS3Policy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      # MUST HAVE - Policy name begins with "project-"
      # MUST HAVE for mult-region - Stack name starts with "project-"
      # Auto-assigned name = <stack-name>-<resourcename>-<random>
      # Example Auto assigned name = project-s3-role-example-rS3Policy-1ST3R7IYC9JJ5
      Description: Project Administrator created S3 Policy
      # MUST HAVE - Policy document Path: /project/
      Path: /project/
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: AllowS3ReadAccess
            Effect: Allow
            Action:
              - s3:ListObject
              - s3:ListBucket
              - s3:GetObject
              - s3:GetObjectVersion
              - s3:PutObjectAcl
              - s3:PutObject
              - s3:DeleteObject

            Resource:
              - !Sub arn:${AWS::Partition}:s3:::dev-dm
              - !Sub arn:${AWS::Partition}:s3:::dev-dm/*

  rEc2Role:
    Type: AWS::IAM::Role
    Properties:
      # MUST HAVE - IAM role name begins with "project-"
      # MUST HAVE for mult-region - Stack name starts with "project-"
      # Auto-assigned name = <stack-name>-<resourcename>-<random>
      # Example Auto assigned name = project-s3-role-example-rEc2Role-1ST3R7IYC9JJ5
      # MUST HAVE - Role's Path: /project/
      Path: /project/
      # MUST HAVE - PermissionsBoundary set exactly as shown below, otherwise role creation will fail
      PermissionsBoundary: !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:policy/vaec/project-admin
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Principal:
            Service: ec2.amazonaws.com
          Action:
          - sts:AssumeRole
      ManagedPolicyArns:
      - !Ref rS3Policy

  rInstProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      # MUST HAVE - IAM role name begins with "project-"
      # MUST HAVE for mult-region - Stack name starts with "project-"
      # Auto-assigned name = <stack-name>-<resourcename>-<random>
      # Example Auto assigned name = project-s3-role-example-rInstProfile-1ST3R7IYC9JJ5
      # MUST HAVE - Instance Profile Path: /project/
      Path: /project/
      Roles:
      - !Ref rEc2Role
  • Attach role to the server created in step 3

Create Postgres instance in RDS

  • Postgres 9.6
  • Be sure to enable encryption for the data
  • Be sure to securely save the username, password, database name, and the database host (endpoint)

Create a Redis instance

  • Be sure to securely save the endpoint
  • Also add it to the security group and/or VPC that your server is on
  • Expose port 6379 of the security group

Generate certs

  • sudo openssl req -new -newkey rsa:2048 -keyout /root/cert/<common.name>.key -out /root/cert/<common.name>.csr -config /root/cert/<common.name>.cnf
  • request ssl certs https://vaww.pki.va.gov/ssltls/

Install SSL Certs

  • import SSL cert to AWS' Certificate Manager
  • create an ELB (ElasticLoadBalancer)
  • add your instance to the ELB
  • add your certificate to the ELB
  • open up port 443, close port 80
  • health check: make sure it is tcp to port 80

Request a CNAME record to resolve load balancer dns to whatever host name we want, such as staging.marketplace.va.gov

Affected User
Blake Henderson

Assignment group
IO.SS.FF.ACTIVEDIRECTORY.TEAM8

Assigned to
Denver Griffith (vaaitcgriffd) 

Preview record for field: Assigned to
Impact
1 - Critical

Urgency
3- Medium

Priority
3 - Moderate

Short Description
Create/Modify Existing DNS Entry - Diffusion Marketplace -VA Enterprise Cloud AWS GovCloud

Description
Create dev.marketplace.va.gov
and point to dev Diffusion Marketplace load balancer
internal-dev-dm-2048994434.us-gov-west-1.elb.amazonaws.com

I would like https://internal-dev-dm-2048994434.us-gov-west-1.elb.amazonaws.com/ to resolve to https://dev.marketplace.va.gov/ if at all possible, please. Thank you :)

Get access to jumpbox via Reflections

  • That’s a whole process in itself to get into the group cldunixp_userprofiles using the ePas process
  • vac10acsecs201.va.gov
  • vac10acsecs202.va.gov

ssh into created server in step 3

  • Get key onto this jumpbox
    • vi <environment name>-<application name>.key
    • Open up the key from step 3b in a text editor and copy the contents
    • Paste the contents in the VI opened file
    • Save the file
  • ssh -i <path to key here> ec2-user@<private ip address of server here>
  • You’ll usually find this in the ec2 dashboard when you click on the instance

Install all dependencies:

  • sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
    • sometimes this works and sometimes it doesn't
  • sudo yum install -y nano wget openssl-devel gcc-c++ expat-devel autoconf yum-utils device-mapper-persistent-data lvm2
    • curl-devel wasn't working this time
  • sudo yum install -y gettext-devel openssl-devel perl-CPAN perl-devel zlib-devel
  • sudo yum install -y http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.107-1.el7_6.noarch.rpm
  • yum-config-manager --enable rhel-7-server-rhui-optional-rpms
  • yum-config-manager --enable rhel-7-server-rhui-supplementary-rpms
  • yum-config-manager --enable rhel-7-server-rhui-extras-rpms

Install git

  • yum install -y wget openssl-devel curl-devel gcc-c++ expat-devel autoconf
  • sudo wget https://www.kernel.org/pub/software/scm/git/git-2.28.0.tar.gz
  • sudo tar -zxf git-2.28.0.tar.gz
  • cd git-2.28.0
  • sudo make configure
  • sudo ./configure --prefix=/usr
  • sudo make all
  • sudo make install
  • git --version
  • cd ~

Install docker

  • sudo yum install -y http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.107-1.el7_6.noarch.rpm
  • sudo yum install -y yum-utils device-mapper-persistent-data lvm2
  • sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  • wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-18.09.9-3.el7.x86_64.rpm
  • sudo yum install -y docker-ce-18.09.9-3.el7.x86_64.rpm
    • If this doesn't work, do sudo yum install -y docker-ce with the preface that the yum-config-manager command worked
  • sudo usermod -aG docker $USER
  • sudo systemctl start docker
    • Note: if there is a way to start docker without sudo, that would be the way to go
  • docker --version

Install docker-compose

  • sudo curl -L https://github.com/docker/compose/releases/download/1.25.0-rc2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose -k
  • sudo chmod +x /usr/local/bin/docker-compose
  • sudo docker-compose --version
  • Note, you have to start docker as root, but compose is installed as the user, so, we need to link the executable to root, so run this:
    • sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
  • Jut ran into this error: docker-compose: error while loading shared libraries: libz.so.1: failed to map segment from shared object: Operation not permitted
    • resolved with this: sudo mount /tmp -o remount,exec

Clone the repo

  • git config --global http.sslVerify false
  • git clone https://github.com/agilesix/diffusion-marketplace.git
  • cd diffusion-marketplace
    • (optional) git fetch origin <branch> && git checkout <branch>
      • This is if there are different configs or features on a certain branch that only work in vaec or a certain build we want to deploy

Install certificates on Application Image:

  • sudo cp /etc/pki/ca-trust/source/anchors/VA-Internal-S2-RCA1-v1.cer ~/diffusion-marketplace

Set environment variables

  • sudo yum install -y nano
    • I like using nano as a text editor
    • Nothing against vi or vim, I just suck at using them
  • nano .env
  • Ask dm-devops team for contents of .env file, or figure it out from the README on Github
  • Example .env:
# AWS_ACCESS_KEY_ID=S3BUCKETUSER # may not need this
AWS_REGION=us-gov-west-1
# AWS_SECRET_ACCESS_KEY=1234 # may not need this
FORCE_SSL=true
[email protected]
GA_TRACKING_ID=fix-me
HOSTNAME=https://dev.marketplace.va.gov
LDAP_USERNAME=fix-me
LDAP_PASSWORD=fix-me
[email protected]
POSTGRES_DB=vaecdiffusionmarketplacedev
POSTGRES_HOST=vaecdiffusionmarketplacedev.cychy2qycdiv.us-gov-west-1.rds.amazonaws.com
POSTGRES_PASSWORD=fix-me
POSTGRES_PORT=5432
POSTGRES_USER=vaecdiffusionmarketplacedev
RAILS_ENV=production
REDIS_URL=redis://dev-dm.sybd24.0001.usgw1.cache.amazonaws.com:6379/0
S3_BUCKET_NAME=dev-dm
SES_SMTP_PASSWORD=fix-me
SES_SMTP_USERNAME=fix-me
SURVEY_MONKEY_PASSWORD=fix-me
SURVEY_MONKEY_TOKEN=fix-me
SURVEY_MONKEY_USERNAME=fix-me
SURVEY_MONKEY_EP201=fix-me
SURVEY_MONKEY_EP202=fix-me
SURVEY_MONKEY_EP203=fix-me
USE_NTLM=true
GOOGLE_API_KEY=fix-me
  • Place the contents into the .env file you opened in step b
  • Replace values with any real values you have available
  • Crucial ones are the db connection variables
POSTGRES_HOST
POSTGRES_DB
POSTGRES_USER
POSTGRES_PORT
POSTGRES_PASSWORD
  • Set environment variables via the .env file
    • while read LINE; do export "$LINE"; done < ./.env
  • printenv
  • Check if all of the variables were added

Build and run the docker containers

  • docker-compose build
  • docker-compose up -d
  • note: you may need sudo in front of these two commands

Browse to the private IP address provided for the server made in step 3

  • You’ll usually find this in the ec2 dashboard when you click on the instance

Enable ipv4 Forwarding

  • sudo nano /etc/sysctl.conf
  • add net.ipv4.ip_forward = 1 to the bottom of the file
  • Save the file

Create Restart Server Service

  • vi /usr/local/lib/systemd/system/appstartup.service
#!/bin/bash

################################################################################
# appstartup.service
#
# This service unit is for testing my systemd startup service
# By David Both
# Licensed under GPL V2
#
################################################################################
# This program should be placed in /usr/local/lib/systemd/system/.
# Create a symlink to it from the /etc/systemd/system directory.
################################################################################

[Unit]

Description=Runs /usr/local/bin/appstartup.sh


[Service]

ExecStart=/usr/local/bin/appstartup.sh


[Install]

WantedBy=multi-user.target
  • Create a symlink to systemd directory ln -s /usr/local/lib/systemd/system/appstartup.service /etc/systemd/system/appstartup.service
  • Create shell script to start application vi /usr/local/bin/appstartup.sh 
#!/bin/bash

################################################################################
# mystartup.sh
#
# This shell program is for testing a startup like rc.local using systemd.
# By DMVA
# Licensed under GPL V2
#
################################################################################

# This program should be placed in /usr/local/bin

################################################################################
# This is a test entry

echo `date +%F" "%T` "Startup worked" >> /root/mystartup.log

cd /home/ec2-user/diffusion-marketplace
docker-compose up -d
  • Enable service sudo systemctl enable appstartup.service
Clone this wiki locally