Gentrack Support

Installing the Data Gateway

Before you can connect your instance of Gentrack Velocity or Junifer to Gentrack Cloud Integration Services, Data Gateway MUST be installed.

Data Gateway acts as a communication broker for an instance of Junifer or Velocity, and Gentrack Cloud Integration Services running as cloud service. It is multi-tenanted so that one Data Gateway will be used for all your environments including Development, Test, and Production.

Who should install Data Gateway?

  • Customers using our managed services - Data Gateway has already been set up. Proceed to Connecting a Core System.
  • All other customers, including those with an on-premise environment -  You must install and manage the Data Gateway. Use the instructions in this article.

System requirements

The Data Gateway uses Docker which can be run on a physical machine, virtual machine, or container service like Amazon ECS or Azure AKS.

  • Operating System and kernel that is supported by Docker, this includes: CentOS, Debian, Fedora, Ubuntu.
  • 2 vCPU (4 vCPU or more is recommended).
  • 4 GB of RAM (8 GB or more is recommended).
  • 30 GB of Hard Disk.
  • A static IP address.

Before you begin

Before installing Data Gateway, ensure that you:

  • Confirm that your Data Gateway server meets the minimal system requirements
  • Have administrator (sudo) access to the server
  • Know the hostname of the server

Installation

To install and set up Data Gateway, complete the following tasks:

Set up a Docker Hub user

The Data Gateway is containerised application that is deployed using Docker.  The Docker images are hosted in a private Docker Hub repository.

To get ready to install Data Gateway using Docker:

  1. Create your own Docker Hub user account
  2. Provide the user name to Gentrack using a dedicated support channel.

The Gentrack support team will give that user read permissions to the repository. 

The management of passwords and credentials for that user is your responsibility.  If you wish to change the Docker Hub user please contact us directly.  If the Docker Hub password is ever exposed you must immediately reset the password and then contact us.

Set up Docker

The following instructions are targeted to Docker Community Edition on CentOS server.  Docker compose is used for container orchestration and management.  Docker is managed as a non-root user and must automatically start on boot in the case of server restart. 

Docker installation instructions to achieve this can be found in the following Docker Docs:

The sudo/root user that installs Data Gateway must be a member of the docker group.

Set up a Data Gateway user

A new user must be created to run the Data Gateway.  These instructions assume the name of the user will be gatewayuser.

  1. Add a user named gatewayuser, add home directory for the user, and add it to the docker group.

    sudo adduser gatewayuser -m --system -g docker
  2. Temporarily switch to gatewayuser and login into Docker Hub.  You will need to use the Docker Hub credentials previously provided to Gentrack.

    sudo su - gatewayuser -c "docker login"

gatewayuser MUST be added in the sudoers file in the Linux Operating System.

Install Data Gateway

  1. Download the following files and copy them into the user home directory.

  2. Add execution permissions to the downloaded installation script 

    chmod +x ~/setup-gateway.sh
  3. Execute the installation script by using the following settings as a sudo/root user.

    Note: if a Data Gateway has already been installed, the installation script will remove the exsiting gateway along with all previously configured tenants, and perform a fresh installation on the server. Also, the installation script will select a correct docker image tag for your Data Gateway, based on your region indicated by the Integration Services URL.

    sudo ~/setup-gateway.sh [Gentrack Cloud Url] [Gateway Host Name] ~/docker-compose.yml  

    The bash script setup-gateway.sh will create the following:

    • A directory with Data Gateway configuration files in the gatewayuser’s home directory
    • Security encryption keys and certificates
    • Docker Volumes that will be used to store persistent data
    • Daily jobs to remove old application logs and Docker images
    • System service so that Data Gateway will automatically start or restart after system maintenance or reboot
    • Unique credentials for the Gateway application to connect to the RabbitMQ instance

Test your installation

  1. Check if the Data Gateway service is running, it should show the service is active.

    sudo systemctl status gentrack-gateway-docker
  2. Navigate to https://[hostname] and you should see the Data Gateway landing page where [hostname] is the domain name of the Data Gateway server

  3. Navigate to http://[hostname:15672] and log in to the RabbitMQ management interface with credentials defined in _platform/docker-compose.yml _in the gateway user’s home directory

Install Custom Certificate

By completing the above installation, the Data Gateway app server uses a self-signed certificate by default. If you want to use a certificate signed by a trusted certificate authority (CA), there are two ways to install it:

  1. Use an API Gateway or Application Load Balancer If Data Gateway is deployed in AWS EC2 instance, you can use an Application Load Balancer or API Gateway in front of the Data Gateway. Then import your SSL certificate to AWS Certificate manager. Configure your API Gateway or Application Load Balancer to use the certificate. Please refer to following AWS documentation:

  2. Install a certificate via the certificate import tool You can also install a SSL certificate inside the Data Gateway docker container. Follow the steps below:

    • Step 1. Prepare a PKCS12 certificate The certificate must be in p12/pfx file format.

    • Step 2. Copy the certificate to gateway docker container config folder Assume the certificate path is /etc/ssl/certs/cert.p12, run the following command to copy the certificate to gateway docker container config folder

    sudo cp /etc/ssl/certs/cert.p12 /var/lib/docker/volumes/platform_GatewayConfig/_data/cert.p12
    • Step 3. Run import certificate tool Run the following command to import the certificate. It will ask for the password of the certificate. Then it will update the certificate information into gateway_vault docker container.
    [user@gcis-data-gateway ~]$ docker exec -it gateway_app node import-cert -c ./config/cert.p12
    ✔ Please enter certificate password or press enter if no password. Press Ctrl+C to exit: … ******
    2019-09-22T23:22:18.458Z - [info]: [VAULT] Certificate updated
    • Step 4. Remove the certificate in gateway docker container config folder
    sudo rm /var/lib/docker/volumes/platform_GatewayConfig/_data/cert.p12
    • Step 5. Restart the gateway docker. Then the installation is complete.
    sudo systemctl restart gentrack-gateway-docker

Enable TLS Certificates for Core System

When connecting to the core system using HTTPS, e.g., calling a Velocity REST API, the GCIS Data Gateway verifies server’s certificate as part of SSL/TLS handshake. It is recommended to use trusted CA signed certificates for the core system.

If the server certificate is signed from a unknown CA, e.g. self-signed or privately signed certificates, the GCIS Data Gateway will refuse the connection.

If you want to allow the use of self-signed or private CA signed certificate, please follow below guidance to enable it.

If you are using a private CA certificate, you can import the root CA to the gateway by:

  1. Use the update-ca-certificates program of the ca-certificates utility to import additional CA certificates. On different Linux distributions,, the process is slightly different. See here

  2. Mount CA certificates to the gateway_app docker

    • Locate the destination file generated by the update-ca-certificates program, which will contain the additional CA certificates. For example, on CentOS 7, the destination file is /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.

    • Update docker-compose.yml to mount the CA bundle from the host to the docker and add an environment variable NODE_EXTRA_CA_CERTS to point to the file path within docker. Then Restart the docker service

version: "3.1"
services:
app:
...
    container_name: gateway_app
    volumes:
      # mount the CA bundle from host to docker.
      - /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/etc/ssl/certs/ca-bundle.crt:ro
      ....
    environment:
      # instruct NodeJs to use the imported CA bundle
      - NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-bundle.crt
...

Update docker-compose.yml to include NODE_TLS_REJECT_UNAUTHORIZED=0 environment variable and restart the docker service.

version: "3.1"
services:
app:
...
  container_name: gateway_app
  ...
  environment:
    - NODE_TLS_REJECT_UNAUTHORIZED=0
...

Useful commands

  • Start/Stop/Restart Data Gateway using Systemd/Linux

    sudo systemctl start gentrack-gateway-docker
    sudo systemctl stop gentrack-gateway-docker
    sudo systemctl restart gentrack-gateway-docker
  • Start/Stop Data Gateway manually

    /usr/local/bin/docker-compose -f /home/gatewayuser/platform/docker-compose.yml up -d
    /usr/local/bin/docker-compose -f /home/gatewayuser/platform/docker-compose.yml down
  • Check the running user of the gateway container

    docker exec -t -i gateway_app id
  • List and inspect volumes

    docker volume ls
    docker volume inspect [platform_GatewayData]
  • List and inspect containers

    docker container ls
    docker container inspect [gateway_app]

Set up firewall security

Once the Data Gateway has been installed Firewall rules must be used to restrict access.  The rules may be enforced externally or through a service like IP tables.

The current Gentrack support IP addresses are

  • 203.167.214.160 (Auckland, New Zealand)
  • 81.108.106.50 (London, England)

The current Gentrack Cloud Integration Services IP addresses are

  • 52.65.53.1
  • 13.236.220.175
  • 35.176.213.75
  • 18.130.141.102

Data Gateway ports and firewall rules

DirectionProtocol and PortRestricted to IP or ServiceReason
InboundHTTPS - 443/tcp
  • IP address of the core systems
  • IP address of the administrator system
  • Gentrack support IP addresses
  • Gentrack Cloud Integration Services IP addresses
  • Allows API events to be sent from Junifer or Velocity
  • Allows your support staff to perform tenant registration
  • Allows Gentrack support staff to assist with tenant registration
  • Allows Gentrack Cloud Integration Services API access to core system
Inbound15672/tcp
  • Gentrack support IP addresses
  • Allows Gentrack support staff to monitor message throughput and identify potential issues
Outbound

Velocity - 443/tcp

Junifer - 43002/tcp

These are the default ports but may differ

  • IP address of the core systems
  • Allows webservice API on the core systems to be invoked by the Data Gateway
OutboundHTTPS/443
  • Docker Hub
  • Datadog
  • Sumo Logic
  • Operating System update servers and repositories
  • Amazon Web Services (eu-west-2 or ap-southeast-2)
  • Used to download and update Data Gateway
  • Availability and performance metrics
  • Application log monitoring
  • Receive updates and security patches for the server
  • Connect to Gentrack Cloud Integration Services
OutboundDNS
  • Must be able to resolve domain names
OutboundDHCP
  • IP address assignment
OutboundNTP
  • Access to get time from global time servers

Example using IP tables

Warning: The script in this example may stop SSH connections if run with an incorrect input.

The following bash script contains IP table rules for a Junifer deployment.  It will drop all inbound network traffic that is not related to the Data Gateway.  It uses two environment variables which must be set prior to execution.

  • ADMIN_IP - The static IP address of the administrator that is able to manage the Data Gateway.  SSH is allowed from this IP address.
  • JUNIFER_IP - The static IP address of the Junifer server.
#!/bin/bash

# stop the Data Gateway and Docker
echo ""Stopping Docker service...""
systemctl stop gentrack-gateway-docker.service
systemctl stop docker
sleep 5

# forget old rules
echo ""Replacing all firewall rules...""
iptables -F
iptables -X
iptables -Z

# set default policy to drop
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP

# allow loopback
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT

# drop invalid packets
iptables -A INPUT  -m state --state INVALID -j DROP
iptables -A OUTPUT -m state --state INVALID -j DROP
iptables -A FORWARD -m state --state INVALID -j DROP

# allow established, related packets we've already seen
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# output chain
iptables -A OUTPUT -p tcp -m tcp --dport 53 -m comment --comment ""DNS-TCP"" -j ACCEPT
iptables -A OUTPUT -p udp -m udp --dport 53 -m comment --comment ""DNS-UDP"" -j ACCEPT
iptables -A OUTPUT -p udp -m udp --dport 67:68 -m comment --comment ""DHCP"" -j ACCEPT
iptables -A OUTPUT -p tcp -m tcp --dport 80 -m comment --comment ""HTTP"" -j ACCEPT
iptables -A OUTPUT -p tcp -m tcp --dport 443 -m comment --comment ""HTTPS"" -j ACCEPT

# allow access to Junifer
iptables -A OUTPUT -p tcp -m tcp --dport 43002 -m comment --comment ""JUNIFER"" -j ACCEPT

# allow icmp packets (e.g. ping...)
iptables -A INPUT -p icmp -m state --state NEW -j ACCEPT

# allow SSH from key IP addresses
iptables -A INPUT -s $ADMIN_IP -p tcp --dport 22 -j ACCEPT

# pre-docker rules
iptables -N DOCKER-USER
iptables -A DOCKER-USER -i eth0 -s 203.167.214.160 -p tcp --dport 3000 -j ACCEPT
iptables -A DOCKER-USER -i eth0 -s 203.167.214.160 -p tcp --dport 15672 -j ACCEPT
iptables -A DOCKER-USER -i eth0 -s 52.65.53.1 -p tcp --dport 3000 -j ACCEPT
iptables -A DOCKER-USER -i eth0 -s 13.236.220.175 -p tcp --dport 3000 -j ACCEPT
iptables -A DOCKER-USER -i eth0 -s 35.176.213.75 -p tcp --dport 3000 -j ACCEPT
iptables -A DOCKER-USER -i eth0 -s 18.130.141.102 -p tcp --dport 3000 -j ACCEPT
iptables -A DOCKER-USER -i eth0 -s $ADMIN_IP -p tcp --dport 3000 -j ACCEPT
iptables -A DOCKER-USER -i eth0 -s $JUNIFER_IP -p tcp --dport 3000 -j ACCEPT
iptables -A DOCKER-USER -i eth0 -p tcp --dport 443 -j DROP
iptables -A DOCKER-USER -i eth0 -p tcp --dport 3000 -j DROP
iptables -A DOCKER-USER -i eth0 -p tcp --dport 15672 -j DROP
iptables -A DOCKER-USER -j RETURN

# restarting Docker service will add DOCKER tables
echo ""Starting Docker service...""
systemctl start docker.service
sleep 5

# save iptable rules and restart service
echo ""Saving firewall rules, and restarting firewall...""
service iptables save
systemctl restart iptables
sleep 5

# starting Data Gateway will add DOCKER rules
echo ""Starting Data Gateway...""
systemctl start gentrack-gateway-docker.service

Set up monitoring services

Operational monitoring of the Data Gateway is important.  For the Gentrack support team to assist with issues, we recommend enabling the following monitoring services.  The examples below do not include proxy server settings.  Additional configuration steps may be needed if using a web proxy.

Sumo Logic

Sumo Logic is used to collect application logs from Docker. A collection agent is run as a Docker container and will send logs back to Sumo Logic.

  1. Uncomment the following container lines at the bottom of the services section in platform/docker-compose.yml by removing the # character.

      sumologic:
          image: sumologic/collector:latest
          container_name: gateway_sumologic
          volumes:
            - /var/run/docker.sock:/var/run/docker.sock
          environment:
            - SUMO_ACCESS_ID=[insert API key here]
            - SUMO_ACCESS_KEY=[insert API key here]
            - SUMO_COLLECTOR_NAME=[insert name here]
  2. Update the Sumo Logic environment variables with the following settings:

    • SUMO_ACCESS_ID - Access identifier for the collector. This will be setup in Sumo Logic first. This may be provided by Gentrack if we are going to monitor this for you, or you can use your own Sumo Logic collector.
    • SUMO_ACCESS_KEY - Access key for the collector. This will be setup in Sumo Logic first. This may be provided by Gentrack if we are going to monitor this for you, or you can use your own Sumo Logic collector.
    • SUMO_COLLECTOR_NAME - Name that will be used to unique identify the Data Gateway.
  3. Restart the Data Gateway

    sudo systemctl restart gentrack-gateway-docker

Datadog

Datadog is used to capture performance metrics and system availability.  An agent is run as a Docker container and will send information back to Datadog. Using Datadog, alerts can be triggered if Data Gateway is not running.

  1. Uncomment the following container lines at the bottom of the services section in platform/docker-compose.yml by removing the # character.

      datadog:
        image: datadog/docker-dd-agent:latest
        container_name: gateway_datadog
        volumes:
          - /var/run/docker.sock:/var/run/docker.sock
          - /proc/:/host/proc/
          - /sys/fs/cgroup/:/host/sys/fs/cgroup
          - ~/platform/http_check.yaml:/conf.d/http_check.yaml
        environment:
          - API_KEY=[insert API key here]
          - SD_BACKEND=docker
          - DD_HOSTNAME=[insert name here]
  2. Update the Datadog environment variables with the following settings:

    • API_KEY - This may be provided by Gentrack if we are going to monitor this for you, or you can use your own API key.
    • DD_HOSTNAME - Name that will be used to unique identify the Data Gateway. This is needed because the API key will be shared by multiple agents. You may want to use the hostname of the server. If omitted a randomly generated name may be used which will change each time Docker is restarted.
  3. Restart the Data Gateway

    sudo systemctl restart gentrack-gateway-docker

Updates to Data Gateway

The Data Gateway automatically updates itself by regularly checking Docker Hub for new images.  When a new version is found it will stop the Data Gateway, pull the new images, and then start the Data Gateway again.  This is achieved by using Watchtower which is a Docker container that is started alongside the Data Gateway.  It uses the same Docker Hub credentials for the Data Gateway docker user.

Data Gateway Backup and Restore

Managed Data Gateway

GCIS is responsible for backup and restore of managed Data Gateway

Self-hosted Data Gateway

The Data Gateway stores these types of data:

  • Application data - Event data stored while the Data Gateway processing core API events. Application data is time-sensitive and there is very little value of backup.
  • Gateway configuration - The configuration is created by the setup script during the installation of the Data Gateway.
  • Tenant data - Information about every connected tenant.

Image-based backup

If the Data Gateway is hosted on a virutal machine, either locally or in the cloud, it is recommended to create an backup of the entire virtual machine, including the operating system and all the data associated with it. For example, for a Data Gateway hosted on an AWS EC2 instance, you can create a schedule task to back up the Data Gateway as AMI using tools provided AWS.

During disaster recovery you can restore a Data Gatway from the backup image, for example, by launching a fresh EC2 instance from the lastest AMI.

Backup key configuration files

You can also selectively save these key configuration files if applicalbe.

  • Docker compose file /home/gatewayuser/platform/docker-compose.yaml
  • SumoLogic collector configuration file /opt/SumoCollector/config/user.properties

Note: You need to keep a copy of the Docker compose file only if you have customised it for your Data Gateway. For example, you have changed the default value of and environment variable in the file. For the SumoLogic collector configuration, you should use the same name to keep the Gateway logs be published to the same collector in the SumoLogic cloud.

For disaster recovery you can rebuild the Data Gateway by the following process.

  • Install a new Data Gateway.
  • Apply configuration files from your backup if there is any.
  • Log into the GCIS deverloper portal and navigate to the tenants page.
  • Reconnect tenants that are connected to the old Data Gateway.
    • Click the Disconnect tenant button to disconnect.
    • Click the Connect tenant button to connect to the new Data Gateway.

Troubleshooting


Loading the web page displays a security exception

Cause

The Data Gateway uses a self-signed certificate for encryption. The web browser can’t validate the authenticity of the certificate so it will stop you accessing it.

Resolution

Add the certificate as trusted in your browser.


The web page fails to display

Cause

The Docker containers failed to start.

Resolution

Check which Docker containers are running

docker ps -a

The following containers should be running

  • gateway_app
  • gateway_rabbitmq
  • gateway_vault
  • gateway_redis
  • gateway_watchtower

If any of these are stopped then the logs can be viewed by using the container name

docker logs gateway_app

Docker containers are running but the web page fails to display

Cause

This may be due to firewall ports not being open.

Resolution

Internally Docker maps port 3000 on the container to the default HTTPS port 443.

Check that port 443 and 3000 has been enabled in any firewalls.


Error: [VAULT] unable to retrieve credentials ‘key’ must be specified in request body as JSON, or ‘reset’ set to true

Cause

The Data Gateway has initialised the Vault successfully but failed to save received credentials to the file vault/vault_cred.

Vault is used to securely store authentication tokens and API keys.

Resolution

  1. Make sure the Gateway is run as the correct user and has permission to write to the directory vault/vault_cred under the gatewayuser_GatewayData volume.

    docker exec -t -i gateway_app id
  2. Delete all files in the directory data/vault/file under the gatewayuser_VaultData volume, to allow the Gateway to re-initialise the vault.


Fail to start the Data Gateway service

Cause

The following services are not installed. - proc-sys-fs-binfmt_misc.mount - proc-sys-fs-binfmt_misc.automount

Resolution

The above services are required by the Datadog docker container to collect system logs. In some cases, they might not be applicable, therefore, not installed in the Operating System.

  1. Edit the file /etc/systemd/system/gentrack-gateway-docker.service to remove proc-sys-fs-binfmt_misc.mount and proc-sys-fs-binfmt_misc.automount in both After and Requires units.

    After=docker.service
    Requires=docker.service
    
  2. Reload the updated service configuration.

    systemctl daemon-reload
  3. Restart the Data Gateway service.

Last updated on 17 Oct 2019