Installing the Data Gateway
Before you can connect your instance of Gentrack Velocity or Junifer to Gentrack Cloud Integration Services, Data Gateway MUST be installed.
Data Gateway acts as a communication broker for an instance of Junifer or Velocity, and Gentrack Cloud Integration Services running as cloud service. It is multi-tenanted so that one Data Gateway will be used for all your environments including Development, Test, and Production.
Who should install Data Gateway?
- Customers using our managed services - Data Gateway has already been set up. Proceed to Connecting a Core System.
- All other customers, including those with an on-premise environment - You must install and manage the Data Gateway. Use the instructions in this article.
The Data Gateway uses Docker which can be run on a physical machine, virtual machine, or container service like Amazon ECS or Azure AKS.
- Operating system and kernel that is supported by Docker, this includes: CentOS, Debian, Fedora, Ubuntu.
- 2 vCPU (4 vCPU or more is recommended).
- 4 GB of RAM (8 GB or more is recommended).
- 30 GB of Hard Disk.
- A static IP address.
Before you begin
Before installing Data Gateway, ensure that you:
- Confirm that your Data Gateway server meets the
- Have administrator (sudo) access to the server
- Know the hostname of the server
To install and set up Data Gateway, complete the following tasks:
Set up a Docker Hub user
The Data Gateway is containerised application that is deployed using Docker. The Docker images are hosted in a private Docker Hub repository.
To get ready to install Data Gateway using Docker:
- Create your own Docker Hub user account
- Provide the user name to Gentrack using a dedicated support channel.
The Gentrack support team will give that user read permissions to the repository.
The management of passwords and credentials for that user is your responsibility. If you wish to change the Docker Hub user please contact us directly. If the Docker Hub password is ever exposed you must immediately reset the password and then contact us.
Set up Docker
The following instructions are targeted to Docker Community Edition on CentOS server. Docker compose is used for container orchestration and management. Docker is managed as a non-root user and must automatically start on boot in the case of server restart.
Docker installation instructions to achieve this can be found in the following Docker Docs:
The sudo/root user that installs Data Gateway must be a member of the docker group.
Set up a Data Gateway user
A new user must be created to run the Data Gateway. These instructions assume the name of the user will be
Add a user named gatewayuser, add home directory for the user, and add it to the docker group.
sudo adduser gatewayuser -m --system -g docker
Temporarily switch to gatewayuser and login into Docker Hub. You will need to use the Docker Hub credentials previously provided to Gentrack.
sudo su - gatewayuser -c "docker login"
gatewayuser MUST be added in the sudoers file in the Linux Operating System.
Install Data Gateway
Download the following files and copy them into the user home directory.
Add execution permissions to the downloaded installation script
chmod +x ~/setup-gateway.sh
Execute the installation script by using the following settings as a sudo/root user.
This installation script removes the Data Gateway if it is already installed, and removes all previously configured tenants.
sudo ~/setup-gateway.sh [Gentrack Cloud Url] [Gateway Host Name] ~/docker-compose.yml
The setup-gateway bash script will create the following:
- Directory containing configuration files in the gatewayuser home directory
- Security encryption keys and certificates
- Docker Volumes that will be used to store persistent data
- Daily jobs to remove old application logs and Docker images
- System service so that Data Gateway will automatically start or restart after system maintenance or reboot
- Unique credentials for the Gateway application to connect to the RabbitMQ instance
Test your installation
Check if the Data Gateway service is running, it should show the service is active.
sudo systemctl status gentrack-gateway-docker
Navigate to https://[hostname] and you should see the Data Gateway landing page where [hostname] is the domain name of the Data Gateway server
Navigate to http://[hostname:15672] and log in to the RabbitMQ management interface with credentials defined in
_platform/docker-compose.yml _in the gateway user’s home directory
Loading the web page displays a security exception
The Data Gateway uses a self-signed certificate for encryption. The web browser can’t validate the authenticity of the certificate so it will stop you accessing it.
Add the certificate as trusted in your browser.
The web page fails to display
The Docker containers failed to start.
Check which Docker containers are running
docker ps -a
The following containers should be running
If any of these are stopped then the logs can be viewed by using the container name
docker logs gateway_app
Docker containers are running but the web page fails to display
This may be due to firewall ports not being open.
Internally Docker maps port 3000 on the container to the default HTTPS port 443.
Check that port 443 and 3000 has been enabled in any firewalls.
Error: [VAULT] unable to retrieve credentials ‘key’ must be specified in request body as JSON, or ‘reset’ set to true
The Data Gateway has initialised the Vault successfully but failed to save received credentials to the file vault/vault_cred.
Vault is used to securely store authentication tokens and API keys.
Make sure the Gateway is run as the correct user and has permission to write to the directory vault/vault_cred under the gatewayuser_GatewayData volume.
docker exec -t -i gateway_app id
Delete all files in the directory data/vault/file under the gatewayuser_VaultData volume, to allow the Gateway to re-initialise the vault.
Start/Stop/Restart Data Gateway using Systemd/Linux
sudo systemctl start gentrack-gateway-docker sudo systemctl stop gentrack-gateway-docker sudo systemctl restart gentrack-gateway-docker
Start/Stop Data Gateway manually
/usr/local/bin/docker-compose -f /home/gatewayuser/platform/docker-compose.yml up -d /usr/local/bin/docker-compose -f /home/gatewayuser/platform/docker-compose.yml down
Check the running user of the gateway container
docker exec -t -i gateway_app id
List and inspect volumes
docker volume ls docker volume inspect [platform_GatewayData]
List and inspect containers
docker container ls docker container inspect [gateway_app]
Set up monitoring services
Operational monitoring of the Data Gateway is important. For the Gentrack support team to assist with issues, we recommend enabling the following monitoring services. The examples below do not include proxy server settings. Additional configuration steps may be needed if using a web proxy.
Sumo Logic is used to collect application logs from Docker. A collection agent is run as a Docker container and will send logs back to Sumo Logic.
Uncomment the following container lines at the bottom of the services section in
platform/docker-compose.ymlby removing the # character.
sumologic: image: sumologic/collector:latest container_name: gateway_sumologic volumes: - /var/run/docker.sock:/var/run/docker.sock environment: - SUMO_ACCESS_ID=[insert API key here] - SUMO_ACCESS_KEY=[insert API key here] - SUMO_COLLECTOR_NAME=[insert name here]
Update the Sumo Logic environment variables with the following settings:
- SUMO_ACCESS_ID - Access identifier for the collector. This will be setup in Sumo Logic first. This may be provided by Gentrack if we are going to monitor this for you, or you can use your own Sumo Logic collector.
- SUMO_ACCESS_KEY - Access key for the collector. This will be setup in Sumo Logic first. This may be provided by Gentrack if we are going to monitor this for you, or you can use your own Sumo Logic collector.
- SUMO_COLLECTOR_NAME - Name that will be used to unique identify the Data Gateway.
Restart the Data Gateway
sudo systemctl restart gentrack-gateway-docker
Datadog is used to capture performance metrics and system availability. An agent is run as a Docker container and will send information back to Datadog. Using Datadog, alerts can be triggered if Data Gateway is not running.
Uncomment the following container lines at the bottom of the services section in platform/docker-compose.yml by removing the # character.
datadog: image: datadog/docker-dd-agent:latest container_name: gateway_datadog volumes: - /var/run/docker.sock:/var/run/docker.sock - /proc/:/host/proc/ - /sys/fs/cgroup/:/host/sys/fs/cgroup - ~/platform/http_check.yaml:/conf.d/http_check.yaml environment: - API_KEY=[insert API key here] - SD_BACKEND=docker - DD_HOSTNAME=[insert name here]
Update the Datadog environment variables with the following settings:
- API_KEY - This may be provided by Gentrack if we are going to monitor this for you, or you can use your own API key.
- DD_HOSTNAME - Name that will be used to unique identify the Data Gateway. This is needed because the API key will be shared by multiple agents. You may want to use the hostname of the server. If omitted a randomly generated name may be used which will change each time Docker is restarted.
Restart the Data Gateway
sudo systemctl restart gentrack-gateway-docker
Set up firewall security
Once the Data Gateway has been installed Firewall rules must be used to restrict access. The rules may be enforced externally or through a service like IP tables.
The current Gentrack support IP addresses are
- 126.96.36.199 (Auckland, New Zealand)
- 188.8.131.52 (London, England)
The current Gentrack Cloud Integration Services IP addresses are
Data Gateway ports and firewall rules
|Direction||Protocol and Port||Restricted to IP or Service||Reason|
|Inbound||HTTPS - 443/tcp|
Velocity - 443/tcp
Junifer - 43002/tcp
These are the default ports but may differ
Example using IP tables
Warning: The script in this example may stop SSH connections if run with an incorrect input.
The following bash script contains IP table rules for a Junifer deployment. It will drop all inbound network traffic that is not related to the Data Gateway. It uses two environment variables which must be set prior to execution.
- ADMIN_IP - The static IP address of the administrator that is able to manage the Data Gateway. SSH is allowed from this IP address.
- JUNIFER_IP - The static IP address of the Junifer server.
#!/bin/bash # stop the Data Gateway and Docker echo ""Stopping Docker service..."" systemctl stop gentrack-gateway-docker.service systemctl stop docker sleep 5 # forget old rules echo ""Replacing all firewall rules..."" iptables -F iptables -X iptables -Z # set default policy to drop iptables -P INPUT DROP iptables -P OUTPUT DROP iptables -P FORWARD DROP # allow loopback iptables -A INPUT -i lo -j ACCEPT iptables -A OUTPUT -o lo -j ACCEPT # drop invalid packets iptables -A INPUT -m state --state INVALID -j DROP iptables -A OUTPUT -m state --state INVALID -j DROP iptables -A FORWARD -m state --state INVALID -j DROP # allow established, related packets we've already seen iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT # output chain iptables -A OUTPUT -p tcp -m tcp --dport 53 -m comment --comment ""DNS-TCP"" -j ACCEPT iptables -A OUTPUT -p udp -m udp --dport 53 -m comment --comment ""DNS-UDP"" -j ACCEPT iptables -A OUTPUT -p udp -m udp --dport 67:68 -m comment --comment ""DHCP"" -j ACCEPT iptables -A OUTPUT -p tcp -m tcp --dport 80 -m comment --comment ""HTTP"" -j ACCEPT iptables -A OUTPUT -p tcp -m tcp --dport 443 -m comment --comment ""HTTPS"" -j ACCEPT # allow access to Junifer iptables -A OUTPUT -p tcp -m tcp --dport 43002 -m comment --comment ""JUNIFER"" -j ACCEPT # allow icmp packets (e.g. ping...) iptables -A INPUT -p icmp -m state --state NEW -j ACCEPT # allow SSH from key IP addresses iptables -A INPUT -s $ADMIN_IP -p tcp --dport 22 -j ACCEPT # pre-docker rules iptables -N DOCKER-USER iptables -A DOCKER-USER -i eth0 -s 184.108.40.206 -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s 220.127.116.11 -p tcp --dport 15672 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s 18.104.22.168 -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s 22.214.171.124 -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s 126.96.36.199 -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s 188.8.131.52 -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s $ADMIN_IP -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s $JUNIFER_IP -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -p tcp --dport 443 -j DROP iptables -A DOCKER-USER -i eth0 -p tcp --dport 3000 -j DROP iptables -A DOCKER-USER -i eth0 -p tcp --dport 15672 -j DROP iptables -A DOCKER-USER -j RETURN # restarting Docker service will add DOCKER tables echo ""Starting Docker service..."" systemctl start docker.service sleep 5 # save iptable rules and restart service echo ""Saving firewall rules, and restarting firewall..."" service iptables save systemctl restart iptables sleep 5 # starting Data Gateway will add DOCKER rules echo ""Starting Data Gateway..."" systemctl start gentrack-gateway-docker.service
Updates to Data Gateway
The Data Gateway automatically updates itself by regularly checking Docker Hub for new images. When a new version is found it will stop the Data Gateway, pull the new images, and then start the Data Gateway again. This is achieved by using Watchtower which is a Docker container that is started alongside the Data Gateway. It uses the same Docker Hub credentials for the Data Gateway docker user.