Installing the Data Gateway
On this page
Before you can connect your instance of Gentrack Velocity or Junifer to Gentrack Cloud Integration Services, Data Gateway MUST be installed.
Data Gateway acts as a communication broker for an instance of Junifer or Velocity, and Gentrack Cloud Integration Services running as cloud service. It is multi-tenanted so that one Data Gateway will be used for all your environments including Development, Test, and Production.
Who should install Data Gateway?
- Customers using our managed services - Data Gateway has already been set up. Proceed to Connecting a Core System.
- All other customers, including those with an on-premise environment - You must install and manage the Data Gateway. Use the instructions in this article.
The Data Gateway uses Docker which can be run on a physical machine, virtual machine, or container service like Amazon ECS or Azure AKS.
- Operating System and kernel that is supported by Docker, this includes: CentOS, Debian, Fedora, Ubuntu.
- 2 vCPU (4 vCPU or more is recommended).
- 4 GB of RAM (8 GB or more is recommended).
- 30 GB of Hard Disk.
- A static IP address.
Before you begin
Before installing Data Gateway, ensure that you:
- Confirm that your Data Gateway server meets the minimal system requirements
- Have administrator (sudo) access to the server
- Know the hostname of the server
To install and set up Data Gateway, complete the following tasks:
Set up a Docker Hub user
The Data Gateway is containerised application that is deployed using Docker. The Docker images are hosted in a private Docker Hub repository.
To get ready to install Data Gateway using Docker:
- Create your own Docker Hub user account
- Provide the user name to Gentrack using a dedicated support channel.
The Gentrack support team will give that user read permissions to the repository.
The management of passwords and credentials for that user is your responsibility. If you wish to change the Docker Hub user please contact us directly. If the Docker Hub password is ever exposed you must immediately reset the password and then contact us.
Set up Docker
The following instructions are targeted to Docker Community Edition on CentOS server. Docker compose is used for container orchestration and management. Docker is managed as a non-root user and must automatically start on boot in the case of server restart.
Docker installation instructions to achieve this can be found in the following Docker Docs:
The sudo/root user that installs Data Gateway must be a member of the docker group.
Set up a Data Gateway user
A new user must be created to run the Data Gateway. These instructions assume the name of the user will be
Add a user named
gatewayuser, add home directory for the user, and add it to the docker group.
sudo adduser gatewayuser -m --system -g docker
Temporarily switch to
gatewayuserand login into Docker Hub. You will need to use the Docker Hub credentials previously provided to Gentrack.
sudo su - gatewayuser -c "docker login"
gatewayuser MUST be added in the sudoers file in the Linux Operating System.
Install Data Gateway
Download the following files and copy them into the user home directory.
Add execution permissions to the downloaded installation script
chmod +x ~/setup-gateway.sh
Execute the installation script by using the following settings as a sudo/root user.
- Integration Services URL
- Gateway Host Name - Fully qualified domain name of the Data Gateway server. This will be used for certificates that are automatically generated.
Note: if a Data Gateway has already been installed, the installation script will remove the exsiting gateway along with all previously configured tenants, and perform a fresh installation on the server. Also, the installation script will select a correct docker image tag for your Data Gateway, based on your region indicated by the Integration Services URL.
sudo ~/setup-gateway.sh [Gentrack Cloud Url] [Gateway Host Name] ~/docker-compose.yml
The bash script
setup-gateway.shwill create the following:
- A directory with Data Gateway configuration files in the gatewayuser’s home directory
- Security encryption keys and certificates
- Docker Volumes that will be used to store persistent data
- Daily jobs to remove old application logs and Docker images
- System service so that Data Gateway will automatically start or restart after system maintenance or reboot
- Unique credentials for the Gateway application to connect to the RabbitMQ instance
Test your installation
Check if the Data Gateway service is running, it should show the service is active.
sudo systemctl status gentrack-gateway-docker
Navigate to https://[hostname] and you should see the Data Gateway landing page where [hostname] is the domain name of the Data Gateway server
Navigate to http://[hostname:15672] and log in to the RabbitMQ management interface with credentials defined in
_platform/docker-compose.yml _in the gateway user’s home directory
Install Custom Certificate
By completing the above installation, the Data Gateway app server uses a self-signed certificate by default. If you want to use a certificate signed by a trusted certificate authority (CA), there are two ways to install it:
Use an API Gateway or Application Load Balancer If Data Gateway is deployed in AWS EC2 instance, you can use an Application Load Balancer or API Gateway in front of the Data Gateway. Then import your SSL certificate to AWS Certificate manager. Configure your API Gateway or Application Load Balancer to use the certificate. Please refer to following AWS documentation:
Install a certificate via the certificate import tool You can also install a SSL certificate inside the Data Gateway docker container. Follow the steps below:
Step 1. Prepare a PKCS12 certificate The certificate must be in p12/pfx file format.
Step 2. Copy the certificate to gateway docker container config folder Assume the certificate path is /etc/ssl/certs/cert.p12, run the following command to copy the certificate to gateway docker container config folder
sudo cp /etc/ssl/certs/cert.p12 /var/lib/docker/volumes/platform_GatewayConfig/_data/cert.p12
- Step 3. Run import certificate tool Run the following command to import the certificate. It will ask for the password of the certificate. Then it will update the certificate information into gateway_vault docker container.
[user@gcis-data-gateway ~]$ docker exec -it gateway_app node import-cert -c ./config/cert.p12 ✔ Please enter certificate password or press enter if no password. Press Ctrl+C to exit: … ****** 2019-09-22T23:22:18.458Z - [info]: [VAULT] Certificate updated
- Step 4. Remove the certificate in gateway docker container config folder
sudo rm /var/lib/docker/volumes/platform_GatewayConfig/_data/cert.p12
- Step 5. Restart the gateway docker. Then the installation is complete.
sudo systemctl restart gentrack-gateway-docker
Enable TLS Certificates for Core System
When connecting to the core system using HTTPS, e.g., calling a Velocity REST API, the GCIS Data Gateway verifies server’s certificate as part of SSL/TLS handshake. It is recommended to use trusted CA signed certificates for the core system.
If the server certificate is signed from a unknown CA, e.g. self-signed or privately signed certificates, the GCIS Data Gateway will refuse the connection.
If you want to allow the use of self-signed or private CA signed certificate, please follow below guidance to enable it.
Private CA signed certificate (Recommended)
If you are using a private CA certificate, you can import the root CA to the gateway by:
Use the update-ca-certificates program of the ca-certificates utility to import additional CA certificates. On different Linux distributions,, the process is slightly different. See here
Mount CA certificates to the gateway_app docker
Locate the destination file generated by the update-ca-certificates program, which will contain the additional CA certificates. For example, on CentOS 7, the destination file is /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.
Update docker-compose.yml to mount the CA bundle from the host to the docker and add an environment variable NODE_EXTRA_CA_CERTS to point to the file path within docker. Then Restart the docker service
version: "3.1" services: app: ... container_name: gateway_app volumes: # mount the CA bundle from host to docker. - /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/etc/ssl/certs/ca-bundle.crt:ro .... environment: # instruct NodeJs to use the imported CA bundle - NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-bundle.crt ...
Self-signed certificate (Not recommended)
Update docker-compose.yml to include NODE_TLS_REJECT_UNAUTHORIZED=0 environment variable and restart the docker service.
version: "3.1" services: app: ... container_name: gateway_app ... environment: - NODE_TLS_REJECT_UNAUTHORIZED=0 ...
Start/Stop/Restart Data Gateway using Systemd/Linux
sudo systemctl start gentrack-gateway-docker sudo systemctl stop gentrack-gateway-docker sudo systemctl restart gentrack-gateway-docker
Start/Stop Data Gateway manually
/usr/local/bin/docker-compose -f /home/gatewayuser/platform/docker-compose.yml up -d /usr/local/bin/docker-compose -f /home/gatewayuser/platform/docker-compose.yml down
Check the running user of the gateway container
docker exec -t -i gateway_app id
List and inspect volumes
docker volume ls docker volume inspect [platform_GatewayData]
List and inspect containers
docker container ls docker container inspect [gateway_app]
Set up firewall security
Once the Data Gateway has been installed Firewall rules must be used to restrict access. The rules may be enforced externally or through a service like IP tables.
The current Gentrack support IP addresses are
- 22.214.171.124 (Auckland, New Zealand)
- 126.96.36.199 (London, England)
The current Gentrack Cloud Integration Services IP addresses are
Data Gateway ports and firewall rules
|Direction||Protocol and Port||Restricted to IP or Service||Reason|
|Inbound||HTTPS - 443/tcp|
Velocity - 443/tcp
Junifer - 43002/tcp
These are the default ports but may differ
Example using IP tables
Warning: The script in this example may stop SSH connections if run with an incorrect input.
The following bash script contains IP table rules for a Junifer deployment. It will drop all inbound network traffic that is not related to the Data Gateway. It uses two environment variables which must be set prior to execution.
- ADMIN_IP - The static IP address of the administrator that is able to manage the Data Gateway. SSH is allowed from this IP address.
- JUNIFER_IP - The static IP address of the Junifer server.
#!/bin/bash # stop the Data Gateway and Docker echo ""Stopping Docker service..."" systemctl stop gentrack-gateway-docker.service systemctl stop docker sleep 5 # forget old rules echo ""Replacing all firewall rules..."" iptables -F iptables -X iptables -Z # set default policy to drop iptables -P INPUT DROP iptables -P OUTPUT DROP iptables -P FORWARD DROP # allow loopback iptables -A INPUT -i lo -j ACCEPT iptables -A OUTPUT -o lo -j ACCEPT # drop invalid packets iptables -A INPUT -m state --state INVALID -j DROP iptables -A OUTPUT -m state --state INVALID -j DROP iptables -A FORWARD -m state --state INVALID -j DROP # allow established, related packets we've already seen iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT # output chain iptables -A OUTPUT -p tcp -m tcp --dport 53 -m comment --comment ""DNS-TCP"" -j ACCEPT iptables -A OUTPUT -p udp -m udp --dport 53 -m comment --comment ""DNS-UDP"" -j ACCEPT iptables -A OUTPUT -p udp -m udp --dport 67:68 -m comment --comment ""DHCP"" -j ACCEPT iptables -A OUTPUT -p tcp -m tcp --dport 80 -m comment --comment ""HTTP"" -j ACCEPT iptables -A OUTPUT -p tcp -m tcp --dport 443 -m comment --comment ""HTTPS"" -j ACCEPT # allow access to Junifer iptables -A OUTPUT -p tcp -m tcp --dport 43002 -m comment --comment ""JUNIFER"" -j ACCEPT # allow icmp packets (e.g. ping...) iptables -A INPUT -p icmp -m state --state NEW -j ACCEPT # allow SSH from key IP addresses iptables -A INPUT -s $ADMIN_IP -p tcp --dport 22 -j ACCEPT # pre-docker rules iptables -N DOCKER-USER iptables -A DOCKER-USER -i eth0 -s 188.8.131.52 -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s 184.108.40.206 -p tcp --dport 15672 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s 220.127.116.11 -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s 18.104.22.168 -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s 22.214.171.124 -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s 126.96.36.199 -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s $ADMIN_IP -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -s $JUNIFER_IP -p tcp --dport 3000 -j ACCEPT iptables -A DOCKER-USER -i eth0 -p tcp --dport 443 -j DROP iptables -A DOCKER-USER -i eth0 -p tcp --dport 3000 -j DROP iptables -A DOCKER-USER -i eth0 -p tcp --dport 15672 -j DROP iptables -A DOCKER-USER -j RETURN # restarting Docker service will add DOCKER tables echo ""Starting Docker service..."" systemctl start docker.service sleep 5 # save iptable rules and restart service echo ""Saving firewall rules, and restarting firewall..."" service iptables save systemctl restart iptables sleep 5 # starting Data Gateway will add DOCKER rules echo ""Starting Data Gateway..."" systemctl start gentrack-gateway-docker.service
Set up monitoring services
Operational monitoring of the Data Gateway is important. For the Gentrack support team to assist with issues, we recommend enabling the following monitoring services. The examples below do not include proxy server settings. Additional configuration steps may be needed if using a web proxy.
Sumo Logic is used to collect application logs from Docker. A collection agent is run as a Docker container and will send logs back to Sumo Logic.
Uncomment the following container lines at the bottom of the services section in
platform/docker-compose.ymlby removing the # character.
sumologic: image: sumologic/collector:latest container_name: gateway_sumologic volumes: - /var/run/docker.sock:/var/run/docker.sock environment: - SUMO_ACCESS_ID=[insert API key here] - SUMO_ACCESS_KEY=[insert API key here] - SUMO_COLLECTOR_NAME=[insert name here]
Update the Sumo Logic environment variables with the following settings:
- SUMO_ACCESS_ID - Access identifier for the collector. This will be setup in Sumo Logic first. This may be provided by Gentrack if we are going to monitor this for you, or you can use your own Sumo Logic collector.
- SUMO_ACCESS_KEY - Access key for the collector. This will be setup in Sumo Logic first. This may be provided by Gentrack if we are going to monitor this for you, or you can use your own Sumo Logic collector.
- SUMO_COLLECTOR_NAME - Name that will be used to unique identify the Data Gateway.
Restart the Data Gateway
sudo systemctl restart gentrack-gateway-docker
Datadog is used to capture performance metrics and system availability. An agent is run as a Docker container and will send information back to Datadog. Using Datadog, alerts can be triggered if Data Gateway is not running.
Uncomment the following container lines at the bottom of the services section in platform/docker-compose.yml by removing the # character.
datadog: image: datadog/docker-dd-agent:latest container_name: gateway_datadog volumes: - /var/run/docker.sock:/var/run/docker.sock - /proc/:/host/proc/ - /sys/fs/cgroup/:/host/sys/fs/cgroup - ~/platform/http_check.yaml:/conf.d/http_check.yaml environment: - API_KEY=[insert API key here] - SD_BACKEND=docker - DD_HOSTNAME=[insert name here]
Update the Datadog environment variables with the following settings:
- API_KEY - This may be provided by Gentrack if we are going to monitor this for you, or you can use your own API key.
- DD_HOSTNAME - Name that will be used to unique identify the Data Gateway. This is needed because the API key will be shared by multiple agents. You may want to use the hostname of the server. If omitted a randomly generated name may be used which will change each time Docker is restarted.
Restart the Data Gateway
sudo systemctl restart gentrack-gateway-docker
Updates to Data Gateway
The Data Gateway automatically updates itself by regularly checking Docker Hub for new images. When a new version is found it will stop the Data Gateway, pull the new images, and then start the Data Gateway again. This is achieved by using Watchtower which is a Docker container that is started alongside the Data Gateway. It uses the same Docker Hub credentials for the Data Gateway docker user.
Data Gateway Backup and Restore
Managed Data Gateway
GCIS is responsible for backup and restore of managed Data Gateway
Self-hosted Data Gateway
The Data Gateway stores these types of data:
- Application data - Event data stored while the Data Gateway processing core API events. Application data is time-sensitive and there is very little value of backup.
- Gateway configuration - The configuration is created by the setup script during the installation of the Data Gateway.
- Tenant data - Information about every connected tenant.
If the Data Gateway is hosted on a virutal machine, either locally or in the cloud, it is recommended to create an backup of the entire virtual machine, including the operating system and all the data associated with it. For example, for a Data Gateway hosted on an AWS EC2 instance, you can create a schedule task to back up the Data Gateway as AMI using tools provided AWS.
During disaster recovery you can restore a Data Gatway from the backup image, for example, by launching a fresh EC2 instance from the lastest AMI.
Backup key configuration files
You can also selectively save these key configuration files if applicalbe.
- Docker compose file
- SumoLogic collector configuration file
Note: You need to keep a copy of the Docker compose file only if you have customised it for your Data Gateway. For example, you have changed the default value of and environment variable in the file. For the SumoLogic collector configuration, you should use the same
name to keep the Gateway logs be published to the same collector in the SumoLogic cloud.
For disaster recovery you can rebuild the Data Gateway by the following process.
- Install a new Data Gateway.
- Apply configuration files from your backup if there is any.
- Log into the GCIS deverloper portal and navigate to the
- Reconnect tenants that are connected to the old Data Gateway.
- Click the
Disconnect tenantbutton to disconnect.
- Click the
Connect tenantbutton to connect to the new Data Gateway.
- Click the
Loading the web page displays a security exception
The Data Gateway uses a self-signed certificate for encryption. The web browser can’t validate the authenticity of the certificate so it will stop you accessing it.
Add the certificate as trusted in your browser.
The web page fails to display
The Docker containers failed to start.
Check which Docker containers are running
docker ps -a
The following containers should be running
If any of these are stopped then the logs can be viewed by using the container name
docker logs gateway_app
Docker containers are running but the web page fails to display
This may be due to firewall ports not being open.
Internally Docker maps port 3000 on the container to the default HTTPS port 443.
Check that port 443 and 3000 has been enabled in any firewalls.
Error: [VAULT] unable to retrieve credentials ‘key’ must be specified in request body as JSON, or ‘reset’ set to true
The Data Gateway has initialised the Vault successfully but failed to save received credentials to the file vault/vault_cred.
Vault is used to securely store authentication tokens and API keys.
Make sure the Gateway is run as the correct user and has permission to write to the directory vault/vault_cred under the gatewayuser_GatewayData volume.
docker exec -t -i gateway_app id
Delete all files in the directory data/vault/file under the gatewayuser_VaultData volume, to allow the Gateway to re-initialise the vault.
Fail to start the Data Gateway service
The following services are not installed. - proc-sys-fs-binfmt_misc.mount - proc-sys-fs-binfmt_misc.automount
The above services are required by the Datadog docker container to collect system logs. In some cases, they might not be applicable, therefore, not installed in the Operating System.
Edit the file
Reload the updated service configuration.
Restart the Data Gateway service.