Gentrack Integration Resources

Installing the Data Gateway

On this page

Data Gateway acts as a communication broker for the Core system, and Gentrack Cloud Integration Services running as cloud service. It is multi-tenanted so that one Data Gateway can be used for all your environments including Development, Test, and Production. However, it is highly recommended having separate Data Gateways for Production and non-Production instances.

Who should install Data Gateway?

  • Self-hosted Core system - Data Gateway is required.
  • Gentrack-hosted Core system - Data Gateway is NOT required.

System requirements

The Data Gateway uses Docker which can be run on a physical machine, virtual machine, or container service like Amazon ECS or Azure AKS.

  • Operating System and kernel that is supported by Docker. CentOS/Red Hat Enterprise Linux based Linux (RHEL) is recommended.
  • 2 vCPU (4 vCPU or more is recommended).
  • 4 GB of RAM (8 GB or more is recommended).
  • 30 GB of Hard Disk.
  • A static IP address.

Before you begin

Before installing Data Gateway, ensure that you:

  • Confirm that your Data Gateway server meets the minimal system requirements
  • Have administrator (sudo) access to the server
  • Know the hostname of the server

Installation

The instructions and scripts to install Data Gateway provided in this section are for CentOS/Red Hat Enterprise Linux (RHEL) based Linux.

To install and set up Data Gateway, complete the following tasks:

Set up a Docker Hub user

Data Gateway is a set of containerised applications that is deployed using Docker. The Docker images are hosted in a private Gentrack Docker Hub repository.

To get ready to install Data Gateway using Docker:

  1. Create your own Docker Hub user account
  2. Provide your Docker Hub username to Gentrack using a dedicated support channel

The Gentrack support team will give that user read permissions to the repository.

The management of passwords and credentials for that user is your responsibility. If you wish to change the Docker Hub user, please contact us directly. If the Docker Hub password is ever exposed, you must immediately reset the password and then contact us.

Set up Docker

Docker compose is used for container orchestration and management. Docker is managed as a non-root user and must automatically start on boot in the case of server restart.

Docker installation instructions to achieve this can be found in the following Docker Docs:

Set up a Data Gateway user

A new user must be created to run the Data Gateway. These instructions assume the name of the user will be gatewayuser.

  1. Add a user named gatewayuser, add home directory for the user, add it to sudoer (wheel) and docker group, and set a password for gatewayuser.

    sudo adduser gatewayuser -m --system -G wheel,docker \
  && sudo passwd gatewayuser

  2. Temporarily switch to gatewayuser and login into Docker Hub. You will need to use the Docker Hub credentials previously provided to Gentrack.

    sudo su - gatewayuser -c "docker login"

Install Data Gateway

  1. Download the following files and copy them into the user home directory, add execution permission and switch to gatewayuser.

    sudo su - gatewayuser -c "curl https://raw.githubusercontent.com/Gentrack/gcis-setup/master/setup-gateway.sh -o setup-gateway.sh \
  && chmod +x setup-gateway.sh"

    sudo su - gatewayuser

  2. While being gatewayuser, executing the installation script will display a user interface with fields. Follow the on screen prompts to navigate.

    sudo ~/setup-gateway.sh

    Fields:

    ...
Enter data sovereignty region AU or UK *
Enter Gateway Host Name - Fully qualified domain name of the Data Gateway server. This will be used for certificates that are automatically generated *
...

    To exit gatewayuser, type the following:

    exit

    The bash script setup-gateway.sh will create the following:

    • setup-gateway-logs.txt, setup-gateway-variables.txt and docker-compose.yml
    • A directory with Data Gateway configuration files under the home directory of gatewayuser
    • Security encryption keys and certificates
    • Docker volumes that will be used to store persistent data
    • Daily jobs to remove old application logs and Docker images
    • System service so that Data Gateway will automatically start or restart after system maintenance or reboot
    • Unique credentials for the Data Gateway application to connect to the RabbitMQ instance

    Note: if a Data Gateway has already been installed, the installation script will remove the existing Data Gateway along with all previously configured tenants, and perform a fresh installation on the server. Also, the installation script will select a correct Docker image tag for your Data Gateway, based on your region indicated by the Integration Services URL.

Test your installation

  1. Check if the Data Gateway service is running, it should show the service is active.

    sudo systemctl status gentrack-gateway-docker

  2. Navigate to the Data Gateway URL, for example https://uk-energy-gateway, and you should see the Data Gateway landing page.

Install Custom Certificate

By completing the above installation, the Data Gateway app server uses a self-signed certificate by default. If you want to use a certificate signed by a trusted certificate authority (CA), there are two ways to install it:

Option 1: Use an API Gateway or Application Load Balancer

If Data Gateway is deployed in AWS EC2 instance, you can use an Application Load Balancer or API Gateway in front of the Data Gateway. Then import your SSL certificate to AWS Certificate manager. Configure your API Gateway or Application Load Balancer to use the certificate. Please refer to following AWS documentation:

Option 2: Import a certificate using the certificate import tool

You can also import your own SSL certificate to the Data Gateway, by the following steps:

  • Step 1. Prepare a PKCS12 certificate. The certificate import tool supports only PKCS#12 certificate format (also known as PKCS12 or PFX). You can use openssl to create a PKCS#12 file from a private key, a SSL certificate, and additional certificates for creating a complete trust chain. It is important to include a complete certificate trust chain in the PKCS#12 file to allow a client to verify the Data Gateway’s SSL certificate correctly.
  openssl pkcs12 -export -out <Output p12 file> -inkey <Private key PEM> -in <Certificate PEM> -certfile <Certificate chain PEM> -password pass:<Password>
  • Step 2. Copy the PKCS#12 certificate to the config folder of the gateway_app Docker container. Assuming the certificate path is /etc/ssl/certs/cert.p12, run the following command:
sudo cp /etc/ssl/certs/cert.p12 /var/lib/docker/volumes/platform_GatewayConfig/_data/cert.p12

# copy your private key to the config folder if your certificate's key type is ECDSA
sudo cp /etc/ssl/certs/privkey.pem /var/lib/docker/volumes/platform_GatewayConfig/_data/privkey.pem
  • Step 3. Use the certificate import tool to import the certificate by the following command. The command will prompt you to enter the password that you set when creating the PKCS12 file.
# certificate private key is RSA
docker exec -it gateway_app node src/import-cert -c ./config/cert.p12 -f true 

# certificate private key is ECDSA
docker exec -it gateway_app node src/import-cert-ecdsa -c ./config/cert.p12 -k ./config/privkey.pem -f true
  • Step 4. Remove the PKCS#12 certificate from the config folder of the gateway_app Docker container.
sudo rm /var/lib/docker/volumes/platform_GatewayConfig/_data/cert.p12
  • Step 5. Restart the Data Gateway service.
sudo systemctl restart gentrack-gateway-docker

Enable TLS Certificates for Core System

When connecting to the core system using HTTPS, e.g., calling a Velocity REST API, the Data Gateway verifies server’s certificate as part of SSL/TLS handshake. It is recommended to use trusted CA signed certificates for the core system.

If the server certificate is signed from an unknown CA, e.g. self-signed or privately signed certificates, the Data Gateway will refuse the connection.

If you want to allow the use of self-signed or private CA signed certificate, please follow below guidance to enable it.

If you are using a private CA certificate, you can import the root CA to the Data Gateway by:

  1. Use the update-ca-certificates program of the ca-certificates utility to import additional CA certificates. On different Linux distributions, the process is slightly different. See here.

  2. While being gatewayuser, run the setup-gateway.sh to mount CA certificates to the gateway_app Docker container

    sudo ~/setup-gateway.sh

    Fields:

    Will you be using a private CA signed certificiate yes or no
  Enter path CA signed .pem file (*.pem)
  Enter node_extra_ca_certs path (*.crt)
    • Locate the destination file generated by the update-ca-certificates program, which will contain the additional CA certificates. For example, on CentOS 7, the destination file is /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.
    • When prompted, enter the CA certificate path. E.g. /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
    • When prompted, enter the NODE_EXTRA_CA_CERTS path. E.g. /etc/ssl/certs/ca-bundle.crt

    This will update the following in docker-compose.yml:

    version: "3.1"
services:
app:
...
    container_name: gateway_app
    volumes:
      # mount the CA bundle from host to docker.
      - /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/etc/ssl/certs/ca-bundle.crt:ro
      ....
    environment:
      # instruct NodeJs to use the imported CA bundle
      - NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-bundle.crt
...

  3. Restart the Data Gateway

    sudo systemctl restart gentrack-gateway-docker

  1. While being gatewayuser, run the setup-gateway.sh.

    sudo ~/setup-gateway.sh

    Field

    Will you be using a self signed certificate in Core systems

    This will update the following in docker-compose.yml:

    version: "3.1"
services:
app:
...
  container_name: gateway_app
  ...
  environment:
    - NODE_TLS_REJECT_UNAUTHORIZED=0
...

  2. Restart the Data Gateway

    sudo systemctl restart gentrack-gateway-docker

Useful commands

  • Start/Stop/Restart Data Gateway using Systemd/Linux

    sudo systemctl start gentrack-gateway-docker
sudo systemctl stop gentrack-gateway-docker
sudo systemctl restart gentrack-gateway-docker

  • Start/Stop Data Gateway manually

    /usr/local/bin/docker-compose -f /home/gatewayuser/platform/docker-compose.yml up -d
/usr/local/bin/docker-compose -f /home/gatewayuser/platform/docker-compose.yml down

  • Check the running user of the gateway_app Docker container

    docker exec -t -i gateway_app id

  • List and inspect volumes

    docker volume ls
docker volume inspect platform_GatewayData

  • List and inspect containers

    docker container ls
docker container inspect gateway_app

Set up Firewall security

The following are the Gentrack Cloud Integration Services IP addresses that need to be allowed to connect Data Gateway:

  • Asia Pacific region:
    • 52.65.53.1
    • 13.236.220.175
  • European region:
    • 35.176.213.75
    • 18.130.141.102

Data Gateway port-based & protocol-based Firewall rules

DirectionProtocol and PortRestricted to IP or ServiceReason
InboundHTTPS - 443/tcp
  • IP address of the Core systems
  • Gentrack Cloud Integration Services IP addresses
  • Allows API events to be sent from the Core systems
  • Allows Gentrack Cloud Integration Services API access to the Core systems
Outbound

Velocity - 443/tcp

Junifer - 43002/tcp

These are the default ports but may differ

  • IP address of the Core systems
  • Allows Webservice API on the Core systems to be invoked by the Data Gateway
OutboundHTTPS/443
  • Docker Hub
  • Datadog
  • Sumo Logic
  • Operating System update servers and repositories
  • Amazon Web Services (eu-west-2 or ap-southeast-2)
  • Used to download and update Data Gateway
  • Availability and performance metrics
  • Application log monitoring
  • Receive updates and security patches for the server
  • Connect to Gentrack Cloud Integration Services
OutboundDNS
  • Must be able to resolve domain names
OutboundDHCP
  • IP address assignment
OutboundNTP
  • Access to get time from global time servers

Set up monitoring services

Operational monitoring of the Data Gateway is important. For the Gentrack support team to assist with issues, we recommend enabling the following monitoring services. The examples below do not include proxy server settings. Additional configuration steps may be needed if using a web proxy.

Sumo Logic

Sumo Logic is used to collect application logs from Docker. A collection agent is run as a Docker container and will send logs back to Sumo Logic.

  1. Download the following file and copy it into the /home/gatewayuser/platform directory.

    sudo su - gatewayuser

    cd platform \
  && sudo curl https://raw.githubusercontent.com/Gentrack/gcis-setup/master/custom-sumo-sources.json -o custom-sumo-sources.json \
  && cd -

  2. While being gatewayuser, run the setup-gateway.sh.

      sudo ~/setup-gateway.sh

    Fields:

    Enable Sumologic collectors yes or no
  Enter Sumologic Access Id
  Enter Sumologic Access Key
  Enter Sumologic Collector Name

    Field Notes:

    • SUMO_ACCESS_ID - Access identifier for the collector. This will be setup in Sumo Logic first. This may be provided by Gentrack if we are going to monitor this for you, or you can use your own Sumo Logic collector.
    • SUMO_ACCESS_KEY - Access key for the collector. This will be setup in Sumo Logic first. This may be provided by Gentrack if we are going to monitor this for you, or you can use your own Sumo Logic collector.
    • SUMO_COLLECTOR_NAME - Name that will be used to unique identify the Data Gateway. The name has to include the prefix GCIS Data Gateway - and it’s suggested to use your company name followed by the prefix. e.g. GCIS Data Gateway - Energise
    • SUMO_COLLECTOR_NAME_PREFIX - The prefix of collector name. Do not add any value. Keep it empty without any quotation marks for overriding the default “collector_container-” prefix.
    • SUMO_SOURCES_JSON - The file path of collector’s source setting which should match the volumes setting. No change required.

    This will uncomment and update the following in docker-compose.yml:

    sumologic:
    image: sumologic/collector:latest
    container_name: gateway_sumologic
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ~/platform/custom-sumo-sources.json:/etc/custom-sumo-sources.json
    environment:
      - SUMO_ACCESS_ID=[insert API ID here]
      - SUMO_ACCESS_KEY=[insert API key here]
      - SUMO_COLLECTOR_NAME=GCIS Data Gateway - [insert name here]
      - SUMO_COLLECTOR_NAME_PREFIX=
      - SUMO_SOURCES_JSON=/etc/custom-sumo-sources.json

  3. Restart the Data Gateway

    sudo systemctl restart gentrack-gateway-docker

Datadog

Datadog is used to capture performance metrics and system availability. An agent is run as a Docker container and will send information back to Datadog. Using Datadog, alerts can be triggered if Data Gateway is not running.

  1. While being gatewayuser, run the setup-gateway.sh.

      sudo ~/setup-gateway.sh

    Fields:

    Enable Datadog yes or no
  Enter Datadog API Key
  Enter Datadog Hostname

    Field Notes:

    • DD_API_KEY - This may be provided by Gentrack if we are going to monitor this for you, or you can use your own API key.
    • DD_HOSTNAME - Name that will be used to unique identify the Data Gateway. This is needed because the API key will be shared by multiple agents. You may want to use the hostname of the server. If omitted, a randomly generated name may be used which will change each time Docker is restarted.

    This will uncomment and update the following in docker-compose.yml:

      datadog:
    image: datadog/agent:7
    container_name: gateway_datadog
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /proc/:/host/proc/
      - /sys/fs/cgroup/:/host/sys/fs/cgroup
      - ~/platform/http_check.yaml:/conf.d/http_check.yaml
    environment:
      - DD_API_KEY=[insert API key here]
      - DD_SITE=datadoghq.eu
      - DD_HOSTNAME=[insert name here]

  2. Restart the Data Gateway service.

    sudo systemctl restart gentrack-gateway-docker

Updates to Data Gateway

The Data Gateway uses watchtower to auto-update itself by regularly checking Docker Hub for new images. When a new version is found, it will pull the new images, stop the Data Gateway, and then start the Data Gateway using the new images again. It uses the same Docker Hub credentials for the Data Gateway docker user.

If you run the Data Gateway on a managed Docker infrastructure, please check for new versions of Data Gateway regularly. If your Data Gateway’s version is more than 60 days behind the latest release, you will be reminded via email. Any issues associated with the Data Gateway functionality is considered part of our service availability.

Data Gateway Backup and Restore

The Data Gateway stores these types of data:

  • Application data - Event data stored while the Data Gateway processing core API events. Application data is time-sensitive and there is very little value of backup.
  • Gateway configuration - The configuration is created by the setup script during the installation of the Data Gateway.
  • Tenant data - Information about every connected tenant.

Image-based backup

If the Data Gateway is hosted on a virtual machine, either locally or in the cloud, it is recommended to create a backup of the entire virtual machine, including the operating system and all the data associated with it. For example, for a Data Gateway hosted on an AWS EC2 instance, you can create a schedule task to back up the Data Gateway as AMI using tools provided AWS.

During disaster recovery, you can restore a Data Gateway from the backup image, for example, by launching a fresh EC2 instance from the latest AMI.

Backup key configuration files

You can also selectively save these key configuration files if applicable.

  • Docker compose file /home/gatewayuser/platform/docker-compose.yaml
  • SumoLogic collector configuration file /opt/SumoCollector/config/user.properties

Note: You need to keep a copy of the Docker compose file only if you have customised it for your Data Gateway. For example, you have changed the default value of and environment variable in the file. For the SumoLogic collector configuration, you should use the same name to keep the Gateway logs be published to the same collector in the SumoLogic cloud.

For disaster recovery you can rebuild the Data Gateway by the following process.

  • Install a new Data Gateway.
  • Apply configuration files from your backup if there is any.
  • Log into the GCIS Portal and navigate to the Tenants page.
  • Select and reconnect tenants that are connected to the old Data Gateway.
    • Click on Disconnect button to disconnect.
    • Click on Connect button to connect to the new Data Gateway.

Troubleshooting

Loading the web page displays a security exception

Cause

The Data Gateway uses a self-signed certificate for encryption. The web browser can’t validate the authenticity of the certificate, so it will stop you accessing it.

Resolution

Add the certificate as trusted in your browser.

The web page fails to display

Cause

The Docker containers failed to start.

Resolution

Check which Docker containers are running

docker ps -a

The following containers should be running

  • gateway_app
  • gateway_rabbitmq
  • gateway_vault
  • gateway_redis
  • gateway_watchtower

If any of these are stopped then the logs can be viewed by using the container name

docker logs gateway_app

Docker containers are running, but the web page fails to display

Cause

This may be due to firewall ports not being open.

Resolution

Internally Docker maps port 3000 on the container to the default HTTPS port 443.

Check that port 443 and 3000 has been enabled in any firewalls.

Error: [VAULT] unable to retrieve credentials ‘key’ must be specified in request body as JSON, or ‘reset’ set to true

Cause

The Data Gateway has initialised the Vault successfully but failed to save received credentials to the file vault/vault_cred.

Vault is used to securely store authentication tokens and API keys.

Resolution

  1. Make sure the Gateway is run as the correct user and has permission to write to the directory vault/vault_cred under the gatewayuser_GatewayData volume.

    docker exec -t -i gateway_app id

  2. Delete all files in the directory data/vault/file under the gatewayuser_Vault Data volume, to allow the Gateway to re-initialise the vault.

Fail to start the Data Gateway service

Cause

The following services are not installed.

  • proc-sys-fs-binfmt_misc.mount
  • proc-sys-fs-binfmt_misc.automount

Resolution

The above services are required by the Datadog Docker container to collect system logs. In some cases, they might not be applicable, therefore, not installed in the Operating System.

  1. Edit the file /etc/systemd/system/gentrack-gateway-docker.service to remove proc-sys-fs-binfmt_misc.mount and proc-sys-fs-binfmt_misc.automount in both After and Requires units.

    After=docker.service
Requires=docker.service

  2. Reload the updated service configuration.

    systemctl daemon-reload

  3. Restart the Data Gateway service.

    sudo systemctl restart gentrack-gateway-docker
Last updated on 5 Feb 2024