1. Setup

1.1. Requirements

DATAGERRY has the following system requirements:

  • Linux Operating System

  • MongoDB 4.2+

  • RabbitMQ

Although, DATAGERRY comes with an own webserver, we recomend Nginx as a reverse proxy for performance reasons.

There are several setup options for DATAGERRY, which are described in the sections below more in detail:

  • Docker Image

  • RPM file (for RHEL/CentOS distributions)

  • tar.gz archive with setup script (for Debian/Ubuntu or other distributions)

If you want to have a fast and easy start, use our Docker Image and docker-compose file.

1.2. Docker Image

The fastest way for getting started with DATAGERRY is using Docker. We provide a docker-compose file, which creates four containers (DATAGERRY, MongoDB, RabbitMQ, Nginx). All data were stored in MongoDB using Docker volumes on the Docker host system.

To start, copy the following docker-compose.yml in a directory of your Docker host.

version: "3.0"
services:
  nginx:
    image: nethinks/nginx:latest
    hostname: nginx
    ports:
    - "80:80"
    - "443:443"
    depends_on:
    - datagerry
    environment:
      NGINX_SSL_CERT: "/data/cert/cert.pem"
      NGINX_SSL_KEY: "/data/cert/key.pem"
      NGINX_LOCATION_DEFAULT: "/;/;http://datagerry:4000"
    restart: unless-stopped
    volumes:
      - ./cert:/data/cert

  datagerry:
    image: nethinks/datagerry:latest
    hostname: datagerry
    depends_on:
    - db
    - broker
    environment:
      DATAGERRY_Database_host: "db"
      DATAGERRY_MessageQueueing_host: "broker"
    restart: unless-stopped

  db:
    image: mongo:4.2-bionic
    hostname: db
    restart: unless-stopped
    volumes:
      - mongodb-data:/data/db
      - mongodb-config:/data/configdb
    
  broker:
    image: rabbitmq:3.8
    hostname: broker
    restart: unless-stopped
    volumes:
      - rabbitmq-data:/var/lib/rabbitmq

volumes:
  rabbitmq-data:
  mongodb-data:
  mongodb-config:

Create a subdirectory called cert with an SSL certificate (called cert.pem) and key (called key.pem). Your directory structure should look like this:

./docker-compose.yml
./cert/cert.pem
./cert/key.pem

If you don’t need SSL and just want to have a quick start, use the follwing docker-compose.yml:

version: "3.0"
services:
  nginx:
    image: nethinks/nginx:latest
    hostname: nginx
    ports:
    - "80:80"
    depends_on:
    - datagerry
    environment:
      NGINX_LOCATION_DEFAULT: "/;/;http://datagerry:4000"
    restart: unless-stopped

  datagerry:
    image: nethinks/datagerry:latest
    hostname: datagerry
    depends_on:
    - db
    - broker
    environment:
      DATAGERRY_Database_host: "db"
      DATAGERRY_MessageQueueing_host: "broker"
    restart: unless-stopped

  db:
    image: mongo:4.2-bionic
    hostname: db
    restart: unless-stopped
    volumes:
      - mongodb-data:/data/db
      - mongodb-config:/data/configdb
    
  broker:
    image: rabbitmq:3.8
    hostname: broker
    restart: unless-stopped
    volumes:
      - rabbitmq-data:/var/lib/rabbitmq

volumes:
  rabbitmq-data:
  mongodb-data:
  mongodb-config:

Now, run docker-compose to start the application:

$ docker-compose up -d

To access the DATAGERRY frontend, use the following parameters:

http://<<host> or https://<host>
user: admin
password: admin

We provide Docker images for every version of DATAGERRY in Docker Hub. You can use one of the following Docker tags:

latest

The latest tag is a symlink to the latest stable version of DATAGERRY. It is nice for a quickstart but will also do an automatic upgrade to the next major release if one is available. For production, we recommend the <release> tag below.

<release> (e.g. 1.0.1)

Every release has its own tag. So, if you want to use a specific version (or prevent your environment from automatic upgrades, just use a specific version as a Docker tag.

branches_<branch-name> (e.g. branches_development)

If you follow our development process and want to test a specific branch from our git repository, just use this tag. Please keep in mind, that these things aren’t released yet and a branch is a work-in-progress. So it can change at any time and break existing things. We don’t recommend to use these images in production.

To use a specific Docker tag, just replace the following line of the docker-compose.yml:

# replace this line
image: nethinks/datagerry:latest

# by the following:
image: nethinks/datagerry:<tagname>
# example:
image: nethinks/datagerry:1.0.2

1.3. RPM setup

For Red Hat Enterprise Linux (RHEL) or RHEL based systems like CentOS or Oracle Linux, we provide a RPM file for installing DATAGERRY.

The following RHEL/CentOS versions are supported and tested:

  • RHEL/CentOS 7

  • RHEL/CentOS 8

Before we can install DATAGERRY, we need to install the required dependencies MongoDB and Rabbit MQ.

1.3.1. Setup MongoDB

MongoDB 4.2+ is required as database for DATAGERRY.

Note

The setup of MongoDB is described in detail on the MongoDB website: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-red-hat/ The following section is a quick install guide of MonogDB.

To setup MongoDB, place the follwing file under /etc/yum.repos.d/mongodb.repo:

[MongoDB]
name=MongoDB Repository
baseurl=http://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc

After that, install the mongodb-org package and start the server with SystemD:

$ sudo yum install -y mongodb-org
$ sudo systemctl enable mongod
$ sudo systemctl start mongod

1.3.2. Setup RabbitMQ

RabbitMQ 3.8+ is used as messaging bus between the processes of DATAGERRY.

Note

The setup of RabbitMQ is described in detail on the RabbitMQ website: https://www.rabbitmq.com/install-rpm.html The following section is a quick install guide of RabbitMQ

For setting up RabbitMQ, we can use the RPM repository provided by Bintray. Place the following file under /etc/yum.repos.d/rabbitmq.repo:

[bintray-rabbitmq-server]
name=bintray-rabbitmq-rpm
baseurl=https://dl.bintray.com/rabbitmq/rpm/rabbitmq-server/v3.8.x/el/$releasever/
gpgcheck=1
gpgkey=https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
enabled=1

[bintraybintray-rabbitmq-erlang-rpm]
name=bintray-rabbitmq-erlang-rpm
baseurl=https://dl.bintray.com/rabbitmq-erlang/rpm/erlang/22/el/$releasever/
gpgcheck=0
repo_gpgcheck=0
enabled=1

Now, RabbitMQ can be installed and started:

$ sudo yum install -y rabbitmq-server
$ sudo systemctl enable rabbitmq-server
$ sudo systemctl start rabbitmq-server

1.3.3. Setup DATAGERRY

If all requirements were installed, we’ll can install the downloaded DATAGERRY RPM file:

$ sudo rpm -ivh DATAGERRY-<version>.x86_64.rpm

To change the parameters for connecting to MongoDB and RabbitMQ, edit the configuration file /etc/datagerry/cmdb.conf

After that, activate and start DATAGERRY with Systemd:

$ sudo systemctl enable datagerry.service
$ sudo systemctl start datagerry.service

To access the DATAGERRY frontend, use the following parameters:

http://<<host>:4000
user: admin
password: admin

Note

If you can’t access the webfrontend of DATAGERRY, check the firewall settings of your server. Port 4000 should ba accessible.

1.4. tar.gz archive setup

For all non rpm based Linux distributions, we provide a tar.gz archive with a setup shell script. Systemd is a requirement for that setup. This should work on most distributions, and is tested with the following distributions:

  • Ubuntu 18.04

Before we can install DATAGERRY, we need to install the required dependencies MongoDB and Rabbit MQ.

1.4.1. Setup MongoDB

MongoDB 4.2+ is required as database for DATAGERRY.

Please follow the offical MongoDB documentation <https://docs.mongodb.com/manual/administration/install-on-linux/> to setup MongoDB for your distribution.

1.4.2. Setup RabbitMQ

RabbitMQ 3.8+ is used as messaging bus between the processes of DATAGERRY.

Please follow the offical RabbitMQ documentation <https://www.rabbitmq.com/download.html#installation-guides> to setup RabbitMQ for your distribution.

1.4.3. Setup DATAGERRY

Extract the provided tar.gz archive and execute the setup script as root:

$ tar -xzvf datagerry-<version>.tar.gz
$ cd datagerry
$ sudo ./setup.sh

To change the parameters for connecting to MongoDB and RabbitMQ, edit the configuration file /etc/datagerry/cmdb.conf

After that, activate and start DATAGERRY with Systemd:

$ sudo systemctl enable datagerry.service
$ sudo systemctl start datagerry.service

To access the DATAGERRY frontend, use the following parameters:

http://<<host>:4000
user: admin
password: admin

Note

If you can’t access the webfrontend of DATAGERRY, check the firewall settings of your server. Port 4000 should ba accessible.

1.5. Setting up Nginx as reverse proxy

We recomend to run Nginx as reverse proxy for DATAGERRY. After installing Nginx for your platform, you can adapt the following example configuration for Nginx (nginx.conf):

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events
{
    worker_connections  1024;
}


http
{
    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
    log_format  main    '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
    access_log          /var/log/nginx/access.log  main;
    sendfile            on;
    keepalive_timeout   65;
    client_max_body_size 0;
    
    server
    {
        listen          80;
        server_name     localhost;
        # HTTPS redirect
        return 301 https://$host$request_uri;
    }
    

    server
    {
        
        listen              443 ssl;
        ssl_certificate     /data/cert/cert.pem;
        ssl_certificate_key /data/cert/key.pem;
        
        server_name         localhost;

        location /
        {
            proxy_set_header        Host $host;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header        X-Forwarded-Proto $scheme;
            set $upstream_host http://127.0.0.1:4000;
                
            rewrite ^/(.*) /$1 break;
                
            proxy_pass $upstream_host;
        }

    }

}

This will Nginx listen on port 80 (HTTP) and 443 (HTTPS) and create a redirect from HTTP to HTTPS. If someone access https://<host>/, Nginx will contact http://127.0.0.1:4000, where DATAGERRY is listening.

1.6. Update DATAGERRY

To update DATAGERRY to a new version, simply install the new version and start DATAGERRY. During the first start, DATAGERRY will detect the version of the existing database and apply required updates. This may take a few seconds or minutes. After that, the application will be started.

2. Configuration — DATAGERRY experimental-redesigning-the-navigation-bar-50958ec2f9d4b31f083813e9ddd18e253f4b02a2 documentation

2. Configuration

Most of DATAGERRY’s configuration is stored in MongoDB. There are a view parameters which cannot be stored at this place (e.g. database connection to MongoDB itself). For this parameters, we provide an INI style configuration file, called cmdb.conf. Please see the following example:

[Database]
host = 127.0.0.1
port = 27017
database_name = cmdb
;username = username
;password = password

[WebServer]
host = 0.0.0.0
port = 4000

[MessageQueueing]
host = 127.0.0.1
port = 5672
username = guest
password = guest
exchange = datagerry.eventbus
connection_attempts = 2
retry_delay = 6
use_tls = False

It is possible to overwrite settings in the configuration file with OS environment variables. Please see the following example:

DATAGERRY_<section_name>_<option_name>
DATAGERRY_Database_port=27018

This feature is especially useful, if you want to run DATAGERRY in Docker.