1. Setup

1.1. Requirements

DATAGERRY has the following system requirements:

  • Linux Operating System

  • MongoDB 4.2+

  • RabbitMQ

Although, DATAGERRY comes with an own webserver, we recomend Nginx as a reverse proxy for performance reasons.

There are several setup options for DATAGERRY, which are described in the sections below more in detail:

  • Docker Image

  • RPM file (for RHEL/CentOS distributions)

  • tar.gz archive with setup script (for Debian/Ubuntu or other distributions)

If you want to have a fast and easy start, use our Docker Image and docker-compose file.

1.2. Docker Image

The fastest way for getting started with DATAGERRY is using Docker. We provide a docker-compose file, which creates four containers (DATAGERRY, MongoDB, RabbitMQ, Nginx). All data were stored in MongoDB using Docker volumes on the Docker host system.

To start, copy the following docker-compose.yml in a directory of your Docker host.

version: "3.0"
services:
  nginx:
    image: nethinks/nginx:latest
    hostname: nginx
    ports:
    - "80:80"
    - "443:443"
    depends_on:
    - datagerry
    environment:
      NGINX_SSL_CERT: "/data/cert/cert.pem"
      NGINX_SSL_KEY: "/data/cert/key.pem"
      NGINX_LOCATION_DEFAULT: "/;/;http://datagerry:4000"
    restart: unless-stopped
    volumes:
      - ./cert:/data/cert

  datagerry:
    image: nethinks/datagerry:latest
    hostname: datagerry
    depends_on:
    - db
    - broker
    environment:
      DATAGERRY_Database_host: "db"
      DATAGERRY_MessageQueueing_host: "broker"
    restart: unless-stopped

  db:
    image: mongo:4.2-bionic
    hostname: db
    restart: unless-stopped
    volumes:
      - mongodb-data:/data/db
      - mongodb-config:/data/configdb
    
  broker:
    image: rabbitmq:3.8
    hostname: broker
    restart: unless-stopped
    volumes:
      - rabbitmq-data:/var/lib/rabbitmq

volumes:
  rabbitmq-data:
  mongodb-data:
  mongodb-config:

Create a subdirectory called cert with an SSL certificate (called cert.pem) and key (called key.pem). Your directory structure should look like this:

./docker-compose.yml
./cert/cert.pem
./cert/key.pem

If you don’t need SSL and just want to have a quick start, use the follwing docker-compose.yml:

version: "3.0"
services:
  nginx:
    image: nethinks/nginx:latest
    hostname: nginx
    ports:
    - "80:80"
    depends_on:
    - datagerry
    environment:
      NGINX_LOCATION_DEFAULT: "/;/;http://datagerry:4000"
    restart: unless-stopped

  datagerry:
    image: nethinks/datagerry:latest
    hostname: datagerry
    depends_on:
    - db
    - broker
    environment:
      DATAGERRY_Database_host: "db"
      DATAGERRY_MessageQueueing_host: "broker"
    restart: unless-stopped

  db:
    image: mongo:4.2-bionic
    hostname: db
    restart: unless-stopped
    volumes:
      - mongodb-data:/data/db
      - mongodb-config:/data/configdb
    
  broker:
    image: rabbitmq:3.8
    hostname: broker
    restart: unless-stopped
    volumes:
      - rabbitmq-data:/var/lib/rabbitmq

volumes:
  rabbitmq-data:
  mongodb-data:
  mongodb-config:

Now, run docker-compose to start the application:

$ docker-compose up -d

To access the DATAGERRY frontend, use the following parameters:

http://<<host> or https://<host>
user: admin
password: admin

We provide Docker images for every version of DATAGERRY in Docker Hub. You can use one of the following Docker tags:

latest

The latest tag is a symlink to the latest stable version of DATAGERRY. It is nice for a quickstart but will also do an automatic upgrade to the next major release if one is available. For production, we recommend the <release> tag below.

<release> (e.g. 1.0.1)

Every release has its own tag. So, if you want to use a specific version (or prevent your environment from automatic upgrades, just use a specific version as a Docker tag.

branches_<branch-name> (e.g. branches_development)

If you follow our development process and want to test a specific branch from our git repository, just use this tag. Please keep in mind, that these things aren’t released yet and a branch is a work-in-progress. So it can change at any time and break existing things. We don’t recommend to use these images in production.

To use a specific Docker tag, just replace the following line of the docker-compose.yml:

# replace this line
image: nethinks/datagerry:latest

# by the following:
image: nethinks/datagerry:<tagname>
# example:
image: nethinks/datagerry:1.0.2

1.3. RPM setup

For Red Hat Enterprise Linux (RHEL) or RHEL based systems like CentOS or Oracle Linux, we provide a RPM file for installing DATAGERRY.

The following RHEL/CentOS versions are supported and tested:

  • RHEL/CentOS 7

  • RHEL/CentOS 8

Before we can install DATAGERRY, we need to install the required dependencies MongoDB and Rabbit MQ.

1.3.1. Setup MongoDB

MongoDB 4.2+ is required as database for DATAGERRY.

Note

The setup of MongoDB is described in detail on the MongoDB website: https://docs.mongodb.com/manual/tutorial/install-mongodb-on-red-hat/ The following section is a quick install guide of MonogDB.

To setup MongoDB, place the follwing file under /etc/yum.repos.d/mongodb.repo:

[MongoDB]
name=MongoDB Repository
baseurl=http://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.2/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.2.asc

After that, install the mongodb-org package and start the server with SystemD:

$ sudo yum install -y mongodb-org
$ sudo systemctl enable mongod
$ sudo systemctl start mongod

1.3.2. Setup RabbitMQ

RabbitMQ 3.8+ is used as messaging bus between the processes of DATAGERRY.

Note

The setup of RabbitMQ is described in detail on the RabbitMQ website: https://www.rabbitmq.com/install-rpm.html The following section is a quick install guide of RabbitMQ

For setting up RabbitMQ, we can use the RPM repository provided by Bintray. Place the following file under /etc/yum.repos.d/rabbitmq.repo:

[bintray-rabbitmq-server]
name=bintray-rabbitmq-rpm
baseurl=https://dl.bintray.com/rabbitmq/rpm/rabbitmq-server/v3.8.x/el/$releasever/
gpgcheck=1
gpgkey=https://github.com/rabbitmq/signing-keys/releases/download/2.0/rabbitmq-release-signing-key.asc
enabled=1

[bintraybintray-rabbitmq-erlang-rpm]
name=bintray-rabbitmq-erlang-rpm
baseurl=https://dl.bintray.com/rabbitmq-erlang/rpm/erlang/22/el/$releasever/
gpgcheck=0
repo_gpgcheck=0
enabled=1

Now, RabbitMQ can be installed and started:

$ sudo yum install -y rabbitmq-server
$ sudo systemctl enable rabbitmq-server
$ sudo systemctl start rabbitmq-server

1.3.3. Setup DATAGERRY

If all requirements were installed, we’ll can install the downloaded DATAGERRY RPM file:

$ sudo rpm -ivh DATAGERRY-<version>.x86_64.rpm

To change the parameters for connecting to MongoDB and RabbitMQ, edit the configuration file /etc/datagerry/cmdb.conf

After that, activate and start DATAGERRY with Systemd:

$ sudo systemctl enable datagerry.service
$ sudo systemctl start datagerry.service

To access the DATAGERRY frontend, use the following parameters:

http://<<host>:4000
user: admin
password: admin

Note

If you can’t access the webfrontend of DATAGERRY, check the firewall settings of your server. Port 4000 should ba accessible.

1.4. tar.gz archive setup

For all non rpm based Linux distributions, we provide a tar.gz archive with a setup shell script. Systemd is a requirement for that setup. This should work on most distributions, and is tested with the following distributions:

  • Ubuntu 18.04

Before we can install DATAGERRY, we need to install the required dependencies MongoDB and Rabbit MQ.

1.4.1. Setup MongoDB

MongoDB 4.2+ is required as database for DATAGERRY.

Please follow the offical MongoDB documentation <https://docs.mongodb.com/manual/administration/install-on-linux/> to setup MongoDB for your distribution.

1.4.2. Setup RabbitMQ

RabbitMQ 3.8+ is used as messaging bus between the processes of DATAGERRY.

Please follow the offical RabbitMQ documentation <https://www.rabbitmq.com/download.html#installation-guides> to setup RabbitMQ for your distribution.

1.4.3. Setup DATAGERRY

Extract the provided tar.gz archive and execute the setup script as root:

$ tar -xzvf datagerry-<version>.tar.gz
$ cd datagerry
$ sudo ./setup.sh

To change the parameters for connecting to MongoDB and RabbitMQ, edit the configuration file /etc/datagerry/cmdb.conf

After that, activate and start DATAGERRY with Systemd:

$ sudo systemctl enable datagerry.service
$ sudo systemctl start datagerry.service

To access the DATAGERRY frontend, use the following parameters:

http://<<host>:4000
user: admin
password: admin

Note

If you can’t access the webfrontend of DATAGERRY, check the firewall settings of your server. Port 4000 should ba accessible.

1.5. Setting up Nginx as reverse proxy

We recomend to run Nginx as reverse proxy for DATAGERRY. After installing Nginx for your platform, you can adapt the following example configuration for Nginx (nginx.conf):

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events
{
    worker_connections  1024;
}


http
{
    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
    log_format  main    '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
    access_log          /var/log/nginx/access.log  main;
    sendfile            on;
    keepalive_timeout   65;
    client_max_body_size 0;
    
    server
    {
        listen          80;
        server_name     localhost;
        # HTTPS redirect
        return 301 https://$host$request_uri;
    }
    

    server
    {
        
        listen              443 ssl;
        ssl_certificate     /data/cert/cert.pem;
        ssl_certificate_key /data/cert/key.pem;
        
        server_name         localhost;

        location /
        {
            proxy_set_header        Host $host;
            proxy_set_header        X-Real-IP $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header        X-Forwarded-Proto $scheme;
            set $upstream_host http://127.0.0.1:4000;
                
            rewrite ^/(.*) /$1 break;
                
            proxy_pass $upstream_host;
        }

    }

}

This will Nginx listen on port 80 (HTTP) and 443 (HTTPS) and create a redirect from HTTP to HTTPS. If someone access https://<host>/, Nginx will contact http://127.0.0.1:4000, where DATAGERRY is listening.

1.6. Update DATAGERRY

To update DATAGERRY to a new version, simply install the new version and start DATAGERRY. During the first start, DATAGERRY will detect the version of the existing database and apply required updates. This may take a few seconds or minutes. After that, the application will be started.

8. Authentication — DATAGERRY bugfix-NET-967_typo-error-systemd-conf-f632f6989cd00ee2e4747d3c63a845e5017f0dd4 documentation

8. Authentication

DATAGERRY uses a hybrid authentication method for issuing access tokens. These tokens can be used to authenticate against the Rest API.

When logging in, the user is searched in the database using his user name. If a user is found, his stored provider is requested. Depending on the provider implementation, an authentication is now attempted using the submitted password. If the authentication is successful, the provider returns a user instance which is used to generate a valid access token. If the authentication fails, the login is aborted.

If no user is found in the database, each provider is called according to the provider order. If a successful authentication takes place, a new user is created with the submitted login data and stored in a predefined group. If no provider confirms a successful request, the login is aborted.

8.1. Access Token

Authentication uses JSON Web Tokens as specified in RFC 7519 for identity verification. The Issuer, Issued at, Expiration time and the custom DATAGERRY claims are used.

Warning

The asymmetrical RSA key for signing the tokens is stored in the database under settings.conf.

8.2. Providers

Providers are authenticators who can locate a user in the system using a username and password. They are divided into internal and external providers. In the case of internal providers, the system searches for the user in the database. External providers identify the user in an external third party system.

Preinstalled providers are:

  1. LocalAuthenticationProvider - Searches for users in the database by username and compares the password with the stored SHA256 HMAC.

  2. LdapAuthenticationProvider - Using the user name, tries to find a user in the directory service and authenticate with the password.

Note

The order in which the providers are queried is determined by the installation order of the providers in the authentication module. A special factor here is that the local provider is always in first place.

8.2.1. LDAP group mapping

The LDAP authentication provider offers the possibility to create a mapping to the datagerry groups based on the LDAP groups. Since Datagerry does not allow users to be assigned to multiple groups, the possibility of multiple LDAP groups must be reduced.

The mapping of groups must first be activated manually. If mapping is disabled, all LDAP users are assigned to the default group. This is also the case if the mapping is deactivated afterwards.

../_images/auth_provider_ldap_default.png

After activating the mapping, a search filter can be created for selecting the groups at login. In the configuration interface you can now assign a group name of the LDAP to a datagerry group. Here an LDAP group can be assigned exactly to one datagerry group, however different LDAP groups can be assigned to the same datagerry group.

../_images/auth_provider_ldap_mapping.png

The order of the mappings is important. If a LDAP user appears in several mappings, the first successful mapping is taken. If the user cannot be found in any mapping, he will be moved to the default group.