Apps

The applications which are provided by blackbar consist of

1. A frontend for generating training data & manual validation of results (Inception) and a place for storing models which allow to detect personally identifiable information (Minio) (link)
2. A cockpit application to easily create projects, upload data, auto anonymize and pseudonymize, build and maintain models. And web applications to show the results of the anonymization and pseudonymization, run and deploy models (apps using Shinyproxy) (link)

Inception and Minio

Inception and Minio are integrated in one docker image allowing you to deploy the annotation and model storage framework quickly as shown below:

docker pull ghcr.io/bnosac/blackbar-inception-minio:latest
docker run --rm -it --name blackbar-apps-inception \
    -p 9901:9001 \
    -p 9900:9000 \
    -p 8180:8080 \
    --env-file .env \
    ghcr.io/bnosac/blackbar-inception-minio

• Inception
  • Runs at port 8080
  • You can log in with the values that you have set with the environment variables INCEPTION_USERNAME and INCEPTION_PASSWORD as specified in .env
• For Minio
  • The frontend which is also known as the console is configured to run at port 9001
  • The minio server S3 environment is configured to run at port 9000
  • You can log in with the values that you have set with the environment variables BLACKBAR_S3_ACCESS_KEY_ID and BLACKBAR_S3_SECRET_ACCESS_KEY in .env

The docker container stores the files on Minio internally in the container in folder /data, while the annotations from Inception are stored at folder /export. Make sure you persist these folders such that at container restart, the models and annotations are not gone. E.g. with the following volumes.

docker run -d --name blackbar-apps-inception \
    -p 9901:9001 \
    -p 9900:9000 \
    -p 8180:8080 \
    -v $(pwd)/minio:/data \
    -v $(pwd)/inception:/export \
    --restart always \
    --env-file .env \
    ghcr.io/bnosac/blackbar-inception-minio
docker logs blackbar-apps-inception

Inception alongside MariaDB

By default the above setup uses a HyperSQL database to store it’s metadata, you can make this more reliable by using Inception alongside your own MariaDB to store the metadata of Inception.

Note that the annotations which you perform in Inception are stored in xml/xmi files on the host and not in the database.

Inception with non-external database

If you don’t have an existing MariaDB, you can use the image ghcr.io/bnosac/blackbar-inception-mariadb which will run a MariaDB under supervisord alongside the Inception app and the Minio storage engine. In that case make sure you as well create a volume which persistently stores the MariaDB as shown below.

docker pull ghcr.io/bnosac/blackbar-inception-minio-mariadb:latest
docker run -d --name blackbar-apps-inception \
    -p 9901:9001 \
    -p 9900:9000 \
    -p 8180:8080 \
    -v $(pwd)/s3:/data \
    -v $(pwd)/inception:/export \
    -v $(pwd)/inception-db:/var/lib/mysql \
    --restart always \
    --env-file .env \
    ghcr.io/bnosac/blackbar-inception-minio-mariadb

Inception with an external database

If you have already a MariaDB running, you need to make sure Inception can talk to it as outlined below.

• make sure an inception user is allowed full rights on a database called inception in your MariaDB which should be configured for 4-byte UTF-8 character set (utf8mb4) and a case sensitive collation (utf8mb4_bin)

CREATE DATABASE inception DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
CREATE USER 'inception'@'localhost' IDENTIFIED BY 'inception_db_password';
GRANT ALL PRIVILEGES ON inception.* TO 'inception'@'localhost';
FLUSH PRIVILEGES;

• specify the following environment variables in .env when launching the container and replacing the URL/password according to your setup.

INCEPTION_DB_URL=jdbc:mariadb://localhost:3306/inception?useSSL=false&serverTimezone=UTC&useUnicode=true&characterEncoding=UTF-8
INCEPTION_DB_USERNAME=inception
INCEPTION_DB_PASSWORD=inception_db_password
INCEPTION_DB_DIALECT=org.hibernate.dialect.MariaDB106Dialect
INCEPTION_DB_DRIVER=org.mariadb.jdbc.Driver

Web applications using Shinyproxy

blackbar contains several applications which can be used to

• Monitor the evolution and progress of annotations
• Easily upload and pre-annotate and pseudonymize texts
• Monitor, train and run models which detect personally identifiable information
• See how the anonymization and pseudonymization work on your own texts
• Administrative tasks on the automation of the anonymization and pseudonymization

The applications are made available as Docker-based apps which are deployed behind ShinyProxy.

For readers who are not familiar with ShinyProxy, it launches for each person which starts an app a separate Docker container such that the apps are isolated. For this ShinyProxy needs to have access to the Docker daemon and the default user under which the Docker container runs (which is for our Docker images user blackbar) needs to be part of the docker group such that the apps can be launched and as well the apps can launch anonymization and pseudonymization runs or backend flows using Docker.

In our setting ShinyProxy is as well deployed as a docker container which requires to create a network with exactly the name blackbar-apps-network such that it knows which webapps has started up.

In order to launch the webapps, pull the image, create the network and start up the container as shown below such that it can connect to the docker daemon.

docker pull registry.datatailor.be/blackbar-apps:latest
docker network create blackbar-apps-network
DOCKER_GROUP=$(getent group docker | cut -d: -f3)
docker run -d --name blackbar-apps \
  -p 8280:8080 \
  --net blackbar-apps-network --group-add $DOCKER_GROUP -v /var/run/docker.sock:/var/run/docker.sock:ro \
  --restart always \
  --env-file .env \
  registry.datatailor.be/blackbar-apps

This will launch the webapps on port 8080.

In order to add authentication, you can use environment variables which will be passed on to ShinyProxy, allowing to authenticate in different ways.

• There are 3 user groups: ‘blackbar-admin’, ‘blackbar-developer’ and ‘blackbar-user’.
• ‘blackbar-admin’ and ‘blackbar-developer’ have access to all apps including the apps which allow to develop the webapps
• ‘blackbar-user’ has only access to the applications which allow to inspect the data

The below example launches the apps with a simple authentication with one user which is part of the blackbar-admin group and another user which is part of the blackbar-developer group.

docker run -d --name blackbar-apps \
  -p 8280:8080 \
  --net blackbar-apps-network --group-add $DOCKER_GROUP -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -e PROXY_AUTHENTICATION="simple" \
  -e PROXY_USERS_0_NAME="blackbar" -e PROXY_USERS_0_PASSWORD="yourpassword" -e PROXY_USERS_0_GROUPS="blackbar-admin" \
  -e PROXY_USERS_1_NAME="bbdev123" -e PROXY_USERS_1_PASSWORD="yourpassword" -e PROXY_USERS_1_GROUPS="blackbar-developer" \
  --restart always \
  --env-file .env \
  registry.datatailor.be/blackbar-apps

After you logged in with the user that you provided, you can visit the applications.