Setup Hygieia DevOps Dashboard on Azure Kubernetes (AKS)

6 min readAug 9, 2020

Hygieia is a open source dashboard developed by Capitalone for addressing the complexity of monitoring ever growing list of devops tools. Given the number of tools and processes introduced to DevOps over the time, it is good to have a single pane of view for tracking the status of various components. I am not in a position to recommend or compare this Dashboard as this is first DevOps dashboard I am interacting with. However, it has interesting views of delivery pipelines, source control management, security violations and etc. On the other had, they are still developing the Dashboard and upgrade it but their documentation is not up to the mark yet where you have to put some extra effort and investigations to get things up and running. This is where I thought of summarizing steps for benefit of others.

These are the steps I followed as of July 2020 and things may change based on the time you read this.

There are two different components or dashboards of Hygieia, one for engineers and one for executives. This article focus on the first one known as Hygieia dashboard focused on engineers or Devops team. However Hygieia executive dashboard relies on Hygiea for data collection so it can’t run alone itself.

Hygieia consists of four different layers as illustrated in below diagram. Let me summarize them in the order of deployment sequence from start to end. You can find in depth details from Hygieia official documentation.

Architecture diagram (source: Hygieia official documentation)

Database — currently supports mongo db but they are in process of enabling other NoSQL databases. In this case, mongodb will be deployed in a container on AKS.
API — Contains all the API services to collect data from collectors. It will be deployed as a container as well.
Collectors — Fetches data from DevOps tools such as Jira, Git, Bamboo, etc. Each collector can fetch data from single source where we will be deploying a container for each collector type.
UI — Front end gui for the end users to view the dashboard. This is again a container on AKS

We need to have a environment to build application jar files and create docker containers from scratch. While there are numerous ways to achieve this, We will use a temporary Ubuntu VM in Azure for this case. It is possible to automate whole process using Azure DevOps pipelines which I will explain in another article. These are the steps for creating the build VM.

Create a VM with ubuntu OS. Let’s use root account for all the steps for convenience.
Install Java, Maven and docker. Note that java version should be 8.x other versions didnt work.

apt install openjdk-8-jdk maven docker.io

Create directory for downloading source code and build source.

mkdir /opt/hygieia

There should be a container registry for uploading containers. Create one in Azure if doesn’t exist.
Login to the container registry from the build VM

docker login containerregname.azurecr.io

Now all the prerequisites are done and ready for the Hygieia package build

Hygieia API

Clone the API source code

cd /opt/hygieia
git clone https://github.com/Hygieia/Hygieia.git
cd Hygieia
git clone https://github.com/Hygieia/api.git

Run maven build

cd /opt/hygieia/Hygieia/api
mvn clean install

You will see the below output at the end of successful build completion.

[INFO] BUILD SUCCESS

Create the container and upload to container registry

docker build . -t hygieia-api:1.0
docker tag hygieia-api:1.0 containerregname.azurecr.io/hygieia-api
docker push containerregname.azurecr.io/hygieia-api

2. Hygieia UI

Clone the UI source code

cd /opt/hygieia/Hygieia
git clone https://github.com/Hygieia/UI.git

Run maven build

cd /opt/hygieia/Hygieia/UI
mvn clean install

You will see the below output at the end of a successful build completion.

[INFO] BUILD SUCCESS

Create the container and upload to container registry

docker build . -t hygieia-ui:1.0
docker tag hygieia-ui:1.0 containerregname.azurecr.io/hygieia-api
docker push containerregname.azurecr.io/hygieia-ui

3. GitHub collector

Clone the source code

cd /opt/hygieia/Hygieia
git clone https://github.com/Hygieia/hygieia-scm-github-collector.git

Run maven build

cd /opt/hygieia/Hygieia/hygieia-scm-github-collector/
mvn clean install

You will see the below output at the end of successful build completion.

[INFO] BUILD SUCCESS

Create the container and upload to container registry

docker build . -t hygieia-github-collector:1.0
docker tag hygieia-ui:1.0 containerregname.azurecr.io/hygieia-github-collector
docker push containerregname.azurecr.io/hygieia-github-collector

Login to your Github account and create a personal access token to grant Hygieia access. This token will pass to the github collector container later. Assign permission () to personal access token

4. Bamboo collector

Clone the source code

cd /opt/hygieia/Hygieia
git clone https://github.com/Hygieia/hygieia-build-bamboo-collector.git

Maximum no of bamboo plans that can be queried are hard coded to 2000. If you have more than 2000 plans, update this config before the build.

src/main/java/com/capitalone/dashboard/collector/DefaultBambooClient.javaprivate static final String JOBS_URL_SUFFIX = "rest/api/latest/plan?expand=plans&max-result=2000";

Run maven build

cd /opt/hygieia/Hygieia/hygieia-build-bamboo-collector
mvn clean install

You will see the below output at the end of a successful build completion.

[INFO] BUILD SUCCESS

Create the container and upload to container registry

docker build . -t hygieia-bamboo-collector:1.0
docker tag hygieia-bamboo:1.0 containerregname.azurecr.io/hygieia-bamboo-collector
docker push containerregname.azurecr.io/hygieia-bamboo-collector

Create a account in Bamboo master server for querying the data from Hygieia. Just creating account with read access is sufficient. Later this account credentials must pass to containers as environment variables.

Now we have completed the source code build and container creation. Even though I only illustrated here Bamboo and Github collectors, there are many other collectors available in Hygieia.

Lets jump in to the Hygieia deployment on Azure Kubernetes Services (AKS).

In order to improve the portability of containers and its configuration, all the parameters are configured as configmaps and secrets. We will have one configmap and one secret to store all these variables.

Configmap

Mongo db admin username, Hygieia username, port and mongo service name
Hygieia API service name and port
Bamboo server name and user
Other collector information

Secret

Mongo db admin password and user password
API secret
GitHub token, bamboo user password and other collector confidential data

Persistent volumes

Persistent volumes required for Mongo db data storage and application log file storage. AKS comes with default storage classes for persistent storage.

Use managed-premium storage class for the mongodb data storage as it supports read write once (RWO) access mode. If you assign RWX storage, mongodb will run into errors in case of auto restarting container.

Use azurefile-premium storage for log files of application components which supports read write many (RWX) access mode. RWX will allow all the containers to write to the same azure file storage concurrently.

Deployments

There will be a deployment for each of the component of hygieia including individual collectors.

Mongodb deployment contains single container with environment variables to define above configmaps and secrets

Similarly, API, UI and Collectors have to be configured with correct variables and pvc’s
It is important to enable liveliness probes for the container to detect the application health and restart container automatically.

Services

Each deployment will expose to each other via services. These service names are included in configmap so that each container knows each others access points.

Now all the services are up and running yet not accessible from outside. Let’s create Nginx service and expose the UI to the outside of k8s cluster.

First of nginx ingress controller should be deployed if there is no existing one.
Then create nginx resource to expose thr UI cluster service to the outside world.

Now you should be able to access the Hygieia UI dashboard from web browser. Create a account and login to the system. Add the collector credentials inside the settings page which will allow collectors to collect data from other systems.

At this point you are good to create product or team dashboard to display your project status.

This concludes story about Hygieia Dashboard and hope it was useful.