Industry
geolocation, data engineering
Location
Poland
Time
01 - 09.2020
The main goal of the project was to achieve high flexibility, data ingestion throughput and availability of the environments on which the software is hosted in the form of the offered products, i.e. the applications. The requirement was to implement modern and leading technology, streamlining existing processes through deployment, prepared and configured as part of client’s own infrastructure.
The objective of the project was to prepare a high availability environment using the leading technology - Kubernetes, placed on virtualised environment. The technical assumptions also included the acquisition of smooth horizontal and vertical scaling for fast data ingestion, quick and easy deployment of new applications, automatic environment creation, as well as external access to applications using public IP addresses and Load Balancers. Furthermore, an installation of additional applications, functioning as separate containers, for the purposes of communication with services running outside the Kubernetes cluster, but within virtual environment.
Once we established all technical requirements and agreed on the proposed solution with the client, we then divided the scope into several stages of technical work. The first stage was to prepare the environment for Kubernetes - the initial test was on VMWare virtualisation, however, it was rejected as it did not meet the network and throughput requirements. Accordingly, we proposed Proxmox virtualisation for the next testing stage and this way network requirements have been resolved and Kubernetes was set up on the Proxmox platform.
In the next step we have carried out a configuration of the external IP access and we have prepared the Kubernetes manifests for: deployments, ingresses and services for each intended application. We have also configured authorisation and mechanics, enabling the automatic deployment of the application by client CI/CD tool. Additionally, we have carried out a configuration of data pipelines, prepared separate KVMs and installed and configured additional applications outside the Kubernetes Cluster (PostgreSQL, OrientDB).
We have created a ready-made solution for automatic installation of the Kubernetes cluster, based on the KVM on Proxmox virtualisation, on a dedicated server with one public IP address. Accordingly, we have met the project assumptions and goals, the most important requirement of high data ingestion throughput, simple and fast service configuration and automated application deployment achieved, thus facilitating the ongoing work processes of the technical team and more effective and efficient delivery of own products to company’s clients and partners.
