Ciprum Technology

Challenge

An existing Kubernetes powered system had already been in place including a lab environment for Data scientists that contained a variety of different code pipelines and models written in R/Python and Python/NodeJS APIs; but everything was built with a complete PoC mindset.

There weren't any traces of version control, users/backups management nor automation for deployments which led to a variety of different challenges such as risk of losing track of existing code, significant lag in deploying changes either in the Kubernetes infrastructure or in the underlying applications running in Kubernetes.

Solution & Results

The existing system has been scripted in full and CI/CD pipelines have been implemented in CircleCI. In-house built code written in Python and NodeJS has been versioned using BitBucket while the Data Science Lab Kubernetes solution was built on AWS based on open-source JupyterHub Helm Charts and running on EC2 nodes with kops; containing JupyterLab user pods, each one with individual persistent storage backed up by EBS; deployed through Docker images, Helm Charts and integrated with AWS ECR for tagged hosting.

Developed and exposed a secure Python API to individual Kubernetes JupyterLab Pods to enable controlled sourcing of data from the client's data warehouse, with utilising caching libraries like pystore and dask to enhance user experience. In addition SSL of the web application has been offloaded to an AWS Load Balancer working with the JupyterHub application and the AWS Certificate Manager. User management has been automated with nodeJS scripts while backups of user data are stored in a tagged and timestamped approach in long term and cheap S3 storage.

The solution that has been implemeted has rapidly reduced the time to market allowing the client to gain more confidence in the system, act faster on insights and make more accurate decisions; ultimately increasing revenue and customer retention.