Ciprum Technology

Challenge

Supplying big data and devops engineering services in multiple cost optimization projects which have proved millions of pounds in savings, transformed the way Data Scientists work and provided training to permanent staff.

Solution & Results

We lead the design and developed multiple SQL ETL in-memory data pipelines with Alteryx and SAP HANA Stored Procedures, introduced integration testing and written code for analytics, business logic and data transformations bringing multiple data sources together from GSAP, Oracle Database, Sharepoint and Azure SQL Databases.

For workloads that required extensive analytics that a traditional database would not support or handle, we wrote Apache Spark code on PySpark and SparkR initially on Databricks and then migrating the code via Docker images and Helm Charts onto a Kubernetes Cluster we provisioned with Terraform. Automation is key - so as with other clients we implemented CI/CD for code and infrastructure changes via Github and CircleCI and restricted on the fly updates with proper version controlled and enforced Pull Requests.