← Back to portfolio
Oves Enterprise
Cloudera & Spark Optimization: Stabilizing a Petabyte-Scale Data Platform
Oves Enterprise is a global software engineering company formed of analysts, developers, strategists and industry experts. We help enterprises to translate ideas into products in order to deliver real business value.
Business Challenge
We partnered with the client to stabilize and optimize an enterprise navigation application running on Cloudera and Spark, hosted entirely on-premise in a local data center. The platform was handling petabyte-scale workloads and had accumulated a range of systemic issues that were impacting performance, availability, and long-term maintainability.
Challenges Addressed
Corrupt Cloudera installation causing instability and service failures.
Growing data volumes (petabytes) requiring additional nodes for scalability.
Spark jobs running inefficiently, leading to unacceptably long processing times.
Hadoop configuration not fine-tuned for replication and resilience.
Vendor dependency on Cloudera limiting long-term flexibility.
Minimal monitoring and alerting, making it difficult to ensure system availability.
Results
Identified and fixed root causes of corruption in the Cloudera distribution.
Restored stability and availability of critical data services.
Added new nodes to handle petabyte-scale workloads.
Improved Spark job algorithms, significantly reducing execution times and enabling daily job runs.
Fine-tuned resource allocation across YARN for optimized job scheduling.
Tuned replication factors, I/O operations, and failover configurations.
Migrated from Cloudera to a vanilla Hadoop distribution with HA master, eliminating vendor lock-in.
Set up Prometheus and Grafana for real-time monitoring and alerting.
Tech Stack
Java
Jenkins
Apache Spark
Bash
PostgreSQL
Python
C#
Scala