https://www.mdu.se/

Cloud cost optimization is an important challenge for large-scale data platforms, where workloads are often over-provisioned to avoid performance problems. While existing FinOps tools provide visibility into cloud spending, they do not always provide clear and safe recommendations on how cluster resources should be changed.

This thesis presents a data-driven framework for rightsizing Databricks clusters using historical workload telemetry, cluster configuration data, and pricing information. The framework identifies underutilized workloads and evaluates several possible configuration changes, including reducing the number of workers, downsizing worker instances, downsizing driver instances, and combinations of these actions. Statistical Power-Law models and machine-learning models based on XGBoost, LightGBM, and Quantile Regression are used to estimate how CPU and memory utilization may change after rightsizing. Runtime changes are also considered when estimating the financial benefit of each configuration.

The models are evaluated using held-out production transitions together with controlled benchmark experiments. The results show that gradient-boosted models provide the strongest prediction accuracy on production workloads, while the Power-Law model transfers more reliably to controlled workloads that differ from the production data. Quantile regression provides a useful way to reduce false-safe recommendations by making utilization predictions more conservative. The experiments also show that simple worker-related changes are more predictable than compound actions involving driver downsizing.The final safety-constrained policy targets a relatively small set of underutilized, high-cost workloads. Within the underutilized workloads considered for rightsizing, removing high-risk candidate actions still preserves around 80% of the original projected savings. Across the full workload fleet, this corresponds to an estimated monthly cost reduction of approximately 12%. Overall, the work demonstrates how predictive resource modelling can be combined with safety constraints, runtime awareness, and cloud pricing to provide practical and explainable rightsizing recommendations for enterprise FinOps.