Senior Data Engineer | Databricks Optimization Specialist (f/m/div.)

We are a dynamic team responsible for managing the central cloud infrastructure, including the Databricks platform. Our mission is to empower teams, ensuring high-performance, cost-effective solutions that optimize workloads and improve efficiency. We are looking for a Senior Data Engineer with a strong focus on Databricks optimization to help users fine-tune their workloads and contribute to cloud infrastructure enhancements.

As a Senior Data Engineer – Databricks Optimization Specialist, you will play a crucial role in optimizing Databricks workloads, ensuring high performance while minimizing costs and resource usage. You will work closely with users to enhance Spark query performance, fine-tune resource allocation, and implement best practices in Databricks. Proficiency in PySpark, Spark optimization, and query tuning is essential. Knowledge of Azure cloud services is a plus.

Key Responsibilities

• Optimize Databricks workloads to improve efficiency and reduce costs.

• Analyze and fine-tune Spark queries for optimal performance.

• Provide best practices for Databricks resource allocation and cluster management.

• Troubleshoot and resolve performance bottlenecks.

• Implement cost-saving measures and automate workload management.

• Develop and document performance optimization strategies.

• Stay updated with the latest Databricks and Spark advancements and recommend improvements.

• Support cloud infrastructure projects and advise on Databricks-Azure integrations.

• Assist in cloud infrastructure implementation and support various use cases.

Education:

• Degree in Computer Science, Data Engineering, or a related field (or equivalent experience).

Experience:

• Proven experience as a Data Engineer, with a strong focus on Databricks optimization.

• Hands-on expertise in PySpark and Spark performance tuning.

• Experience in query performance tuning and workload optimization.

• Knowledge of cost-saving techniques within Databricks.

Know-how:

• Deep understanding of distributed computing and big data processing.

• Expertise in Databricks resource allocation, cluster management, and automation.

• Familiarity with Azure cloud services is a plus.

Languages:

• Fluency in English (written and spoken) is required.

• Additional languages are a plus.

Working Style and Methods:

• Strong problem-solving skills and ability to troubleshoot performance bottlenecks.

• Ability to work independently while collaborating with teams.

• Proactive in staying updated with Databricks and Spark advancements.

Personality:

• High level of initiative, ownership, and autonomy.

• Excellent communication and collaboration skills to support users effectively.

• Passion for efficiency, optimization, and continuous improvement.

Why Join Us?

Work in a fast-paced, innovative environment.
Opportunity to influence and optimize a critical data platform.
Collaborate with a passionate team focused on data and technology.
Enjoy a competitive salary and benefits package.
Access continuous learning and professional growth opportunities.