Data engineer

We are seeking a highly skilled Data Engineer to drive data engineering initiatives and build scalable, high-performance data solutions. The ideal candidate will have deep expertise in Azure Data Factory (ADF), Azure Databricks, and cloud-based data architecture. This role involves leading a team, designing and implementing ETL/ELT pipelines, and optimizing data workflows for advanced analytics and reporting.

Key Responsibilities:

• Lead and mentor a team of data engineers in building and optimizing data pipelines.
• Design and implement ETL/ELT processes using Azure Data Factory (ADF) and Azure Databricks.
• Develop and maintain scalable big data processing and data lake solutions on Azure.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Optimize and fine-tune Spark-based data transformation jobs for performance and cost efficiency.
• Ensure data quality, governance, and compliance with industry standards.
• Integrate data from diverse sources, including structured and unstructured data.
• Monitor data pipelines and troubleshoot issues to maintain high availability.
• Implement security best practices for data access, encryption, and compliance.

Required Skills & Experience:

• 8+ years of experience in data engineering with a strong background in Azure cloud services.
• Expertise in Azure Data Factory (ADF) for building automated data workflows.
• Hands-on experience with Azure Databricks, Apache Spark, and PySpark.
• Strong knowledge of SQL, data modeling, and data warehousing concepts.
• Experience with Azure Synapse Analytics, Azure Data Lake, and Blob Storage.
• Proficiency in Python or Scala for data processing and scripting.
• Familiarity with CI/CD pipelines for data engineering and DevOps best practices.
• Excellent problem-solving skills and ability to work in a fast-paced environment.
• Strong leadership, communication, and collaboration skills.

Preferred Qualifications:

• Experience with Machine Learning Pipelines on Azure.
• Knowledge of Kafka, Event Hubs, or other real-time streaming technologies.
• Certifications in Microsoft Azure Data Engineering (DP-203) or equivalent.