Senior Data Engineer

Job Summary:

We are seeking a Senior Data Engineer to design, develop, and maintain big data applications using the latest open-source technologies.

The ideal candidate will have extensive experience in GCP, data pipeline development, and distributed data processing.

This role also requires mentorship, project leadership, and collaboration with cross-functional teams to deliver high-quality data solutions.

Key Responsibilities:

Big Data Development & Engineering:

• Design and develop big data applications using Apache Hive, Spark, and Kafka.
• Automate workflows using Apache Airflow.
• Build data pipelines and optimize data processing workflows.
• Develop logical and physical data models for big data platforms.
• Provide ongoing maintenance, enhancements, and on-call support for existing systems.

GCP Expertise & Cloud Data Engineering:

• Build data pipelines in GCP using GCP Dataproc, GCS, and BigQuery.
• Leverage cloud-native tools to optimize performance, scalability, and cost.
• Implement best practices for cloud data processing and storage.

Leadership & Project Execution:

• Lead daily standups, design reviews, and backlog grooming using JIRA.
• Mentor and guide junior engineers on best practices and development standards.
• Act as the point of contact for assigned business domains, ensuring alignment with business needs.
• Work collaboratively with cross-functional teams, including product managers, architects, and DevOps teams.

Code Quality & Best Practices:

• Follow Gitflow development practices and version control using BitBucket, JIRA, and Confluence.
• Implement CI/CD pipelines using Jenkins, Bamboo, or TFS.
• Ensure coding standards, performance optimization, and code reviews are consistently followed.
• Conduct unit testing and validate test results against product specifications.

Required Qualifications:

• 10+ years of experience in data engineering and data warehouse development.
• 6+ years of hands-on experience with Hadoop, Hive, Spark, Airflow, or similar workflow orchestration solutions.
• 4+ years of recent experience working with GCP.
• Strong proficiency in Python, Java, Scala, and scripting languages (Perl, Shell, etc.).
• Experience in schema design and data modeling for data lakes and RDBMS platforms.
• Expertise in processing and managing large datasets (multi-TB/PB scale).
• Hands-on experience with Kubernetes, Docker, and containerized environments.
• Experience with Agile methodologies, Scrum/Kanban development practices.
• Strong analytical, problem-solving, and troubleshooting skills.
• Excellent verbal and written communication skills.

Preferred Qualifications:

• Experience with CI/CD tools (Bamboo, Jenkins, or TFS).
• Familiarity with Atlassian products (BitBucket, JIRA, Confluence).
• Knowledge of Gitflow branching strategies.
• Experience in test-driven development and automated testing frameworks.
• Background in domain-specific knowledge and customer engagement.

Education: Bachelors Degree