Data Engineer

As a Big Data Application Developer, you will be responsible for developing, optimizing, and maintaining big data solutions. You will work with a team of data engineers and analysts to build scalable and efficient data processing pipelines and applications using a variety of technologies.

Key Responsibilities:

• Develop and maintain big data processing applications using Spark, Python, and Scala.
• Design and implement ETL processes using AWS Glue.
• Build and manage workflows with AWS Step Functions and Lambda.
• Develop, test, and deploy data pipelines on Databricks.
• Collaborate with data engineers, analysts, and stakeholders to understand data requirements and deliver solutions.
• Optimize and tune data processing jobs for performance and scalability.
• Ensure data quality and integrity through thorough testing and validation.
• Document processes, solutions, and provide support as needed.

Required Skills and Qualifications:

• Programming Languages: Proficiency in Python and Scala.
• Big Data Technologies: Experience with Apache Spark for large-scale data processing.
• Cloud Services: Hands-on experience with AWS Glue, AWS Step Functions, and AWS Lambda.
• Data Platform: Familiarity with Databricks for data engineering and analytics.
• Database Knowledge: Understanding of SQL and NoSQL databases.
• ETL Processes: Strong experience in designing and implementing ETL pipelines.
• Data Integration: Knowledge of integrating various data sources and systems.
• Performance Tuning: Skills in optimizing big data jobs for performance.
• Collaboration: Excellent communication skills and the ability to work collaboratively in a team environment.
• Problem-Solving: Strong analytical and problem-solving abilities.

Preferred Qualifications:

• Experience with additional big data tools and frameworks.
• Certification in AWS or relevant big data technologies.
• Familiarity with DevOps practices and CI/CD pipelines.
• Experience with machine learning frameworks and libraries.

Education:
Bachelor's degree in Computer Science, Engineering, or a related field. Advanced degrees are a plus.

We are an Equal Opportunity Employer committed to a diverse and inclusive workplace. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, age, gender identity, national origin, disability, or veteran status. We value diverse perspectives and actively seek to create an inclusive environment that celebrates the unique qualities of all employees.