Data Engineer w/Gen AI

This is a Hybrid role with 3 days a week in the office

We are seeking an experienced Data Engineer with 3-5+ years of experience to join our dynamic team. The ideal candidate will possess a strong background in data management and engineering, specifically within cloud environments. You will play a crucial role in developing and managing data pipelines that support AI and Generative AI initiatives, ensuring that our data architecture is robust, scalable, and optimized for performance.

Key Responsibilities:

• Data Pipeline Development: Design, develop, and manage data pipelines to support AI and Generative AI data requirements.
• Workflow Creation: Build self-service onboarding workflows in data federation platforms, particularly using AWS Athena, to facilitate efficient data access and integration.
• Schema Management: Own the ingestion of schemas, metadata APIs (including table schema descriptions), and table registration services to enhance data governance.
• SQL Execution Layer Design: Design and implement a SQL execution layer via AWS Athena that optimizes query performance and ensures data integrity.
• Access Controls: Implement table access controls, audit logging, and schema diffing to maintain security and compliance across our data assets.
• Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and ensure alignment with organizational goals.
• Continuous Improvement: Identify opportunities for process enhancements and drive best practices in data engineering and management.

Required Skills:

• Proficient in SQL and experience with AWS services, particularly Athena.
• Strong experience in ETL processes and data pipeline development.
• Proficiency in Python for data manipulation and automation tasks.
• Familiarity with REST APIs and Git for version control and collaboration.
• Understanding of IAM basics and data access control principles.

Qualifications:

• 3-5+ years of experience in data engineering or a related field.
• Bachelor's degree in Computer Science, Information Technology, or a related discipline (or equivalent experience).
• Strong analytical and problem-solving skills, with a keen attention to detail.
• Excellent communication skills and ability to work collaboratively in a team environment.