Data Platform Engineer

Job Overview:
We are seeking a Platform Engineer with a strong focus on supporting and optimizing data platforms. This individual will play a key role in designing, building, and maintaining CI/CD pipelines, automated testing frameworks, and various tools to support data systems. The ideal candidate will have hands-on experience with ETL pipelines, data engineering processes, and platforms like Snowflake and FiveTran, and a keen understanding of how data flows through these systems. A self-starter with strong communication skills who thrives in a mid-sized environment will be an excellent fit for this role.

Key Responsibilities:

CI/CD Pipeline Creation and Maintenance:

• Design, implement, and maintain continuous integration and continuous deployment (CI/CD) pipelines for data-related projects.
  
• Automate data deployment processes to ensure smooth and efficient updates across environments.
  
• Integrate automated testing frameworks into the CI/CD pipelines to ensure high-quality code delivery.

Development of Tools to Support Data Systems:

• Build and maintain custom tools that support data infrastructure, including those that assist with data processing, monitoring, and reporting.
  
• Develop and optimize automated workflows and frameworks to improve the reliability and scalability of data platforms.

ETL Pipeline Understanding and Optimization:

• Develop, implement, and maintain ETL pipelines, ensuring smooth data flow through the various stages.
  
• Troubleshoot and optimize ETL processes, identifying potential bottlenecks, inefficiencies, and errors.
  
• Understand the interdependencies between data systems, ensuring that ETL pipelines are well integrated with other tools and platforms.

Experience with Snowflake and FiveTran:

• Design and optimize Snowflake data warehouses to ensure efficient data storage, querying, and scaling.
  
• Work with FiveTran for data integration, ensuring that data flows seamlessly between systems and is available for analytics and reporting.
  
• Develop and maintain integration processes that utilize Snowflake and FiveTran.

Collaboration and Communication:

• Collaborate with cross-functional teams, including data engineers, analysts, and DevOps, to ensure alignment and successful implementation of data solutions.
  
• Provide support and guidance to internal teams on best practices for working with data platforms and tools.
  
• Communicate technical concepts and progress to both technical and non-technical stakeholders effectively.

Problem Solving and Automation:

• Continuously monitor and improve the reliability of the data infrastructure, identifying areas for automation and improvement.
  
• Proactively address potential issues with data flow, pipeline performance, and tool functionality.

Qualifications:

• Proven experience as a Platform Engineer or Data Engineer with a strong focus on CI/CD pipelines, automated testing, and data tools.
  
• In-depth experience with ETL pipeline development and optimization.
  
• Strong experience with Snowflake and FiveTran, including architecture design, data loading, and optimization best practices.
  
• Familiarity with common data engineering tools and frameworks.
  
• Excellent problem-solving skills with the ability to troubleshoot and optimize complex data systems.
  
• Self-starter with the ability to work independently and take ownership of projects.
  
• Strong communication skills, both written and verbal, with the ability to collaborate effectively in a mid-sized team environment.
  
• Experience in a fast-paced environment with a focus on delivering results.
  
• Familiarity with cloud-based data platforms (AWS, Azure, GCP) is a plus.