Senior Data Engineer

Apply Now

Company: Technogen, Inc.

Location: Atlanta, GA 30338

Description:

Role: Senior Data Engineer

Location: Dunwoody, GA (Day 1 On Site Hybrid)

Long Term Contract

Responsibilities:

Design, develop, and maintain scalable ETL pipelines to ingest and transform data from multiple sources.
Implement data integration strategies using Python, Spark, and Databricks to enable seamless data ingestion and transformation.
Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
Develop and optimize data workflows to improve data quality, consistency, and performance.
Ensure data security, governance, and compliance best practices are followed.
Conduct performance tuning of Spark jobs and optimize data queries to minimize latency and improve efficiency.
Automate data pipelines using CI/CD tools and ensure continuous data delivery.
Troubleshoot data issues, identify root causes, and implement effective solutions.

Required Qualifications

Strong proficiency in Python, with experience in data manipulation libraries like Pandas and PySpark.
Extensive experience with Apache Spark for data processing and analysis.
Hands-on experience with Databricks for building scalable data pipelines.
Experience with data lakes, data warehouses, and cloud platforms like AWS, Azure, or GCP.
Strong SQL skills for data extraction, transformation, and querying.
Familiarity with version control systems (e.g., Git) and CI/CD processes.

Preferred Skills

Experience with workflow orchestration tools like Apache Airflow, Dagster, or Prefect.
Knowledge of data modeling techniques and best practices.
Familiarity with data visualization tools such as Power BI, Tableau.
Understanding of data privacy regulations and security protocols