GCP data engineer
Apply NowCompany: Diamondpick
Location: Charlotte, NC 28269
Description:
Job Description:
Key Responsibilities:
Required Qualifications:
- Job Overview:
The GCP Data Engineer will be responsible for designing, developing, and maintaining data pipelines, integrating data sources, and building scalable and efficient data architectures on Google Cloud Platform. This role will require strong technical expertise in cloud technologies, data modeling, ETL/ELT processes, and working with large datasets.
Key Responsibilities:
-
- Design, develop, and maintain data pipelines using Google Cloud services such as BigQuery, Dataflow, Pub/Sub, Cloud Storage, and Dataproc.
- Implement and optimize ETL processes to transform raw data into valuable insights.
- Build and maintain cloud-based data lakes, data warehouses, and data marts using GCP technologies.
- Collaborate with data scientists, analysts, and other engineers to understand business requirements and deliver data solutions that support decision-making processes.
- Optimize SQL queries, data models, and cloud-based data architectures to ensure high performance and cost-effectiveness.
- Automate data workflows using orchestration tools like Cloud Composer (Apache Airflow) and other GCP-native services.
- Monitor data pipelines and ensure data integrity, quality, and security.
- Perform data profiling, cleansing, and transformation to ensure high-quality data.
- Stay current with the latest GCP technologies and best practices, and actively evaluate new tools and frameworks to improve data engineering workflows.
- Support the implementation of data governance and compliance practices in accordance with industry standards.
- Participate in code reviews, troubleshooting, and debugging as necessary.
Required Qualifications:
-
- Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or related field.
- Proven experience as a Data Engineer, with a strong focus on GCP (Google Cloud Platform) technologies.
- Solid experience with Google Cloud services, including BigQuery, Dataflow, Pub/Sub, Cloud Storage, Dataproc, Cloud Functions, and Cloud Composer.
- Strong knowledge of SQL and experience with cloud-based data warehouse solutions.
- Experience in data pipeline design, development, and optimization.
- Familiarity with data modeling, ETL/ELT processes, and data lake design.
- Proficiency in programming languages such as Python, Java, or Scala for data engineering tasks.
- Experience with version control systems like Git.
- Familiarity with containerization and orchestration tools such as Docker and Kubernetes is a plus.
- Knowledge of data security and best practices for cloud infrastructure.
- Ability to work independently and in cross-functional teams, effectively communicating technical concepts to both technical and non-technical stakeholders.