Azure Data Engineer- Hybrid, Pittsburgh, PA

Overview

Location: Hybrid- Pittsburgh, PA

Job Type: Full Time/Permanent

Work Authorization: No Sponsorship

The A.C.Coy has an immediate need for an Azure Data Engineer for a full time opportunity.

Responsibilities

• Automate data pipeline monitoring, error handling, and performance tuning to ensure data integrity and availability
• Collaborating with the Digital Data Solutions Team or external consultants as needed
• Design, develop, and implement scalable and high-performance data solutions on the Azure platform, specifically architecting data pipelines and ETL/ELT processes utilizing Azure Data Factory and Azure Databricks to extract, transform, and load data from various sources into Azure data storage solutions
• Optimize data storage and retrieval mechanisms to ensure efficient processing of large datasets, particularly focusing on the capabilities of Databricks
• Develop and maintain data models and schemas to support data analysis and reporting requirements
• Implement data security and privacy measures to protect sensitive data
• Develop and maintain data ingestion processes to acquire data from various sources, including APIs, databases, and streaming platforms
• Implement data quality checks and validation procedures to ensure the accuracy and integrity of data
• Design and implement data transformation and enrichment processes using Azure Databricks, Azure Data Factory, and other relevant tools
• Document technical design, deployment processes, and best practices for data engineering solutions
• Implement data partitioning, indexing, and optimization techniques to improve query performance
• Collaborate with data scientists and analysts to operationalize and deploy machine learning models on Azure, utilizing Databricks for model training and deployment
• Configure and manage data storage options, including Azure Blob Storage, Azure Data Lake Storage, and Databricks LakeHouse
• Collaborate with DevOps teams to deploy and manage data engineering solutions in Azure

Qualifications

Education: Bachelor's degree in Computer Science, Computer Engineering, or other technical field

• Hands-on experience with Azure data services, including Azure Data Lake Storage, Azure SQL Database, Azure Data Factory, and Azure Databricks. Understanding of Azure Synapse Data Warehouse preferred
• Strong experience in data engineering and architecture, with a focus on Azure cloud technologies
• Proficiency in data-centric programming languages, including SQL, Python, and PowerShell
• Knowledge of data integration techniques and tools, such as ETL/ELT processes, data pipelines, and data warehousing concepts
• Familiarity with data modeling and schema design principles
• Understanding of data security and privacy regulations, including GDPR and CCPA
• Excellent problem-solving and analytical skills
• Strong communication and collaboration abilities
• Azure certifications, such as Microsoft Certified: Enterprise Data Engineer Associate, are a plus