Sr. Data Engineer

Overview: The data engineering group, led by Ravi, is focused on building data integrations and data pipelines as part of the broader software engineering efforts. The team needs to onboard two new senior data engineers to meet critical business needs. The group works with multiple teams and offers flexibility to move between projects. They have multiple projects in flight currently: building pipelines to ingest data from their outage management system to IDS product, use cases around identifying and fixing power line issues with crews on the ground, cost planning and load prediction (Predictive analytics), customer service analytics around outage reports and tickets coming in, analytics to understand root cause of the problem.

At the core, this role is responsible for building data ingestion and ETL solutions to move data from sources (i.e. outage management software) to Databricks initially.

Using PySpark, SQL to extract and transform the data, move into their raw layer, landing zone convert into preprocessing layer, delta table, unified layer where they customize datasets for product they're targeting, service layer data quality checks and flags, intelligence layer to Snowflake as a warehouse being used solely for analytics.

At the preprocessing layer, have applications with APIs calling the data and pulling into the application for use. Moving away from using database back-ends.

Currently working on a proof of concept with Kafka to connect to the py/pie? historian and archive streaming data to Snowflake and real-time databases like Cosmos and Databricks Delta tables. Nice to have experience in Kafka.

Requirements:

Senior Data Engineers with expertise in the core Azure stack, including Databricks, Azure Data Lake Storage, Python, PySpark, and SQL.

Experience with real-time streaming technologies like Kafka is a plus.

Proficiency in Python, PySpark, and big data development.

Strong knowledge of Azure stack and Kafka.

Soft Skills:

Ability to work with multiple teams and flexibility to move between projects.

Strong customer-facing abilities.

Required Skills : Expertise in Python, PySpark, Databricks, SQL, Azure Data Factory

Basic Qualification :

Additional Skills :

Background Check : Yes

Drug Screen : Yes

Notes :
Selling points for candidate :
Project Verification Info :
Candidate must be your W2 Employee :Yes
Exclusive to Apex :No
Face to face interview required :No
Candidate must be local :No
Candidate must be authorized to work without sponsorship ::No
Interview times set :Yes
Type of project :
Master Job Title :
Branch Code :