Databricks/Scala Sr. Developer / Data Engineer

Apply Now

Company: Sarian, Inc.

Location: Chicago, IL 60629

Description:

Job Role - Databricks/Scala Sr. Developer / Data Engineer
Duration - Fulltime
Location - Chicago ,IL or Remote

Job Description:

Azure Databricks, Scala, Spark
Must Have Scala Development background
Good Spark Architectural Knowledge Must have implementation using SparkSQL & Scala in Spark framework for data processing
Must have implementation experience on Spark streaming
Design data pipelines
Performance optimization of data pipelines
Knowledge in debugging spark clusters
Databricks experience preferred
Domain expertise - Healthcare preferred but not essential
End-to-end development and deployment of data lake solution
Building proofs of concept and prototypes
Help architecture team in architecting solutions
Pair-programming with customer teams
Build data pipeline between various data lake layers e.g., RAW layer, Curation layer and Serve Layer
Work with data modeling team in designing schema evolution, data models for RAW layer, Curation layer
Work with Walgreens Dev Ops/Run ops team in deployments of solution
Building and deploying a range of data engineering pipelines into production, including using automation best practices for CI/CD
Build data interfaces with various downstream and upstream systems (e.g. EDW, IC+ and others)
Build data streaming and ingestion with various source systems both in batch mode and real time integration mode.
Use expertise in data engineering best practices to guide Walgreens to do the same
Work with Performance engineering team in improving overall performance of data lake solution across various architectural components
Query tuning and troubleshooting and debugging data lake solution
Coordinate with Account Executives, Customer Engineers and Solution Architects for expanding the use of IDI Data lake platform within Walgreens
Train junior team members

Databricks/Scala Sr. Developer / Data Engineer

Similar Jobs