Azure Databricks Lead/Architect
Apply NowCompany: Lorven Technologies Inc
Location: Chicago, IL 60629
Description:
Job Title: Azure Databricks Lead/Architect
Location: Chicago, IL (Hybrid)
Duration: 12 +months
Experience: 12+years
Required Skills & Experience:
Responsibilities:
Preferred Skills (Bonus Points):
Location: Chicago, IL (Hybrid)
Duration: 12 +months
Experience: 12+years
Required Skills & Experience:
- 12+ years of professional experience in Data Engineering, Architecture, or Data Platforms.
- 5+ years of hands-on experience with Azure Databricks, Apache Spark, and PySpark.
- Deep expertise in ETL design, data modelling (Data Vault, Star Schema, Medallion architecture), and query optimization.
- Strong experience with SQL, Python, and Data Pipeline Orchestration.
- Hands-on experience in CI/CD practices, GitHub Actions, and version control tools.
- Expertise in Azure ecosystem (Azure Data Lake Storage, Azure Data Factory, Synapse Analytics, Event Hub, Kafka).
- Knowledge of structured streaming and real-time data processing.
- Experience implementing security and governance best practices for cloud-based data platforms.
Responsibilities:
- Architecture & Strategy:
- Design and develop scalable Lakehouse architectures that enable high-performance data ingestion, processing, and analytics.
- Define and enforce best practices for ETL, data modeling, and data governance within the Azure ecosystem.
- Architect and optimize Medallion Architecture (Bronze, Silver, Gold) for efficient data transformation and analytics.
- Ensure high availability, security, and performance of Azure Databricks environments.
- Define data infrastructure strategies, ensuring alignment with business objectives and scalability requirements.
- Technical Leadership & Development:
- Provide architectural guidance and hands-on development support for ETL pipelines, batch & streaming solutions in Azure Databricks using PySpark and SQL.
- Drive the implementation of Data Vault, Star Schema, and Data Warehouse modeling techniques.
- Implement CI/CD pipelines, version control, and automation to streamline deployment and testing.
- Lead the integration of Azure Data Services such as Azure Data Factory, Azure Synapse, Delta Lake, and Delta Live Tables.
- Apply Chaos Engineering practices to ensure system resilience and fault tolerance.
- Collaboration & Mentorship:
- Work closely with Product Owners, Data Engineers, Data Scientists, and Business Analysts to deliver high-quality data solutions.
- Provide technical mentorship to engineering teams, enabling them to follow best practices in Databricks development, ETL, and data pipeline management.
- Collaborate across departments to design functional and scalable solutions that balance business needs and technical feasibility.
- Analytics & Optimization:
- Conduct performance tuning and optimization of Databricks workloads, Spark jobs, and SQL queries.
- Define and implement monitoring frameworks to track data quality, lineage, and processing performance.
- Design efficient data governance, access control, and security policies for Databricks and Azure environments.
- Lead root-cause analysis and problem-solving for complex data engineering challenges.
Preferred Skills (Bonus Points):
- Experience with Snowflake and BI Tools (Power BI, Tableau).
- Exposure to Graph Databases and Knowledge Graphs.
- Familiarity with Machine Learning or AI-based data solutions in Databricks.
- Experience in Python library development for reusable data processing functions.