Azure Data Engineer/ Architect with Fabric
Apply NowCompany: SysMind Tech
Location: Jersey City, NJ 07305
Description:
Position: Azure Data Engineer/ Architect with Fabric
Location: New Jersey, NJ (Onsite)
Duration: Long Term
Job Description:
Key Responsibilities:
Data Architecture:
Integration Design:
Lakehouse Architecture:
Data Governance:
Microsoft Fabric Expertise:
Data Integration:
Data Pipeline Management:
Azure Databricks Experience:
Data Engineering:
Delta Lake:
Cluster Management:
Integration with Azure Services:
Data Governance:
Security Framework:
Pipeline Development:
Performance Optimization:
Location: New Jersey, NJ (Onsite)
Duration: Long Term
Job Description:
- Deep expertise in modern data architecture, with specific experience in Microsoft's data platform and Delta Lake architecture.
- 6+ years of experience in data architecture and engineering.
- Required 2+ years hands-on experience with Azure Data bricks / ADF and Spark.
- Required recent experience with Microsoft Fabric platform.
Key Responsibilities:
Data Architecture:
- Design end-to-end data architecture leveraging Microsoft Fabric's capabilities.
- Design data flows within the Microsoft Fabric environment.
- Implement OneLake storage strategies.
- Configure Synapse Analytics workspaces.
- Establish Power BI integration patterns.
Integration Design:
- Architect data integration patterns with analytics using Azure Data Factory and Microsoft Fabric.
- Implement medallion architecture (Bronze/Silver/Gold layers).
- Ability to configure real-time data ingestion patterns.
- Establish data quality frameworks.
Lakehouse Architecture:
- Implement modern data lakehouse architecture using Delta Lake, ensuring data reliability and performance.
Data Governance:
- Establish data governance frameworks incorporating Microsoft Purview for data quality, lineage, and compliance.
Microsoft Fabric Expertise:
Data Integration:
- Combining and cleansing data from various sources.
Data Pipeline Management:
- Creating, orchestrating, and troubleshooting data pipelines.
- Analytics Reporting: Building and delivering detailed reports and dashboards to derive meaningful insights from large datasets.
- Data Visualization Techniques: Representing data graphically in impactful and informative ways.
- Optimization and Security: Optimizing queries, improving performance, and securing data
Azure Databricks Experience:
- Apache Spark Proficiency: Utilizing Spark for large-scale data processing and analytics.
Data Engineering:
- Building and managing data pipelines, including ETL (Extract, Transform, Load) processes.
Delta Lake:
- Implementing Delta Lake for data versioning, ACID transactions, and schema enforcement.
Cluster Management:
- Configuring and managing Databricks clusters for optimized performance. (Ex: autoscaling and automatic termination)
Integration with Azure Services:
- Integrating Databricks with other Azure services like Azure Data Lake, Azure SQL Database, and Azure Synapse Analytics.
Data Governance:
- Implementing data governance practices using Unity Catalog and Microsoft Purview
Security Framework:
- Design and implement security patterns aligned with federal and state requirements for sensitive data handling.
- Implement row-level security.
- Configure Microsoft Purview policies.
- Establish data masking for sensitive information.
- Design audit logging mechanisms.
Pipeline Development:
- Design scalable data pipelines using Azure Databricks for ETL/ELT processes and real-time data integration.
Performance Optimization:
- Implement performance tuning strategies for large-scale data processing and analytics workloads.
- Optimize Spark configurations.
- Implement partitioning strategies.
- Design caching mechanisms.
- Establish monitoring frameworks.