Sr. Data Engineer
Apply NowCompany: McLane Global
Location: Houston, TX 77084
Description:
McLane Global is seeking a Sr. Data Engineer to join our dynamic team. In this role, you will play a crucial part in shaping our data strategy and elevating our data infrastructure to new heights. As a leading provider in our industry, we recognize that data is at the heart of our operations and decisions, and we need a skilled engineer to help us harness its full potential. You will work closely with cross-functional teams to design, develop, and maintain robust data pipelines and architectures, enabling the organization to derive actionable insights and drive innovative solutions. We are looking for a candidate who is passionate about data engineering and has a proven track record of building scalable and efficient data systems. The ideal candidate will possess deep technical knowledge of data engineering practices, be adept at utilizing various ETL tools, and have hands-on experience with cloud platforms. If you thrive in a fast-paced environment and are excited about leveraging data to fuel business growth, we want to hear from you. Join us at McLane Global, where you'll have the opportunity to work with cutting-edge technologies and contribute to exciting projects that make a real difference in our industry.
Responsibilities
Design, build, and optimize scalable data pipelines, warehouses, and processing systems using Azure Synapse, Databricks, Spark Pools, and Microsoft Fabric to ensure efficient data flow and accessibility.
Develop and manage data integrations across multiple platforms, including D365, Dual Write, Azure Data Studio, and Power Platform, ensuring seamless data exchange between enterprise systems.
Implement and optimize EDI integrations, API-based data exchanges, and event-driven architectures to facilitate real-time and batch data processing.
Leverage Azure Data Services, including Azure Data Factory, Azure Data Lake, and Azure Storage, to implement cloud-based data solutions that optimize performance, cost, and scalability.
Design and maintain reports and dashboards using Power BI and SSRS, integrating complex data models to provide actionable insights for business decision-making.
Implement data governance, security, and compliance policies, ensuring adherence to industry standards and regulations such as GDPR, HIPAA, and SOC2.
Develop and maintain ETL/ELT processes using Azure Data Factory, Synapse Pipelines, and Databricks, ensuring efficient data transformation and movement across systems.
Manage real-time and batch data processing with Apache Spark, Azure Stream Analytics, and Event Hubs to support high-volume data workflows.
Build and optimize Data Lakehouse Architectures, ensuring structured and unstructured data can be effectively processed and analyzed.
Perform database performance tuning, query optimization, and indexing strategies in Azure SQL, Synapse, and Cosmos DB for high-performance data access.
Develop and manage CI/CD pipelines for data workflows using Azure DevOps and Git, ensuring automated deployment and version control of data assets.
Work with Data Scientists and Analysts to enable machine learning and advanced analytics capabilities by structuring data for AI/ML models.
Troubleshoot and resolve complex data pipeline failures, latency issues, and data integrity problems across distributed cloud environments.
Implement monitoring and alerting mechanisms using Azure Monitor, Log Analytics, and Application Insights to proactively detect and resolve performance bottlenecks.
Collaborate with business leaders, department heads, and cross-functional teams to define data strategies and deliver scalable, high-performance analytics solutions.
Requirements
Strong hands-on experience with Microsoft Azure Data Services, including Azure Synapse, Databricks, Data Factory, Data Lake, and Azure SQL.
Extensive experience in ETL/ELT development, data integration, and data transformation using Azure Data Factory, Synapse Pipelines, and Databricks.
Proficiency in Apache Spark, Spark Pools, and Azure Stream Analytics for real-time and batch data processing.
Strong coding skills in Python, SQL, or PowerShell for data processing and automation.
Expertise in Azure SQL, Synapse Analytics, Cosmos DB, and NoSQL databases, including query optimization and performance tuning.
Experience in API-based data integrations, EDI (Electronic Data Interchange), and working with REST/SOAP APIs for seamless system communication.
Knowledge of data privacy, governance, and security best practices, including compliance with GDPR, HIPAA, and SOC2.
Experience with Power BI, SSRS, and other BI tools for designing interactive reports and dashboards.
Hands-on experience with CI/CD pipelines using Azure DevOps, Git, and Terraform for automating deployments and version control.
Familiarity with Azure Monitor, Log Analytics, and Application Insights to track system health and optimize data workflows.
Strong analytical and debugging skills to identify, diagnose, and resolve data pipeline failures in distributed cloud environments.
Ability to work cross-functionally with business leaders, data scientists, and developers, translating business requirements into scalable data solutions.
Azure certifications such as Microsoft Certified: Azure Data Engineer Associate (DP-203) or equivalent are preferred.
Proficient knowledge of Lean tools and methodologies (e.g., 5S, PDCA, Process Mapping, Value Stream Mapping, Kaizen, Root Cause Analysis).
Strong analytical and problem-solving abilities, with a data-driven approach to decision-making.
Exceptional interpersonal and communication skills, with the ability to influence and engage employees at all organizational levels. Proficiency in data analysis software and Microsoft Excel, PowerPoint, Word, and Visio.
Benefits
Competitive base compensation
Full Benefits including Medical, Dental and Vision effective on the first day of employment
401(k) with Employer Match (quarterly enrollment)
Paid Holidays (no waiting period to receive holiday pay)
Paid vacation, personal, and sick days effective on first day of employment
All offers are contingent upon passing a background check and drug screening
No company visa sponsorships
No phone call or walk in inquiries for this position, apply online to be considered
Responsibilities
Design, build, and optimize scalable data pipelines, warehouses, and processing systems using Azure Synapse, Databricks, Spark Pools, and Microsoft Fabric to ensure efficient data flow and accessibility.
Develop and manage data integrations across multiple platforms, including D365, Dual Write, Azure Data Studio, and Power Platform, ensuring seamless data exchange between enterprise systems.
Implement and optimize EDI integrations, API-based data exchanges, and event-driven architectures to facilitate real-time and batch data processing.
Leverage Azure Data Services, including Azure Data Factory, Azure Data Lake, and Azure Storage, to implement cloud-based data solutions that optimize performance, cost, and scalability.
Design and maintain reports and dashboards using Power BI and SSRS, integrating complex data models to provide actionable insights for business decision-making.
Implement data governance, security, and compliance policies, ensuring adherence to industry standards and regulations such as GDPR, HIPAA, and SOC2.
Develop and maintain ETL/ELT processes using Azure Data Factory, Synapse Pipelines, and Databricks, ensuring efficient data transformation and movement across systems.
Manage real-time and batch data processing with Apache Spark, Azure Stream Analytics, and Event Hubs to support high-volume data workflows.
Build and optimize Data Lakehouse Architectures, ensuring structured and unstructured data can be effectively processed and analyzed.
Perform database performance tuning, query optimization, and indexing strategies in Azure SQL, Synapse, and Cosmos DB for high-performance data access.
Develop and manage CI/CD pipelines for data workflows using Azure DevOps and Git, ensuring automated deployment and version control of data assets.
Work with Data Scientists and Analysts to enable machine learning and advanced analytics capabilities by structuring data for AI/ML models.
Troubleshoot and resolve complex data pipeline failures, latency issues, and data integrity problems across distributed cloud environments.
Implement monitoring and alerting mechanisms using Azure Monitor, Log Analytics, and Application Insights to proactively detect and resolve performance bottlenecks.
Collaborate with business leaders, department heads, and cross-functional teams to define data strategies and deliver scalable, high-performance analytics solutions.
Requirements
Strong hands-on experience with Microsoft Azure Data Services, including Azure Synapse, Databricks, Data Factory, Data Lake, and Azure SQL.
Extensive experience in ETL/ELT development, data integration, and data transformation using Azure Data Factory, Synapse Pipelines, and Databricks.
Proficiency in Apache Spark, Spark Pools, and Azure Stream Analytics for real-time and batch data processing.
Strong coding skills in Python, SQL, or PowerShell for data processing and automation.
Expertise in Azure SQL, Synapse Analytics, Cosmos DB, and NoSQL databases, including query optimization and performance tuning.
Experience in API-based data integrations, EDI (Electronic Data Interchange), and working with REST/SOAP APIs for seamless system communication.
Knowledge of data privacy, governance, and security best practices, including compliance with GDPR, HIPAA, and SOC2.
Experience with Power BI, SSRS, and other BI tools for designing interactive reports and dashboards.
Hands-on experience with CI/CD pipelines using Azure DevOps, Git, and Terraform for automating deployments and version control.
Familiarity with Azure Monitor, Log Analytics, and Application Insights to track system health and optimize data workflows.
Strong analytical and debugging skills to identify, diagnose, and resolve data pipeline failures in distributed cloud environments.
Ability to work cross-functionally with business leaders, data scientists, and developers, translating business requirements into scalable data solutions.
Azure certifications such as Microsoft Certified: Azure Data Engineer Associate (DP-203) or equivalent are preferred.
Proficient knowledge of Lean tools and methodologies (e.g., 5S, PDCA, Process Mapping, Value Stream Mapping, Kaizen, Root Cause Analysis).
Strong analytical and problem-solving abilities, with a data-driven approach to decision-making.
Exceptional interpersonal and communication skills, with the ability to influence and engage employees at all organizational levels. Proficiency in data analysis software and Microsoft Excel, PowerPoint, Word, and Visio.
Benefits
Competitive base compensation
Full Benefits including Medical, Dental and Vision effective on the first day of employment
401(k) with Employer Match (quarterly enrollment)
Paid Holidays (no waiting period to receive holiday pay)
Paid vacation, personal, and sick days effective on first day of employment
All offers are contingent upon passing a background check and drug screening
No company visa sponsorships
No phone call or walk in inquiries for this position, apply online to be considered