AWS Tech Lead with Databricks
Apply NowCompany: LingaTech, Inc.
Location: Raritan, NJ 08869
Description:
Role: AWS Tech Lead with Databricks
Duration: Long Term
Location: Raritan, NJ- fully onsite
Overview:
An AWS Tech Lead with Databricks SME (Subject Matter Expert) knowledge is responsible for designing, deploying, and managing cloud-based infrastructure on AWS and supporting data solutions using Databricks. The role involves leading a team to ensure scalable, secure, and cost-efficient cloud services while integrating data pipelines and analytics solutions.
Roles and Responsibilities:
Cloud Infrastructure Management:
- Design, implement, and maintain AWS cloud solutions, including VPCs, EC2 instances, S3 storage, RDS, and networking configurations.
- Ensure high availability, disaster recovery, and backup solutions for cloud environments.
- Implement Infrastructure as Code (IaC) using tools like Terraform, AWS CloudFormation, or CDK.
Data Solutions with Databricks:
- Serve as a Databricks SME, guiding the architecture and implementation of ETL pipelines, data lakes, and machine learning workflows using Databricks on AWS.
- Optimize Databricks clusters for performance, cost, and security.
- Integrate Databricks with other AWS services like S3, Redshift, and Glue for seamless data processing.
Leadership and Team Collaboration:
- Lead a team of cloud engineers, providing mentorship and technical guidance.
- Collaborate with cross-functional teams including data scientists, analysts, and developers to understand requirements and translate them into scalable cloud and data architectures.
- Manage project timelines, ensuring timely delivery of cloud and data solutions.
Security and Compliance:
- Implement best practices for cloud security, including identity and access management (IAM), encryption, and VPC security.
- Ensure compliance with industry standards (e.g., GDPR, HIPAA) and internal security policies.
- Conduct regular security audits and optimize cloud configurations to reduce risk.
Performance Optimization and Monitoring:
- Set up monitoring, logging, and alerting using AWS CloudWatch, Datadog, or similar tools.
- Analyze system performance and provide recommendations for improvement.
- Troubleshoot complex issues related to cloud infrastructure, data pipelines, and Databricks clusters.
Technical Skills:
- Strong expertise in AWS services (EC2, S3, Lambda, RDS, VPC, CloudFormation, IAM, etc.).
- Proficiency with Databricks, including building and managing data pipelines, data lakes, and machine learning models.
- Experience with Infrastructure as Code (IaC) tools like Terraform or AWS CloudFormation.
- Knowledge of big data technologies like Apache Spark, Hadoop, or Delta Lake.
- Proficient in programming and scripting languages (e.g., Python, SQL, Scala).
- Experience with CI/CD pipelines for deploying cloud infrastructure.
Soft Skills:
- Strong leadership and project management skills.
- Excellent problem-solving and analytical skills.
- Ability to work collaboratively with diverse teams.
- Strong communication skills, with the ability to explain technical concepts to non-technical stakeholders.
Certifications:
Required:
- AWS Certified Solutions Architect (Associate or Professional)
- Databricks Certified Associate Developer for Apache Spark
Preferred:
- AWS Certified DevOps Engineer - Professional
- AWS Certified Big Data - Specialty or AWS Certified Data Analytics - Specialty
- Databricks Certified Professional Data Engineer
Duration: Long Term
Location: Raritan, NJ- fully onsite
Overview:
An AWS Tech Lead with Databricks SME (Subject Matter Expert) knowledge is responsible for designing, deploying, and managing cloud-based infrastructure on AWS and supporting data solutions using Databricks. The role involves leading a team to ensure scalable, secure, and cost-efficient cloud services while integrating data pipelines and analytics solutions.
Roles and Responsibilities:
Cloud Infrastructure Management:
- Design, implement, and maintain AWS cloud solutions, including VPCs, EC2 instances, S3 storage, RDS, and networking configurations.
- Ensure high availability, disaster recovery, and backup solutions for cloud environments.
- Implement Infrastructure as Code (IaC) using tools like Terraform, AWS CloudFormation, or CDK.
Data Solutions with Databricks:
- Serve as a Databricks SME, guiding the architecture and implementation of ETL pipelines, data lakes, and machine learning workflows using Databricks on AWS.
- Optimize Databricks clusters for performance, cost, and security.
- Integrate Databricks with other AWS services like S3, Redshift, and Glue for seamless data processing.
Leadership and Team Collaboration:
- Lead a team of cloud engineers, providing mentorship and technical guidance.
- Collaborate with cross-functional teams including data scientists, analysts, and developers to understand requirements and translate them into scalable cloud and data architectures.
- Manage project timelines, ensuring timely delivery of cloud and data solutions.
Security and Compliance:
- Implement best practices for cloud security, including identity and access management (IAM), encryption, and VPC security.
- Ensure compliance with industry standards (e.g., GDPR, HIPAA) and internal security policies.
- Conduct regular security audits and optimize cloud configurations to reduce risk.
Performance Optimization and Monitoring:
- Set up monitoring, logging, and alerting using AWS CloudWatch, Datadog, or similar tools.
- Analyze system performance and provide recommendations for improvement.
- Troubleshoot complex issues related to cloud infrastructure, data pipelines, and Databricks clusters.
Technical Skills:
- Strong expertise in AWS services (EC2, S3, Lambda, RDS, VPC, CloudFormation, IAM, etc.).
- Proficiency with Databricks, including building and managing data pipelines, data lakes, and machine learning models.
- Experience with Infrastructure as Code (IaC) tools like Terraform or AWS CloudFormation.
- Knowledge of big data technologies like Apache Spark, Hadoop, or Delta Lake.
- Proficient in programming and scripting languages (e.g., Python, SQL, Scala).
- Experience with CI/CD pipelines for deploying cloud infrastructure.
Soft Skills:
- Strong leadership and project management skills.
- Excellent problem-solving and analytical skills.
- Ability to work collaboratively with diverse teams.
- Strong communication skills, with the ability to explain technical concepts to non-technical stakeholders.
Certifications:
Required:
- AWS Certified Solutions Architect (Associate or Professional)
- Databricks Certified Associate Developer for Apache Spark
Preferred:
- AWS Certified DevOps Engineer - Professional
- AWS Certified Big Data - Specialty or AWS Certified Data Analytics - Specialty
- Databricks Certified Professional Data Engineer