Kafka Platform DevOps Engineer

Job Description: Kafka Platform DevOps Engineer

Job Summary

We are seeking a highly skilled Kafka Platform DevOps Engineer with strong experience in Ansible, Terraform, and Azure Cloud to join our DevOps team. The ideal candidate will be responsible for the deployment, configuration, and management of Kafka platforms, ensuring high availability, performance, and scalability in both on-premise and cloud environments.

Key Responsibilities

• Deployment and Configuration:
• Design, deploy, and configure Kafka clusters using Ansible and Terraform.
• Implement automated deployment and configuration management for Kafka and related infrastructure.
• Ensure consistency and reliability of Kafka deployments across multiple environments (on-premise and Azure Cloud).
• Infrastructure Management:
• Manage and maintain Kafka clusters on Azure Cloud and on-premise environments.
• Implement infrastructure as code (IaC) solutions using Terraform for efficient and scalable infrastructure management.
• Optimize Kafka configurations for performance, scalability, and reliability.
• Monitoring and Maintenance:
• Monitor the health and performance of Kafka clusters using monitoring tools (Prometheus, Grafana, etc.).
• Perform regular maintenance activities, including upgrades, patching, and backups.
• Implement best practices for Kafka administration and maintenance.
• Automation and Scripting:
• Develop and maintain automation scripts using Ansible for Kafka deployment, configuration, and management tasks.
• Automate operational tasks, monitoring, and alerting for Kafka clusters.
• Integrate Kafka management with CI/CD pipelines using Azure DevOps.
• Collaboration and Communication:
• Collaborate with development, operations, and infrastructure teams to ensure seamless integration and operation of Kafka services.
• Participate in on-call rotation to provide 24/7 support for critical production systems.
• Communicate effectively with stakeholders regarding deployment status, incident resolution, and preventive measures.

Qualifications

• Education:
• Bachelor's degree in Computer Science, Information Technology, or a related field, or equivalent work experience.
• Experience:
• Minimum of 3-5 years of experience in DevOps, IT operations, or a similar role.
• At least 2 years of hands-on experience managing and supporting Kafka clusters in production environments.
• Proven experience with Ansible and Terraform for automation and infrastructure management.
• Strong experience with Azure Cloud services and infrastructure management.
• Technical Skills:
• Deep knowledge of Apache Kafka architecture, including brokers, topics, partitions, and consumers.
• Proficiency in Kafka ecosystem tools, such as Kafka Connect, Kafka Streams, and Schema Registry.
• Experience with monitoring and logging tools, such as Prometheus, Grafana, Splunk, or ELK stack.
• Expertise in scripting languages, such as Python, Bash, or PowerShell.
• Strong experience with infrastructure as code (IaC) tools, particularly Terraform.
• Knowledge of containerization and orchestration technologies, such as Docker and Kubernetes.
• Experience with CI/CD tools and pipelines, particularly Azure DevOps.
• Soft Skills:
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.
• Ability to work independently and as part of a team in a fast-paced, dynamic environment.
• Detail-oriented with a focus on continuous improvement and operational excellence.

Preferred Qualifications

• Certifications:
• Relevant certifications in Apache Kafka, Azure Cloud, Ansible, or Terraform.
• Experience:
• Experience with hybrid cloud environments.
• Familiarity with other cloud platforms (AWS, GCP) is a plus.