SRE - Performance Engineering

Job Title: Site Reliability Engineering - Performance Engineer

Location: Bay Area preferred/Hybrid

Department: DevOps

At WitnessAI, we're at the intersection of innovation and security in AI. We are seeking a Site Reliability Engineer - This role emphasizes deep systems-level performance analysis, tuning, and optimization to ensure the reliability and efficiency of our cloud-based infrastructure. You will drive performance across a tech stack that includes Cloud Infrastructure, Linux, Kubernetes, databases, message queuing systems, AI workloads, and GPUs. The ideal candidate brings a passion for data-driven methodologies, flame graph analysis, and advanced performance debugging to solve complex system challenges.

Key Responsibilities

• Conduct root cause analysis (RCA) for performance bottlenecks using data-driven approaches like flame graphs, heatmaps, and latency histograms.
• Perform detailed kernel and application tracing using tools based on technologies like eBPF, perf, and ftrace to gain insights into system behavior.
• Design and implement performance dashboards to visualize key performance metrics in real-time.
• Recommend Linux and Cloud Server tuning improvements to increase throughput and latency
• Tune Linux systems for workload-specific demands, including scheduler, I/O subsystem, and memory management optimizations.
• Analyze and optimize cloud instance types, EBS volumes, and network configurations for high performance and low latency.
• Improve throughput and latency for message queues (e.g., ActiveMQ, Kafka, SQS, etc) by profiling producer/consumer behavior and tuning configurations.
• Apply profiling tools to analyze GPU utilization and kernel execution times and implement techniques to boost GPU efficiency.
• Optimize distributed training pipelines using industry-standard frameworks.
• Evaluate and reduce training times through mixed precision training, model quantization, and resource-aware scheduling in Kubernetes.
• Work with AI teams to identify scaling challenges and optimize GPU workloads for inference and training.
• Design observability systems for granular monitoring of end-to-end latency, throughput, and resource utilization.
• Implement and leverage modern observability stacks to capture critical insights into application and infrastructure behavior.
• Work with developers to refactor applications for performance and scalability, using profiling tools
• Mentor teams on performance best practices, debugging workflows, and methodologies inspired by leading performance engineers.

Qualifications Required:

• Deep expertise in Linux systems internals (kernel, I/O, networking, memory management) and performance tuning.
• Strong experience with AWS cloud services and their performance optimization techniques.
• Proficiency in performance analysis and load testing tools and other system tracing frameworks.
• Hands-on experience with database tuning, query analysis, and indexing strategies.
• Expertise in GPU workload optimization, and cloud-based GPU instances
• Familiarity with message queuing systems including performance tuning.
• Programming experience with a focus on profiling and tuning
• Strong scripting skills (e.g., Python, Bash) to automate performance measurement and tuning workflows.

Preferred:

• Knowledge of distributed AI/ML training frameworks
• Experience designing and scaling GPU workloads on Kubernetes using GPU-aware scheduling and resource isolation.
• Expertise in optimizing AI inference pipelines.
• Familiarity with Brendan Gregg's methodologies for systems analysis, such as USE (Utilization, Saturation, Errors) and Workload Characterization Frameworks.

Benefits:

• Hybrid work environment
• Competitive salary
• Health, dental, and vision insurance
• 401(k) plan
• Opportunities for professional development and growth
• Generous vacation policy