Observability Software Engineer, AI Infrastructure

As a member of Tesla's "Insane Visibility" team, you will design, implement & maintain end-to-end observability across our AI Infrastructure stack and develop the framework to benchmark performance & processing of pipelines. You'll be responsible for building dashboards, alerts & monitoring necessary for Autopilot & AI teams to address observability issues in our FSD, Robotaxi & Tesla Bot applications, ensuring these programs run smoothly throughout the full infrastructure stack.

Responsibilities

• Design, develop and maintain observability solutions & tools, including monitoring, logging, and alerting systems, to improve system visibility and performance
• Create dashboards & automated alerts using tools such as Grafana, Prometheus, Splunk, Catchpoint to enhance monitoring frameworks & ensure proactive issue detection and resolution
• Analyze system metrics and logs to identify bottlenecks, optimize application performance, and ensure system reliability end-to-end while scaling
• Partner with developers, DevOps engineers, and AI Infra teams to integrate observability best practices into the development and deployment lifecycle
• Assist in troubleshooting and resolving production issues by leveraging observability data to identify root causes and implement preventative measures
• Develop scripts or workflows to automate routine tasks and improve observability tool integrations
• Create and maintain documentation for observability tools, processes, and workflows to ensure knowledge sharing and accessibility

Requirements

• 3+ years of experience in software engineering, DevOps, or SRE roles with a focus on observability or monitoring
• Proficiency in monitoring and visualization tools (e.g., Prometheus, Grafana, Splunk, Catchpoint)
• Strong analytical and troubleshooting skills with a focus on system performance and reliability
• Working knowledge for High performance computing, Slurm, GPU architecture & Networking
• Working knowledge of logging systems and distributed tracing frameworks such as OpenTelemetry
• Expertise in scripting languages (e.g., Python, Bash) and familiarity with configuration management tools (e.g., Terraform, Ansible)
• Experience with containerized environments (e.g., Docker, Kubernetes) and cloud platforms (e.g., AWS, Azure)
• Strong analytical and troubleshooting skills with a focus on system performance and reliability
• Excellent verbal and written communication skills, with the ability to collaborate effectively across teams
• Bachelor's Degree in Computer Science, Software Engineering, or a related field, or equivalent experience

Compensation and Benefits
Benefits

Along with competitive pay, as a full-time Tesla employee, you are eligible for the following benefits at day 1 of hire:

• Aetna PPO and HSA plans > 2 medical plan options with $0 payroll deduction
• Family-building, fertility, adoption and surrogacy benefits
• Dental (including orthodontic coverage) and vision plans, both have options with a $0 paycheck contribution
• Company Paid (Health Savings Account) HSA Contribution when enrolled in the High Deductible Aetna medical plan with HSA
• Healthcare and Dependent Care Flexible Spending Accounts (FSA)
• 401(k) with employer match, Employee Stock Purchase Plans, and other financial benefits
• Company paid Basic Life, AD&D, short-term and long-term disability insurance
• Employee Assistance Program
• Sick and Vacation time (Flex time for salary positions), and Paid Holidays
• Back-up childcare and parenting support resources
• Voluntary benefits to include: critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insurance
• Weight Loss and Tobacco Cessation Programs
• Tesla Babies program
• Commuter benefits
• Employee discounts and perks program
Expected Compensation

$133,440 - $292,800/annual salary + cash and stock awards + benefits

Pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, and experience. The total compensation package for this position may also include other elements dependent on the position offered. Details of participation in these benefit plans will be provided if an employee receives an offer of employment.