Software Engineer II - Platform & Infrastructure

📋 Job Overview

Abnormal AI is looking for an experienced and driven Platform & Infra software engineer to join the PI team. Join us and help build the platforms that power Abnormal's growth.

🏢 About Abnormal AI

Abnormal AI is a company focused on building platforms that power growth through technology and innovation.

🎯 The Role

You'll work across the Prometheus, Chronosphere, and Grafana stack to ensure engineers can see what their systems are doing in real time — building dashboards, managing metric pipelines at scale, operating the PagerDuty alerting pipeline, and driving cost-efficient observability across all production environments (US, EU, and GovCloud).

✅ Key Responsibilities

• Own the observability stack (Prometheus, Chronosphere, Grafana, PagerDuty) that every team relies on to detect, diagnose, and resolve production issues.
• Design platforms and developer tooling that remove friction — reducing deployment times, simplifying pipeline authoring, and letting product teams focus on building rather than firefighting.
• Drive SLAs and SLOs for critical shared infrastructure ensuring the systems behind our products are resilient and cost-efficient.
• Take ownership of 1-3 key services within Observability (Prometheus, Chronosphere, Grafana, PagerDuty pipeline) or Data Infra (Airflow, Spark) and be accountable for their reliability, performance, and evolution.
• Participate in on-call rotations — triage, diagnose, and resolve production issues independently, building deep operational knowledge of the systems you own.
• Improve system resilience by converting runbooks into automated solutions, refining SLAs/SLOs, and proactively identifying performance bottlenecks and failure modes.
• Assume ownership of the reliability of everything you build, including comprehensive unit tests, integration testing, and observability instrumentation.
• Build platforms, tooling, and APIs that make it easier for other engineering teams to ship — whether that's faster pipeline deployments, better dashboards, or simpler alerting configuration.
• Partner with internal customers (product and engineering teams) to understand their needs and translate them into scalable platform capabilities.
• Communicate effectively in an async-first, distributed environment — proactively providing updates, discussing challenges, and proposing solutions without prompting.
• Mentor junior engineers on the team, helping them ramp up on service operations and development practices.
• Raise the bar of engineering excellence through code reviews, knowledge sharing, design discussions, and contributing to team best practices.

📌 Required Qualifications

• Backend Engineering & Distributed Systems (4+ years)
• 4+ years of hands-on backend engineering experience designing, building, and operating production-grade distributed systems
• Strong proficiency in Python — the primary language for Airflow DAGs, platform services, and automation tooling
• Working proficiency in Golang — used for high-performance infrastructure components, metric pipelines, and platform services
• Experience building systems that process data at scale — whether metric ingestion pipelines, stream/batch processing, or high-throughput API services
• Demonstrated experience owning a service or platform end-to-end — from technical design through production deployment, monitoring, and iteration
• Comfortable balancing feature development with operational responsibilities: you've shipped features and kept them running reliably at scale
• Experience writing technical design documents that articulate trade-offs, propose solutions, and get buy-in from peers and tech leads
• Track record of breaking down ambiguous problems into concrete, deliverable milestones
• Experience with fault tolerance patterns — retries, circuit breakers, graceful degradation, backpressure — and knowing when to apply each
• Proven incident response capability: you've been on-call, diagnosed production issues under pressure, and driven them to resolution
• Strong testing discipline — unit tests, integration tests, and an understanding of what to test and how to keep test suites maintainable
• Ability to design systems with a forward-looking perspective — thinking about how your architecture handles 10x growth, multi-region deployment, and evolving requirements
• Ability to contribute to and influence cross-team technical direction — you're not just implementing specs, you're shaping the solution
• Async-first communication excellence — strong written communication skills for design docs, Slack discussions, PR reviews, and status updates across time zones
• Proactive communicator — you surface blockers early, share con
• Solid understanding of monitoring, alerting, and observability principles — you've instrumented services, set up dashboards, defined SLIs/SLOs, or triaged production incidents using metrics and logs

⭐ Desirable Experience

• Hands-on experience with Prometheus — PromQL queries, recording rules, alerting rules, relabeling configs, and understanding metric cardinality challenges at scale
• Experience with Grafana — building dashboards, templating, managing datasources, and creating meaningful visualizations for operational and business metrics
• Familiarity with commercial observability platforms like Chronosphere, Datadog, New Relic, or Honeycomb — understanding trade-offs between self-hosted and managed solutions
• Experience designing or operating an alerting pipeline — PagerDuty, OpsGenie, or similar — including alert routing, escalation policies, and reducing noise/alert fatigue
• Familiarity with AWS services — EC2, ECS, EKS, S3, RDS, IAM, CloudWatch, Lambda, SQS/SNS — and understanding how to architect cost-effective, secure cloud infrastructure
• Experience with Kubernetes (K8s) — deploying and operating workloads, understanding pods/services/deployments, Helm charts, and debugging cluster-level issues
• Exposure to Infrastructure-as-Code tools — Terraform, Pulumi, or CloudFormation — and understanding the value of declarative infrastructure management
• Experience with CI/CD pipelines — GitHub Actions, Jenkins, or similar — and optimizing build/deploy times for platform services
• Experience with Django or similar Python web frameworks — building APIs, managing migrations, and understanding ORM performance characteristics
• Familiarity with gRPC or protobuf for inter-service communication in a microservices architecture
• Experience leading a small team (2-4 engineers) to build a feature or component from scratch — scoping, task breakdown, code reviews, and delivery management
• Experience building internal developer platforms or tooling — CLIs, SDKs, self-service portals, or automation that improved developer productivity
• Track record of reducing operational toil — automating runbooks

🎁 Benefits

Abnormal AI is an equal opportunity employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability, protected veteran status or other characteristics protected by law.

🛂 Visa & Eligibility

Abnormal AI uses AI-assisted tools to help our recruiting team prepare for candidate interviews. These tools analyze resume content and role requirements to suggest interview questions and areas for the interviewer to explore. They do not make hiring decisions or screen candidates automatically. Every decision about a candidacy is made by a person. Further, if your application is successful and Abnormal AI makes a conditional offer of employment, we will carry out pre-employment checks which must be successfully completed to progress to a final offer. All processes and pre-employment checks are in line with prevailing legislation and Abnormal AI's policies relevant to our security and privacy standards.