Note: The job is a remote job and is open to candidates in USA. Stanford University is seeking a Senior Site Reliability Engineer to join the Data Management team at the Vera C. Rubin Observatory. The role involves ensuring the reliability of the Prompt Processing Framework, which is responsible for real-time data processing and alert distribution for astronomical events.
Responsibilities
- Ensure, through both architecture and practice, the reliable operation of the near-real-time data processing pipeline and timely delivery of alerts to downstream brokers
- Design and develop software that reduces operational risk and improves system resilience, scalability, and usability, including addressing failure modes, error handling, and contention in shared resources
- Improve system performance and resilience by applying architectural and systems-level optimizations to increase throughput and reduce end-to-end latency
- Operate DevOps-oriented continuous deployment of services using modern distributed systems tooling and development practices (e.g., Kubernetes, Helm, ArgoCD, Kafka, Redis)
- Develop monitoring dashboards and alerts for the prompt processing service and work with teammates to design and implement a sustainable on-call rotation that provides coverage during the start of observing hours in Chile (typically 2-5pm Pacific Time), with limited off-hours responsibility
- Define KPIs and metrics for observability and accountability of the pipeline
- Participate in the collective engineering activities of the team, including performing code reviews, acting as a troubleshooting buddy, participating in design discussions, and writing documentation to effectively capture and communicate architectural and implementation choices
- Collaborate with members of the Data Management team to identify opportunities to improve tools, workflows, and operational practices
- Share responsibility with the broader team for the overall success of the Data Management system, beyond the Prompt Processing Framework
Skills
- Bachelor's degree and eight years of relevant experience, or a combination of education and relevant experience designing and operating distributed systems at-scale in production environments
- Experience working in an SRE, DevOps, or data-intensive systems role, with responsibility for building, operating, and improving robust services
- Experience engaging with modern production infrastructure (e.g., containerized services, messaging systems, and databases; see above for our current tech stack), with the ability to learn and apply new tools quickly in a production environment
- Familiarity with contemporary distributed service architectures, including service-to-service communication patterns, common failure modes, and system behavior under load and scale
- Experience working with large-scale datasets or high-throughput data processing systems, and an understanding of the operational challenges that come with data volume and velocity
- Ability to communicate clearly with engineers and scientists from diverse backgrounds, including explaining technical concepts, participating in design discussions, and documenting systems and decisions
- Comfort working with a high degree of autonomy, taking ownership of technical decisions and execution, while being supported by an experienced team with clear priorities and goals
- Fluency in at least one modern programming language (Python preferred) with experience working across the boundary between software engineering and operations
Company Overview