Sr. Lead Software Engineer, Reliability & Monitoring

At Chick-fil-A, Reliability and Monitoring is a technical function which mixes in influence. Across our 3000+ North American Restaurants, cloud, and private data centers, SREs work with our DevOps teams to introduce and hone SRE principles, establish reliability goals, and develop tooling for operational observability. We are a small team working through many different patterns to bring observability to everyone. SREs at Chick-fil-A collaborate across teams and roles, feed learnings back into the organization, and learn all the ways technology is used in the process.  The team is focused on tooling and enablement rather than traditional SRE roles.

The BUILD team is transforming the way that observability data is captured within Chick-fil-A. In our pursuit of what's next, the delivery and adoption of OpenTelemetry is at the core of our updated observability tooling. In this role, the ideal candidate will work independently across platform, SDK, and application teams to gather requirements, build, offer support, and troubleshoot our observability pipelines.

Our Flexible Future model offers a healthy mix of working in person and virtually, strengthening key elements of the Chick-fil-A culture by fostering collaboration and community.

• Own solution architecture decisions for the team’s product
• Lead delivery and operations of the team’s product, including both individual contribution and support as well as delegated tasks and support to your team’s engineers. 
• We desire our lead engineers to be both leads and engineers, spending about half of their time on leading others and half contributing engineering work themselves.
• Lead, mentor, and assess other staff engineers, exemplifying and teaching best practices, helping to solve complex problems, reviewing code, and sharing stories
• Interview, select, onboard, and oversee contract engineers
• Assist in the design and delivery of products outside of your team, develop tools and processes for use outside of your team, and/or mentor other engineers or especially lead engineers outside of your team
• Guide engineering team in adoption of Chick-fil-A software engineering standards
• Document learnings to share with the broader engineering team(s)
• Ensure clear communication around R+M objectives
• Collaborate broadly across the entire engineering organization
• Oversee other SRE teams to bring best practices or learnings from across the organization to them
• Build internal tooling around operational observability
• Bring a strong mindset of continual improvement
• An aversion to toil and automatable tasks
• Advocate for R+M as a part of engineering culture
• Act as a conduit for Architecture, Security, Tools, and DTT Framework
• Keep abreast of industry changes and evaluate for implementation
• Work independently with DevOps teams to refine running production systems
  
  • Building on-call processes
  • Creating Incident Management and Response procedures
  • Instrumenting for observability and coaching on best practices
  • Monitoring SLIs
• Work to varying degrees with DevOps teams
  
  • Provide consultation on SRE best practices
  • Give guidance on specific topics
  • Oversee groups of dedicated engineers
  • Embed directly with teams 
• Work with teams to define SLOs and error budgets
• Ensure services and systems meet availability needs of customers
• Design and develop software solutions 
• Serve as a model developer in programming languages like Java, Go, Python, and Python
• Exercise skills in infrastructure and deployment services like AWS and
• Kubernetes as well as areas like application security, data analytics, and machine learning

Note - Working in a DevOps model, this opportunity includes both building and running solutions that could require off hours support. This support is shared amongst the team members to cover weekends and weeknights.  The goal is to design for failure and, using cloud-native infrastructure patterns, automate responses to possible issues so they can be worked during normal hours.  

• Bachelor’s Degree or the equivalent combination of education, training and experience from which comparable skills can be acquired
• 4+ years of relevant work experience
• Experience designing complex software solutions
• Experience mentoring and leading a team, including good interpersonal and team collaboration skills
• Broad and deep programming experience in Java, JavaScript, Python, or other comparable languages
• Experience with SQL and data modeling
• Experience with source control systems like Git or Subversion
• Experience implementing application security, software design patterns, and the SDLC
• Proven ability to positively influence the engineering culture and practices in a professional environment
• Significant experience with:
  
• Building and supporting systems
  
• Enterprise cloud providers
  
• Production containerized environments
  
• Excellent written and verbal communication
  
• Experience with CI/CD pipelines
  
• Ability to build strong relationships, collaborate, and influence  diverse groups of engineers and non-technical roles
  
• Ability to influence other engineers without organizational authority

• Bachelor's degree in Computer Science, Information Technology, Computer Engineering, or similar area of study
• 9+ years of relevant work experience, with: 
  • 4+ years supporting a production system on a Devops team
  • 2+ years in cloud platforms such as amazon web services, google cloud, or Microsoft azure
• Experience with test-driven development, continuous integration and deployment, Scrum discipline, or comparable software development practices 
• Deep understanding of AWS architecture 
• Familiarity with version control systems and code merging/branching; specific experience with git desirable 
• Experience working with an agile development methodology featuring sprints, points estimation, and daily standups 
• Experience in design, data collection, and data analysis
• Experience with Unix/Linux
• Experience with Kubernetes
• Experience with OpenTelemetry, Fluentd, Grafana Alloy, or Datadog Agent (DDOT preferred)