GitOpsCon North America 2021

Building and managing data-intensive microservices applications have their challenges. Cellular architecture & Gitops together help create an independently deployable, manageable, and observable solution. We moved from running multiple expensive non-prod inconsistent environments to testing in production. Weave Flux, Istio, and Crossplane helped us build multi-cloud, multi-cluster declarative environments in minutes.  With the new deployment model and custom tooling, devs can get a miniaturized version of the SaaS running on their laptops. The devs can pick and choose which components and versions they want to run locally.  The CI/CD metrics have improved dramatically as the integration & E2E tests run against real apps and not mocks. The system spins up real cloud-native solutions for production deployments and falls back to containerized workloads for non-production.  Adding support for new regions has come down to days of work from months.

GitOps is typically seen as a practice to get your things in order for infrastructure changes and infrastructure configuration management. However, there is another interesting use case for GitOps where it is used to chaos test the large scale deployments when changes happen to the application. Complete reliability as such is a complex subject and it requires carefully designed chaos engineering practices and SRE focus. In large scale systems where Chaos Engineering is scaled up, the chaos scenarios also become big enough to be maintained and applied. One solution is to maintain them and apply them through GitOps.  In this lightning talk, I discuss a case study of how GitOps is used in a large customer environment to automate Chaos Engineering using LitmusChaos and FluxCD. The challenge of managing chaos experiments is presented when there are multiple team members and teams are involved. Then I discuss the schematic representation of how GitOps was structured between the application change and chaos experiments. Finally I discuss and summarise the GitOps best practices used to overcome these challenges.  

Mae Large and Pinky Ravi have been a part of the GitOps team at State Farm since its inception in February 2019. GitOps at State Farm is adopted in all our strategic cloud platforms - AWS, Multi-tenant Kubernetes (on-prem), and Cloud Foundry (on-prem). This talk will walkthrough how we scaled GitOps in our large organization. The GitOps team is 5-person team. We have successfully rolled out GitOps, supported our customers, and continue to advance the offering towards observability and resiliency of the applications or products using GitOps.   We will give an insight on how our typical day is like and also highlight the different bumps and challenges (including outages) we’ve faced and how these all became opportunities for us to continuously improve.