TLDR;
The video is a reflection on the speaker's eight years at Atlassian, detailing the projects they built, the technologies used, and the non-technical skills they developed. The speaker discusses building an Open Service Broker, implementing Envoy proxies for load balancing, and the AWS infrastructure that supported these systems. They also touch on the challenges of maintaining software over the long term, managing personality conflicts, and the difficulties of mentoring.
- Building an Open Service Broker for self-service provisioning of load balancing.
- Implementing Envoy proxies and a management server for dynamic configuration.
- Developing AWS infrastructure and machine images (AMIs) to support the proxy deployment.
- Centralizing logic and handling concerns at the edge to improve performance and security.
- Developing diplomacy skills, conflict resolution, and mentoring abilities.
Intro [0:00]
The speaker introduces the video as a reflection on their eight years at Atlassian, prompted by recent layoffs. They aim to share their experiences, focusing on interesting projects and lessons learned, both technical and non-technical. The video is structured into chapters for easy navigation.
Interview process [0:58]
The speaker recalls their interview process at Atlassian, which included a coding quiz, technical interviews, and a values interview. The technical interviews involved analyzing a white paper on custom domains by Cloudflare and troubleshooting a real incident involving a denial-of-service attack. In the values interview, the speaker was asked about their goals for the first year, which led to a discussion about building a self-service load balancing application for internal developers.
Starting at Atlassian [4:16]
Upon joining Atlassian, the speaker describes the initial experience as "drinking from the fire hose" due to the overwhelming amount of information. Their first task was to build the self-service load balancing application discussed during the interview.
Building an Open Service Broker [4:35]
The speaker details the development of an Open Service Broker (OSB), a web application with an API that facilitates the provisioning of resources for a platform. The OSB operates in a Kubernetes environment, binding resources to pods or cloud instances. It abstracts away the underlying infrastructure, allowing developers to provision resources like databases without needing to know the specifics. The speaker initially used the Connection library in Python, then migrated to Flask, and eventually to FastAPI.
Diagram of OSB architecture [7:43]
The architecture of the OSB includes a FastAPI app, a worker, and a DynamoDB database. Clients make provisioning requests to the FastAPI app, which then sends the task details to an SQS queue. The worker processes the tasks asynchronously, such as creating DNS records or CloudFront distributions, and updates the database upon completion. The client polls the web server for the status of the provisioning task.
Picking a proxy technology - Envoy [9:56]
The speaker discusses the decision to replace enterprise load balances with an open-source, cloud-native proxy. Envoy Proxy was chosen for its dynamic configuration capabilities, allowing for runtime reloading of configurations. This enabled developers to self-service their load balancing needs without direct intervention from the operations team.
Envoy XDS Control Plane [11:36]
The speaker describes building an Envoy management server, called Sovereign, to manage the Envoy proxies. Sovereign is a FastAPI app that uses templates and context to generate Envoy configurations. It pulls data from a database and other sources like S3 buckets, merges it with templates, and serves the resulting configuration to the proxies. This allows for dynamic updates to the proxy configurations based on provisioning requests.
AWS Infrastructure [14:33]
The speaker explains the AWS infrastructure used to provision the Envoy proxies. The proxies are provisioned using CloudFormation templates, which define resources such as VPCs, subnets, internet gateways, security groups, and autoscaling groups. The autoscaling groups create EC2 instances based on a specified AMI (Amazon Machine Image).
Creating the machine image (AMI) [17:45]
The process of creating the AMI for the Envoy proxies is detailed. Packer, along with SaltStack for configuration management, is used to create a standard image. An EC2 instance is created in a development account, SaltStack configuration is uploaded, and a provisioning step is performed. The instance is then turned into an AMI, which includes Envoy installation and configuration, logging agents, security hardening, network tuning, and observability agents.
24 month recap [20:22]
The speaker summarizes the first 24 months at Atlassian, highlighting the creation of a system where developers can easily provision load balancing with advanced features. The system includes a broker, management server, pre-provisioned infrastructure, and dynamic configuration updates.
What did I do after building [21:09]
After building the foundation, the focus shifted to migrating larger products to the platform and enforcing the use of the centralized load balancing infrastructure for all microservices. This ensured that all publicly accessible services were properly configured and protected.
Extending the load balancing platform [22:45]
The speaker discusses extending the Envoy-based product by leveraging the groundwork for turning basic inputs into templated configurations. Envoy's extensive configuration options, such as virtual hosts and routing rules, required careful validation and abstraction of the input parameters to ensure valid resource generation.
Envoy extensions [24:37]
The speaker mentions the various extensions that can be applied to Envoy listeners and clusters, such as network filters and the HTTP connection manager. These extensions allow for configuring routing, handling proxies and websockets, and implementing external processing and authorization.
Edge Compute and centralized logic [25:54]
The speaker emphasizes the opportunity to centralize logic and handle concerns early in the request chain. By addressing issues like authentication, authorization, DDoS protection, and rate limiting at the edge, before requests reach backend services, the platform can save time, money, and improve the customer experience.
Handling concerns for dev teams [27:12]
The speaker explains how various concerns are handled at the edge. DDoS protection is provided by CloudFront, while access logs are implemented using network filters within Envoy. More complex features like authentication and authorization are implemented using a sidecar model, with separate services running locally on the proxy.
Diplomacy and conflict resolution [31:35]
The speaker transitions to non-technical aspects of their experience, highlighting the growth in diplomacy skills, conflict avoidance, and conflict resolution. They emphasize the importance of persuasion, teaching, education, and mentoring.
Maintaining software over long-term [32:14]
The speaker discusses the challenges of maintaining software and systems over the long term. Onboarding new team members, writing documentation, and training people are crucial. The speaker notes that code churn is a smell that indicates a part of the service or project is going to keep increasing in size or complexity and something there needs to happen.
Personality Conflicts [35:42]
The speaker reflects on personality conflicts experienced with different managers and colleagues. They stress the importance of self-awareness, understanding of psychology, and taking responsibility for managing these differences to make relationships work.
Mentoring [37:11]
The speaker shares their challenges with mentoring, particularly striking a balance between providing guidance and allowing the mentee to solve problems independently. While their intern achieved excellent results, the speaker remains unsure if they effectively reached that balance. They distinguish mentoring from training and helping colleagues, which they found to be their strength.
