Working

It has been a career bucket list item to work at a startup that is just getting off the ground. I joined as a staff engineer on a central platform team with one other engineer. Eventually the team grew to 4 engineers. As startups tend to do the financials struggled and the team was let go. However with everything I learned I could confidently speak about almost all parts of platform engineering with hands-on confidence I did 1 interview which resulted in 1 offer. [Note: It was a company I had previously worked for and knew lots of people. This isn't a brag about getting a job in a struggling market so this context is important.] Here is some of my experiences. Working at large companies things tend to be somewhat figured out. A typical question I would get when interviewing new engineers for roles on the team was about what the most difficult part of the job is. My honest answer was always there was me and another engineer. If we didn't know the answer that means you have to figure it out because no one else at the company knows. These are some of the parts of the job I always knew about but lacked working hands-on: DNS - At every company I worked for DNS was managed by another team, typically SREs. I knew about various record types and all that and I knew how to dig a DNS record but here we had to setup the R53 ourselves, purchase the domains ourselves, setup the subdomains ourselves and setup all the record ourselves. We had to figure out how to manage this in a sustainable way so DNS isn't spread across a bunch of AWS accounts and do that in a secure fashion. CICD - Of course every engineer oftentimes interfaces with CICD. My experience was typically extending some scripts someone else wrote to do most of the hard stuff. The cluster was owned and operated by someone else. But we had to figure out what CICD to use, how to integrate it into our pipelines, how to build our services, how to house containers, how to deploy the services. There was no service builder helper to extend it had to start from scratch. App Startup - The companies I worked for already had app startup figured out. There was some framework you'd implement which would give you logging, metrics and some form of observability for free out of the box. But we had to figure out how to get these apps running and build out this framework ourselves. This even included selecting which language to use for our platform, how to structure the project and coding standards. Infrastructure - Every place loves to interview you on scalability but I've personally always found it somewhat abstracted away from me as the developer. My job was to crunch the code to make my container and it was someone else's job to scale it up or at least provide a convenient way to do so - usually some core tech team. At the very least usually stuff before it hits your code is owned and operated by someone else. Now again we leveraged AWS so lots of it is batteries included. The API Gateways scale, if we're using serverless it scales, if we're using a managed database it scales. But not everything was managed. So we typically had to write out the entire infrastructure diagrams ourselves and find scale points. This allowed me to walk into system design interviews not just spewing out memorization about various scaling techniques I actually had to live and breathe it for a long time. Questions about trade offs and decisions were now extremely easy to talk about because these were things I actually had to solve personally. IAC - I lacked a lot of Terraform know-how going into the job. So even simple questions which I now know as Terraform 101 such as "How do I save this state file from my ephemeral CICD node?" were complete unknowns. We had to figure out basically everything Terraform from scratch. Companies I've worked for in the past usually didn't expose Terraform to me directly. They were hybrid bare metal and public cloud options. So we had some form of IAC but it was always proprietary to the company, usually owned and managed by someone else and typically pared down. So this gave me direct exposure to how to describe my infrastructure as code including things like I said about how to save state files, how to tag my resources, how to manage int/dev/stage/prod environments and all of that. My previous experience was just writing some YAML and it was someone else's job to interpret all of that. Observability - Like I said previously usually this is something included in the app framework of the company. But this gave me experience setting up Grafana (our tool of choice), pumping or replicating logs to it, setting up collectors for metrics, setting up alerting and all the dashboarding. The alerting and dashboarding I'm sure most people have had experience with but usually for logging and metrics I'd just have to inject something into my application and it "just worked" without having to think about it too terribly much. This is a long winded post and not a question at all but more a bit of advice. If you are given the opportunity to work at a very early stage start up I highly recommend it. My end result was the same as most early start ups - I got let go. However I feel it made me a much more employable engineer that now can talk about times I actually built a lot of components that go into a backend platform. I don't need to talk about these things in the abstract, which of course is still important to understand, but I can speak to them in the concrete now. It was a fantastic growing and learning experience for me, personally.