My Time on the Job Market as a Data Engineer

In December 2018, I had an amazing job at a company I loved, doing work I was immensely proud of. This made my decision to quit extraordinarily difficult. This decision could itself be the topic of a long and rambling blog post, but that’s a story for another day. In this post, I’ll talk through my experience being unemployed, my approach to the job search, and how I ultimately made the decision for my next career move.

Unemployment

I left my prior job without a new one lined up. When my friends heard I was entering a phase of funemployment, they all assumed I would be taking off to backpack around the world for 6 months. I take that to mean that I do a pretty good job of hiding my true workaholic anxious nature. By the time the New Year rolled around, I was already irrationally worrying about my employability and felt like I was behind schedule with everything — interview prep, networking, and everything in-between. Soliciting advice from a few friends who had taken extended time off, I was able to assuage the rising feeling of panic and settle into a loose routine for my time off. I would get a full night’s sleep every day, spend mornings catching up on the latest tech news and analysis (I read a LOT of Stratechery), and take at least one weekday each week to do absolutely nothing job related (this usually meant riding my bike all day). The rest of the time would be split between catching up with old friends and former colleagues, gathering information about companies I was interested in, and practicing my programming and technical whiteboarding skills.

The Job Search

From both my experience as a hiring manager as well as preliminary conversations with a few peers, I understood very quickly that it was a buyer’s market, and I could easily make job hunting a full time effort. I also wanted to give myself the opportunity to truly evaluate a broad spectrum of companies of all shapes and sizes, across many different industries. In total, I spoke to 26 different companies, had tech screens with 11 (withdrew my application from the rest), did 9 onsite interviews, and ended up with 7 offers, the majority of which were for individual contributor roles as a data engineer. I thought this process would be exhausting, but it turned out to be far more fun and exciting than I anticipated. It was fascinating to learn about all the different organizations and businesses, and the unique challenges faced by each of them.

As I talked to more and more people, I started to develop a better sense of how to extract real signal from my conversations. During interviews, most folks are either in evaluation mode or sales mode. In both cases, they’re likely to stick to an HR-approved script. My goal in every interview was to build enough rapport with the interviewer that I could successfully navigate the conversation away from the typical clichés. I also did my best to ask very similar questions to each interviewer. By listening closely to their individual answers, I could then evaluate them each in the context of the whole, which would often paint a much more telling picture of the organization than any individual answer. Do individual contributors understand the vision set forth by their managers? Are the pains of the ICs being heard by management? Are different business units aligned on the company mission?

Every company I talked to had extremely aggressive hiring goals. Most were looking to double their engineering headcount by the end of the year, and more than double the size of their data engineering teams. More often than not, when I asked engineering leaders about their biggest challenges, hiring was #1 on the list. I began to evaluate prospective companies through this lens, asking “how will this company differentiate from all the others when competing for talent?” Every company had a different angle for this, some leveraging recent fundraising events or a high profile consumer brand, others leaning heavily on their social-impact oriented mission. I tried to understand not only how their answers appealed to me, but how they might appeal to the broader segment of job-seekers.

Key Takeaways

I learned a lot during my interviews. Rather than try and tie them all together into a neat narrative, I’ll just list a few things that stood out to me as noteworthy:

• The technical bar for data engineering is reasonable. I did quite a bit of prep using the standard books and websites like Cracking the Coding Interview and leetcode. Never was I asked anything I felt was overly difficult or unfair.
• I don’t consider myself a great programmer, but do think I have better than average soft skills for an engineer. Based on my success during the interview process, I suspect this combination is more valuable than the inverse.
• A 5 hour onsite interview is simply not enough time to effectively evaluate a workplace. During 5 hours of interviewing, a candidate has at most 1 hour available for asking questions about the company. How can someone possibly learn enough about a company during that time to make an informed decision? Doing pre-interview prep and intelligence gathering is absolutely critical, as is being efficient with your time.
• I really enjoyed all of my conversations with companies, except declining offers. It’s emotionally draining to let someone down immediately after they’ve congratulated you and told you how excited everyone is about the possibility of you joining. It also forced me to confront the reality that the choice to go through one door meant closing many others.

The Decision

I count myself as extraordinarily fortunate to have had my pick of some of the best technology companies in San Francisco. I was looking for a company with aggressive growth, a great product, and awesome leadership, and while many of the companies I talked to met these criteria, Snowflake was a clear cut above. When I first started using Snowflake as a customer at my previous job, I was totally blown away by their product. The Snowflake data warehouse was critical to my job as a data engineer, and it was obvious to me how revolutionary a technology they had developed.

The job at Snowflake was in sales engineering, a big change from my prior role as an in-house data engineer. As a sales engineer, the responsibilities are primarily around evaluating the data architecture of potential customers, helping prove out the value of Snowflake within that architecture, and scoping and executing on a proof-of-concept. The chance to get a glimpse of data teams of all shapes and sizes across the San Francisco tech scene and beyond seemed like a unique opportunity.

From a team perspective, I knew Snowflake’s sales and sales engineering org fairly well from my time as a customer. Both groups were great to work with — their sales engineering lead was enormously valuable in helping us with our initial implementation, and the regional sales director struck me as an ambitious, driven individual who would likely push me to realize more of my potential. This gave me a high degree of confidence in the general quality of the team over at Snowflake, which was confirmed yet again during my interview process.

As I alluded to earlier, I spent a fair amount of my funemployment reading through the back catalogue of Ben Thompson’s Stratechery blog. Stratechery focuses primarily on consumer technology, with decidedly fewer articles on enterprise software, especially a product as technical as Snowflake. Even so, many of the themes he emphasizes over and over when discussing consumer tech apply just as well to enterprise. In this light, many of Snowflake’s initiatives made sense as part of a broader strategy. I didn’t see any other players in the space operating at the same level, and this combination of superior product and thought leadership made it an extremely compelling opportunity.

I’m only a few weeks into my new role as a sales engineer at Snowflake, and so far it has not disappointed. The energy around what we’re building, both in terms of the product and the business is absolutely incredible. Funemployment is finally over, but now the real fun begins!