Interview with Zachary Ernst: the philosopher who became a data engineer

Zachary Ernst left a tenured position as a philosophy professor to work in the start up Narrative Science. Now, Ernst works for the company Civis Analytics as a data engineer. Given the increasing role of data science in lots of domains, I thought it would be interesting to interview him again (see here for an earlier interview for his position at Narrative Science, and here for an interview for his job at Trunk Club).

HDC: You’re a philosopher by training, and you work as a data engineer. Can you tell me something about the path that took you there?

ZE: I did my PhD work at the University of Wisconsin-Madison, where I specialized in philosophy of biology, with an emphasis in logic. I enjoyed graduate school a lot, and had a very positive experience throughout. My first job was a tenure-track position at Florida State University, where I stayed for three years. Then I moved to the University of Missouri-Columbia, where I received tenure.

Although I strongly believe in the mission of higher education, and I enjoyed both my teaching and my research, I was not happy as a professor. In short, I was very dissatisfied with my department and with the university as a whole. Realistically, it would have taken me years to get a comparable position somewhere else, assuming that were even possible. And so I started looking for options. Quitting my job was not a decision I made lightly. I took a leave of absence for a year to work at a small center funded by the Defense Intelligence Agency to see if there could be a career for me in government. Then I launched a small startup after returning to my job. I failed (spectacularly) at both, but I learned a lot and came away from the experience with a better sense of the opportunities that were available.

I knew I wanted to work in technology for a few different reasons. First, I had been an amateur programmer for many years, so I felt I could function in an entry-level position. Second, there is intellectually fascinating work happening in the tech sector. Third, technology is an excellent way to have a large impact, and this appealed to me after being isolated in the ivory tower for so long. And fourth, skilled workers in the tech sector are in very high demand. This meant that I had a chance of getting a job coming from a very non-traditional background, and there was a good chance the job would pay at least as well as my academic salary.

It was surprisingly easy to get a job. Within a few weeks of sending out some resumes, I had two very solid job offers as a software engineer. I took an offer with a startup called Narrative Science in Chicago, and it was a tremendous experience. I learned a lot of interesting stuff every single day, and I worked with excellent people who taught me the nuts and bolts of professional software engineering (which was much more challenging than I had anticipated).

After a couple of very good years, I moved from Narrative Science to Trunk Club in order to broaden my experience and get into data science, which is a very hot field. It turned out that data engineering was greatly needed at Trunk Club, so I became their first data engineer. After about a year, I was recruited by Civis Analytics, which is a data science company (also in Chicago) that does consulting and also sells a data science platform. When I learned about the excellent work they're doing and what their plans were for the future, I joined. My role is "Lead Data Engineer", which carries a combination of technical and managerial responsibilities. I manage their first data engineering team; my responsibility is to lead the effort to design and implement a new infrastructure for handling all client data and internal data assets. It is an absolute blast, and I work with some of the sharpest people I have ever met. I've found this work to be high-impact and intellectually fascinating. It's also easily the most challenging role I've taken on.

HDC: The work at Civis Analytics seems to me quite technical and different from any job in the higher education sector. Is there any sense in which you can still use the mindset or skills that you acquired as a philosopher for this job? 

ZE: That's true -- data engineering is highly technical; it requires knowledge of emerging technologies and engineering methodologies. There is virtually nothing from philosophy that directly relates to it. If you're looking for applications of philosophy, you can find some in data science, where Bayesian methods, for example, are very common. Occasionally, you can apply some formal logic as well. But those are the exceptions, not the rule.

General skills such as critical reasoning and the ability to communicate clearly are important, of course. As a technical lead, I'm expected to formulate strategies for how we invest resources and what we develop. Just as importantly, someone in my position is expected to present a clear and compelling argument and present it to executives and the C-suite. Millions of dollars are routinely spent on the basis of these arguments. The ability to concisely present an argument is often the difference between an effective and an ineffective technical lead. These are valuable skills in philosophy; but philosophy doesn't have a monopoly on those skills.

More interestingly, I've found that I needed to unlearn some of the habits that had served me well in philosophy. For example, I've had to ramp up how quickly I learn. As everyone knows, technology is evolving at a dizzying speed. But the same is also true of business, which both affects and is affected by technology. For example, cloud computing has radically altered the economics of startups; and that change ripples through the entire economy. The business landscape is far more competitive, in part because technology has transformed local competition into global competition. Increased competition creates high demand for new technologies that can make businesses more efficient. This feedback loop is accelerating. There is no such dynamic in philosophy, where it takes many years for a new idea to gain traction.

HDC: This sounds like it’s similar to academia in that excellent performance and high competition are the norm (e.g., in academia, for grants). Would you say your current job is more or less demanding than an academic job, and do you have any time for hobbies or socializing?

I'm glad you brought that up. It's a myth that jobs in the technology sector require an outrageous number of hours spent at work. It depends a great deal on the business. If you're at a scrappy startup (like Narrative Science was when I started there), then the work hours can be long and/or unpredictable. If you're in a more established business, then work hours are typical of any other business. There is also a lot of variation by sector; if you're in the finance industry, work hours can be notoriously long, for example (although that's not universally true, either).

There are costs and benefits for any tech sector job. Work hours are long in startups, but you've got an unusual opportunity make a big difference to the company; and the long work hours often provide the chance to learn a lot of different things very quickly. Personally, I like that trade-off. But it's not for everyone. Plenty of very good people prefer to work in a more predictable environment that leaves more time for other things. Like everything else, it's a just a matter of priorities.

The difference between academia and the private sector is that you've got a lot more flexibility as an academic (assuming you don't have a crushing teaching load, which is all-too common). If you're comparing the two on that basis, then academia definitely has the advantage. But that's mainly because academic work in the humanities tends be a solo endeavor. Work in tech is overwhelmingly a team sport, with a lot of cross-functional collaboration. So again, it's a trade-off. If you prefer to work by yourself, then you can control your time. If you prefer to work closely with others, then you've obviously got to coordinate your schedules. Personally, I always felt isolated in academia, and I learn a ton from my colleagues every single day. So I like the trade-off.

As far as competition is concerned, it's just different. The business environment is incredibly competitive and unpredictable. But you are not in competition with other individuals. You're just trying to move the business forward as quickly as possible. Practically all high-impact work is accomplished by teams, so the ability to collaborate and play well with others is crucial. 

ZE: About the job, with Civis Analytics. I was excited to hear that it was involved in consulting the team behind the Doug Jones campaign. Can you tell me a bit more about that, and about what the company does in general? 

Your question raises an important difference between my current work and the academic work I used to do, which is this: I can't answer questions about any specific client, with very rare exceptions. This is standard in tech, where intellectual property is strictly guarded. The rules about what you can and cannot discuss are very jarring to a former academic like me.

Of course, I can describe what Civis does in general. Civis grew out of the data science group in the Obama campaign. Originally, it was a consulting group specializing in data science for liberal organizations. The scope of the business has grown a lot in the past couple of years. Civis still does a lot of political consulting on the left side of the political spectrum (and no consulting for the right -- Civis is not a hired gun). We also have an increasing number of clients in a very wide range of industries for whom we do a lot of consulting. In addition, we also sell licenses to our proprietary data science software platform, which was originally developed to help make our consultants more efficient. Civis now has a large data science group, split roughly evenly between R&D and consulting. Additionally, we have a lot of top-notch expertise in survey methodology. There's a lot of innovation in those departments; and I guarantee that the work done by those people is far beyond the vast majority of current academic work that touches on data science (e.g. behavioral economics, political science, causal inference, etc.).

A typical consulting job may include running surveys, providing a statistical analysis of the results, using those results in a model developed by our R&D department, and applying all of that information to address client needs. Doing all of this requires a great deal of technical infrastructure, automation and custom tooling, which fall under the Civis technical department (which is my home). Our mandate is to continuously improve our infrastructure to make the platform and the various teams more efficient.

HDC: Your answer raises the following question: As philosophers (for instance, political philosophers) we try to think through challenges that face society today, such as the increasing influence of big data in elections, marketing and other domains. As your answer shows, a lot of the work on this is not easily transparent or visible to outsiders, and goes beyond the work we see in behavioral economics and other academic fields. This raises a skeptical worry/challenge: can philosophers still say anything meaningful about these developments, given their lack of expertise or information about it? I also have a broader worry that if academics cannot have a clear grasp on how this work, what broader ramifications this has for accountability and the way our society works.

ZE: The ethical implications of data science are enormous -- far larger, in my opinion, than is currently appreciated. We have the ability to do enormous good and improve the lives of huge numbers of people. I know of very good work done in this space to help cure diseases, improve infrastructure, respond to disasters, and so on. But this technology is very easy to abuse. Privacy implications are huge; and big data will provide ways to subtly influence behavior. If I were to go back to philosophy, I'd write on the ethical implications of data science.

In my opinion, there's not enough general appreciation of the enormous scale of data science, even though that information is out there for the public to see. Most people don't know, for example, that *all* credit card transactions are routinely sold to big hedge funds for enormous sums of money. I'm not talking about aggregations of transactions or statistical summaries -- I'm talking about every individual purchase. Google and Facebook track not only your web searches, but the sites you visit. Your phone is sending location information to your provider continuously. That information is being collected and sold, and this is no secret from the public. But what's under-appreciated is the the scale and the secondary information that you can infer about a person from that raw data. If you combine web searches, credit card transactions, and location data, for example, you could easily tell that a person was searching for medical symptoms of a disease, followed by a visit to a building with a medical specialist, followed by a transaction at a pharmacy or hospital. Big hedge funds do exactly this at scale in order to make bets on certain drugs, to take one example I happen to know for a fact.

With a little research, philosophers could (and in my opinion, should) write about these issues. It would be perfectly reasonable to educate oneself about this issue because it's not hard to learn about the technical capabilities, even though the technical implementations are very difficult.

HDC: As a tenured academic, you found your way outside of academia. However, there are untenured people who have been in academia for a while, often in a succession of poorly paid VAP positions or short-term postdocs, who are thinking of leaving academia but who don’t know where to begin. Do you have any advice for such people in particular? I ask because many of the Philosophers’ Cocoon readers belong to that demographic. 

ZE: Having tenure gave me a lot of flexibility and freedom to explore my options that other academics typically don't have. But regardless of your current circumstances, the goal should be to make a series of low-risk bets that will improve your options.

There are lots of ways to do this; and being embedded in a university (at any position from grad student to tenure) opens a lot of options that most people don't have. For example, I encouraged a couple of my graduate students who had an interest in tech to go talk to computer science professors who had their own grant-funded labs. Offer to do some work in those labs part-time for free so that you can learn. Be up-front and totally honest about your goals. Say, "you've got some really interesting work happening; I'd like to explore whether there's a place for me here. Hopefully, I could contribute something in exchange for the opportunity to learn." It would be highly unusual to be turned down.

There are several ways in which such a strategy is helpful. First, obviously, you can learn some programming or other nuts-and-bolts skill. It gives you something substantive to put on your resume. You'll meet people and naturally expand your network. And you'll be able to say in interviews, "I just wanted to learn; so I offered to work in this lab.". That's a really compelling story to be able to tell someone who's looking for an intellectually curious, self-starting person.

A lot of universities also have places where small startups are being incubated. The same strategy will work there, too. Startups are always starved for smart people to do work. You'd have an excellent shot at finding an interesting startup to contribute to.

This kind of strategy has a really great risk/reward ratio. There is virtually no risk. The rewards can be huge, though. No matter what happens to the lab or the startup, you will be guaranteed to learn, gain valuable experience, meet people, and get something compelling to put on your resume. That's what I mean when I say that you should look for low-risk bets.

Many universities have courses in their business school on entrepreneurship. Get in touch via email and then go to the instructor's office hours; explain your situation and ask for some advice. Take the course (or better yet, audit it if you can for even less risk). Ask the instructor if there's anyone you could talk to for some additional advice. If there is, get in touch and offer to buy that person a cup of coffee at their convenience and chat. People say "yes" to those requests. So long as you're totally honest and authentic, people will be more than happy to talk and share whatever information they can. Your goal is to find opportunities to learn. It's all about learning.

In my experience, many academics think of this sort of networking as manipulative or dishonest. That is absolutely not true. You should always be 100% honest and authentic in these discussions. You’re not trying to “get anything” from anyone except some advice, which people are usually happy to offer.

One last piece of advice -- the playbook for my transition to the private sector was a book by Reid Hoffman called "The Startup of You". If you don't know who Reid Hoffman is, check him out. He's a former philosophy grad student who quit, had some remarkable failures, and ultimately founded LinkedIn. Hoffman is now a highly influential billionaire investor in startups. You can detect the philosophy training in some of his writing, but his advice is very practical and realistic.

Comments

[Pendaran]

I read through this with interest. It seems like Zach had various non-academic skills, i.e. programming, that he could take to the private sector. What if you don't have these skills? Is there a way to learn programming skills without doing a BA that employers will respect? I could teach myself, but then it strikes me as if it's going to be hard to compete with BAs in computer stuff. I'd like to avoid going back to school for another degree if possible.

[Pendaran]

To be clear, I don't currently have a university position. So, I don't think I can just audit classes very easily.

[Derek]

Programming is very welcoming to people without cs degrees. As long as you can prove that you can code by making a few sample projects, people will take you seriously enough to consider you for an entry level position. It may take some time and a lot of job applications, but it is much easier than academia.

[Amanda]

I know in the US there are a number of "coding bootcamps". They teach you code in like 3 months for a few thousand dollars from what I have heard. Some places it is even free because it is with companies that plan to hire you. You have to spend some time researching them because the quality and value for the money varies.

Anyway not sure if Zach is reading this but if so I have a question. Given the timeline of his story, I would sort of assume he was not young when he took this job. I have a close family friend who spent 25 years working for a small business but recently found himself jobless because the business owner retired. He majored in computer science a very long time ago, and briefly worked in coding for the department of defense before quitting to work for the business. He wants to get back into coding but his skills are obviously dated and he is almost 50, so I'm wondering how possible that would be for him, or if there is age bias?