Why data engineering is better (for me) than data science

I thought that a data scientist position would be a dream job. I believed in “the sexiest job of the XXI century” hype. Silly me. Because of the hype, I spent a long time learning data visualization and machine learning. I started blogging about data science in June 2018. I got the data scientist job one year later. And… I quit it two months later. Between June 2018 and the day I left, I wrote over 100 articles about machine learning. I had learned a lot, but data science wasn’t for me. There were many reasons: the job itself, the office where we worked, the location of the office.

The neighborhood

Imagine an old artisan district in a large city in Poland. It is full of old tenement buildings with shops on the ground level and apartments above them. In the place of buildings ruined during World War II, the rulers of the Polish People’s Republic built gray Soviet-style housing blocks. Nowadays, we paint them in bright colors in a futile attempt to make them look less like the human equivalent of battery cage poultry farms. To spice up the mix, we fill the unused space with ridiculously overpriced, modern apartment buildings with walls so thin you can hear the neighbor flushing the toilet.

In the 90s, this place was one of the worst districts in the city. It was infested with drug dealers, aspiring rap musicians, and juvenile criminals. Today, it became the hipster area. You can easily find vegan restaurants, craft beer pubs, and dozens of places selling kombucha. If you are not looking for trouble, you should stay near those hipster venues because when you turn onto the wrong street, you will find a different place - the remains of the 90s.

A few meters away from the hipster spots, you can see children playing at a urine-reeking playground and shattered vodka bottles on the sidewalks. If you are particularly (un)lucky, you may see a police car stopping abruptly in the middle of the street and witness officers running towards one of the buildings. At the end of one such street stands a school. There is nothing special about the school besides one online review. The one-star review features a picture of a teenager having bruised face and a black eye. In the comment, he wrote: “I was beaten here.” (The photo was already removed, only the text remains.)

In front of the school, on the other side of the street, stands a vine-covered building partially repurposed as a coworking space. This is where I worked.

The workplace

The building was “partially” repurposed because the office space was still an apartment with furniture removed and replaced with desks, office chairs, and cool-looking but quite dim lamps. There are old buildings with large windows and bright rooms. Our coworking space wasn’t one of them. We were renting a tiny room with only one window. The room was quite tall, like many rooms in XIX century/early XX century buildings. As you may expect, it was freezing in the winter and hot in the summer. Fortunately, I didn’t stay there long enough to experience both.

In our room, we had five small desks with no space between them. We all had B2B contracts, so Occupational Health And Safety Regulations did not apply. We weren’t employees, so we “enjoyed” being very close to our colleagues. Fortunately, we could work from home, and I was doing it quite frequently.

Naturally, our office could be worse. One day, I saw a Slack message sent by one of the founders who tried to disprove rumors about rats in the NYC office. Our office was way better! As far as I know, we didn’t have rats roaming around.


Subscribe to the newsletter and join the free email course.

The task

I worked at a startup in which founders couldn’t claim they were “making the world a better place,” “disrupting,” or “revolutionizing” anything. We were running websites with celebrity gossip, trivia, crime stories, etc., and we were selling advertisement space. Was this a viable business? Hell no. Almost nobody looks for such websites, so we had to buy tons of ads elsewhere to get any traffic. The whole idea was based on the old-school concept of buying low and selling high. They needed data scientists for the “buying low” part. We were trying to automate bidding and get as cheap traffic as possible.

Why was I working there? It was the only place where an inexperienced data scientist could get a job. We all already know why the experienced people didn’t want it.

Overall, was it a good business idea? Of course not. We were burning money fast. It was so bad we were celebrating every day when we managed to break even. It happened once during my two months at this startup.

I know, I know. This is how startups work. It is true, but usually, startups pivot when their idea doesn’t work. At this place, we were doubling down on things that didn’t work in the past and expected to get different results this time.

Quickly, I realized that my actions made no difference. Also, I don’t enjoy training machine learning models nearly as much as I thought I would. It was fun when I was learning it. Mostly because I was switching to a different problem every time I got a working solution. Not here. This was the real world, not a Kaggle assignment. I had one problem and one problem only: figure out the maximal amount we could spend on buying ads and still make a profit. Worse still, I wasn’t as good data scientist as I wish I had been.

I realized I enjoy deploying the model and building data pipelines way more than training the models. After a few weeks, I focused on doing the data engineering part of the job. Nobody complained because the other data scientist didn’t know how to do it. It was perceived as stepping up to a challenge, not hiding in the comfort zone.

Quitting

At the same time, I was reading the “Career Superpowers” book by James A. Whittaker. In the book, he wrote the “Underachievement Manifesto,” saying:

This may seem harsh, but there is opportunity written all over it for those who aren’t too blind with ambition to see it. The ranks of any fields, no matter how mundane or exciting, are full of people who have stretched to get there. People not quite smart enough for medicine still practice medicine. People not quite dedicated enough to law are still lawyers. People who aren’t particularly mechanically minded still try to fix cars. This is why a good mechanic stands out, he or she is competing against people who aren’t really good enough to be there.

That was me. I stretched to become a data scientist, and I didn’t even enjoy the job. Soon, I switched to data engineering/MLOps and never looked back.

When I handed in my resignation, the team couldn’t understand why I quit a company where I could influence the product and do (almost) whatever I wanted (as long as it is related to buying cheaper ads). I told them I didn’t believe they could survive the next six months. I was wrong. They survived eight months.

I didn’t include the data scientist job in my CV or LinkedIn profile. I am pretty ashamed of it. Most people don’t notice the gap or assume I was taking a long vacation between the jobs. If anybody asks, I tell them this story. Nobody questions my decision.


Remember to share on social media!
If you like this text, please share it on Facebook/Twitter/LinkedIn/Reddit or other social media.

If you want to contact me, send me a message on LinkedIn or Twitter.

Would you like to have a call and talk? Please schedule a meeting using this link.


Bartosz Mikulski
Bartosz Mikulski * MLOps Engineer / data engineer * conference speaker * co-founder of Software Craft Poznan & Poznan Scala User Group

Subscribe to the newsletter and get access to my free email course on building trustworthy data pipelines.