What a data engineer can learn from The Unicorn Project?

Have you ever seen a novel about developers? Reading such a book seems to be a massive waste of time, doesn’t it? After all, the internet is full of stories written by real developers. What would happen if someone put all of those horror stories about terrible projects and incompetent managers into one book?

That is precisely the content of The Unicorn Project by Gene Kim. In that book, we follow Maxine, an engineer who experiences the biggest software engineering problems, all at once.

The Unicorn Project is like “The Martian” by Andy Weir, we know that everything is going to end up well, but the author keeps us asking the question: “What else?”

What else can go wrong? It seems that everything that could happen already happened. We know that there is something else coming, something that will crash the protagonists to the ground. It will look like an unbeatable challenge, but somehow they will overcome it.

It is an enjoyable story to read, but it is also something more substantial. This story is an excellent description of an engineer who makes other engineers more productive. We may call such a person a distinguished engineer. I think that we may also call them data engineers. After all, the sole purpose of a data engineering team is making data usage easier for other people.

In this book, the idea of increasing other engineer’s productivity is defined as The Five Ideals. Those ideals describe the working environment in which programmers are the most productive. Maxine learns about those ideals in a somewhat unusual conversation at a bar.

I will not spoil all of them to you, but one of them is “Focus, Flow, and Joy.” The character portrayed by Gene Kim uses those words to describe it:

“It’s all about how our daily work feels. Is our work marked by boredom and waiting for other people to get things done on our behalf? Do we blindly work on small pieces of the whole, only seeing the outcomes of our work during a deployment when everything blows up? (…) Or do we work in small batches, ideally single-piece flow, getting fast and continual feedback on our work?”

Is cooperation with data teams blissful? Does it bring joy? Are we helpful? How often does a data engineering team become the bottleneck?

Did you enjoy reading this article?
Would you like to learn more about software craft in data engineering and MLOps?

Subscribe to the newsletter or add this blog to your RSS reader (does anyone still use them?) to get a notification when I publish a new essay!

Newsletter

Do you enjoy reading my articles?
Subscribe to the newsletter if you don't want to miss the new content, business offers, and free training materials.

Bartosz Mikulski

Bartosz Mikulski

  • Data/MLOps engineer by day
  • DevRel/copywriter by night
  • Python and data engineering trainer
  • Conference speaker
  • Contributed a chapter to the book "97 Things Every Data Engineer Should Know"
  • Twitter: @mikulskibartosz
Newsletter

Do you enjoy reading my articles?
Subscribe to the newsletter if you don't want to miss the new content, business offers, and free training materials.