Data Engineer vs Data Scientists

 

Basic Data Engineering Skills

3 Learn To Code

Why this is important: Without coding you cannot do much in data engineering. I cannot count the number of times I needed a quick Java hack.

The possibilities are endless:

• Writing or quickly getting some data out of a SQL DB

• Testing to produce messages to a Kafka topic

• Understanding Source code of a Java Webservice

• Reading counter statistics out of a HBase key value store

So, which language do I recommend then?

I highly recommend Java. It’s everywhere!

When you are getting into data processing with Spark you should use Scala. But, after learning Java this is easy to do.

Also Python is a great choice. It is super versatile.

Personally however, I am not that big into Python. But I am going to look into it.

Where to Learn? There’s a Java Course on Udemy you could look at: https://www.udemy.com/java-programming-tutorial-for-beginners

• OOP Object oriented programming

• What are Unit tests to make sure what you code is working

• Functional Programming

• How to use build management tools like Maven

• Resilient testing (?)

 

4 Get Familiar With Git

Why this is important: One of the major problems with coding is to keep track of changes. It is also almost impossible to maintain a program you have multiple versions of.

Another is the topic of collaboration and documentation. Which is super Important.

Let’s say you work on a Spark application and your colleagues need to make changes while you are on holiday. Without some code management they are in huge trouble:

Where is the code? What have you changed last? Where is the documentation? How do we mark what we have changed?

But if you put your code on GitHub your colleagues can find your code. They can understand it through your documentation (please also have in-line comments)

But if you put your code on GitHub your colleagues can find your code. They can understand it through your documentation (please also have in-line comments)

Where to learn: Check out the GitHub Guides page where you can learn all the basics: https://guides.github.com/introduction/flow/

This great GitHub commands cheat sheet saved my butt multiple times: https://www.atlassian.com/git/tutorials/atlassian[1]git-cheatsheet

Also look into:

• Pull

• Push

• Branching

• Forking

 

5 Agile Development – available

Agility, the ability to adapt quickly to changing circumstances.

These days everyone wants to be agile. Big or small company people are looking for the “startup mentality”.

Many think it’s the corporate culture. Others think it’s the process how we create things that matters.

In this article I am going to talk about agility and self-reliance. About how you can incorporate agility in your professional career.

 

5.1 Why is agile so important?

Historically development is practiced as a hard defined process. You think of something, specify it, have it developed and then built in mass production.

It’s a bit of an arrogant process. You assume that you already know exactly what a customer wants. Or how a product has to look and how everything works out.

The problem is that the world does not work this way!

Often times the circumstances change because of internal factors.

Sometimes things just do not work out as planned or stuff is harder than you think.

You need to adapt.

Other times you find out that you build something customers do not like and need to be changed.

You need to adapt.

That’s why people jump on the Scrum train. Because Scrum is the definition of agile development, right?

 

5.2 Agile rules I learned over the years – available

5.2.1 Is the method making a difference?

Yes, Scrum or Google’s OKR can help to be more agile. The secret to being agile however, is not only how you create.

What makes me cringe is people try to tell you that being agile starts in your head. So, the problem is you?

No!

The biggest lesson I have learned over the past years is this: Agility goes down the drain when you outsource work.

 

5.2.2 The problem with outsourcing

I know on paper outsourcing seems like a no brainer: Development costs against the fixed costs.

It is expensive to bind existing resources on a task. It is even more expensive if you need to hire new employees.

The problem with outsourcing is that you pay someone to build stuff for you.

It does not matter who you pay to do something for you. He needs to make money.

His agenda will be to spend as less time as possible on your work. That is why outsourcing requires contracts, detailed specifications, timetables and delivery dates.

He doesn’t want to spend additional time on a project, only because you want changes in the middle. Every unplanned change costs him time and therefore money.

If so, you need to make another detailed specification and a contract change.

He is not going to put his mind into improving the product while developing. Firstly because he does not have the big picture. Secondly because he does not want to.

He is doing as he is told.

Who can blame him? If I was the subcontractor I would do exactly the same!

Does this sound agile to you?

 

5.2.3 Knowledge is king: A lesson from Elon Musk

Doing everything in house, that’s why startups are so productive. No time is wasted on waiting for someone else.

If something does not work, or needs to be changed, there is someone in the team who can do it right away.

One very prominent example who follows this strategy is Elon Musk.

Tesla’s Gigafactories are designed to get raw materials in on one side and spit out cars on the other. Why do you think Tesla is building Gigafactories who cost a lot of money?

Why is SpaceX building its one space engines? Clearly there are other, older, companies who could do that for them.

Why is Elon building tunnel boring machines at his new boring company?

At first glance this makes no sense!

 

5.2.4 How you really can be agile

If you look closer it all comes down to control and knowledge. You, your team, your company, needs to do as much as possible on your own. Self-reliance is king.

Build up your knowledge and therefore the teams knowledge. When you have the ability to do everything yourself, you are in full control.

You can build electric cars, rocket engines or bore tunnels.

Don’t largely rely on others and be confident to just do stuff on your own.

Dream big and JUST DO IT!

 

5.3 Agile Frameworks

5.3.1 Scrum

There’s a interesting Scrum Medium publication with a lot of details about Scrum: https: //medium.com/serious-scrum

Also this scrum guide webpage has good infos about Scrum: https://www.scrumguides. org/scrum-guide.html

 

5.3.2 OKR

I personally love OKR, been doing it for years. Especially for smaller teams OKR is great. You don’t have a lot of overhead and get work done. It helps you stay focused and look at the bigger picture.

I recommend to do a sync meeting every Monday. There you talk about what happened last week and what you are going to work on this week.

I talked about this in this Podcast: https://anchor.fm/andreaskayy/embed/episodes/ Agile-Development-Is-Important-But-Please-Dont-Do-Scrum--PoDS-041-e2e2j4

This is also this awesome 1,5 hours Startup guide from Google: https://youtu.be/mJB83EZtAjc I really love this video, I rewatched it multiple times.