Breaking into Data Engineering
Data Engineering has become very popular over the past 4–6 years and many folks now want to switch to this data field. I am writing small know-how for this. A few things I will cover in this write-up are:
-> the reason for DE's popularity.
-> What to study to understand the crux of DE.
-> How can one move laterally or begin a career in this field?
Let’s start.
First things first. Why the sudden rise in DE popularity? Up until 5 years back, everyone wanted to break into Data Science. Why? Fake answer: it’s interesting, Machine Learning is challenging, tough problems to solve.
Honest Answer: Money. Companies began a hiring spree, but nobody (for most who follow the tidal wave) knew why they needed Data Scientists as it is very tough to calculate the ROI on it (be it the compute, time, engineering bandwidth, or the outcome). But everyone wanted to do something with Data Science and hence the demand but there needed to be more supply. So if you were a Data Scientist with actual experience you could have any amount in the market and easily get that. What happened?
Well Time passed, those who were in the field stayed, and those who tried it did not like - too much mathematics, statistics, or the fact that it wasn’t closely related to what…
