I wanted to write this post about whether Data Engineers need to know about Object Orientated Programming for two reasons: Object Orientated Programming (OOP) is a topic of interest to me I like to think about and learn about how OOP can or has benefitted data engineers. Functional Programming as it says on the tin, … Read more
If you’re into data engineering but you have had your head in the sand, been on a vacation for months without internet or have simply been too busy to notice then chances are you haven’t yet heard of ChatGPT. But all that is unlikely and I’m sure you have heard about ChatGPT so in … Read more
Photo by NASA on Unsplash This is a nice fun little data engineering project idea that you can do in Python – tracking the International Space Station and visualising its path across the globe using Looker Studio (formerly Google Data Studio). We’ll be using Python 3.10 and we will be hosting the project in Google … Read more
Data Engineers typically build pipelines but they often do a lot more than this and so if you’re thinking of becoming a Data Engineer or are already one and just looking to broaden your knowledge then I prepared some high level topics here to help with that. In summary these are: Data Lake Data Warehouse … Read more
In this post, I talk about Pandas. Not the furry bears from China that munch on bamboo that we all know and love (sorry, couldn’t resist adding a crappy joke) but the super fantastic library available to install in Python that a Data Engineer, Data Analyst or Data Scientist can do SO MUCH with. What … Read more
If you’re building data pipelines then you’re going to need to be familiar with the different types of Python data structures. It would not be befitting to not know how to use these to your advantage. You will often find yourself either creating them or working with them when returned from library methods. In this … Read more
In today’s world, a Data Engineer will typically build “data pipelines“. I say “typically” as this would vary from organisation to organisation and may include additional responsibilities such as designing data warehouse schema’s and building reports. At a minimum, a data pipeline will move data from source (Extract) to destination (Load). There may also be … Read more