Remember that scene in The Matrix when Neo finally “sees” the matrix? It’s a great analogy for data flow thinking.
Once you understand how data flows and how to think in those terms, you’re kind of like Neo seeing the matrix. When you can not only see, but understand and control data flows, data flow thinking becomes an IT superpower.
Data flow thinking is a key skill to develop if you want to thrive in the cloud. Sure, you can probably pass some certification exams without it, but if you want to truly understand what you’re learning, retain the concepts, and use them effectively in the real world, data flow thinking is your ticket.
Get more than certified
Join A Cloud Guru and get access to all of our courses, labs, quizzes, and our new learning paths, which take you step-by-step from novice to guru in your chosen area of the cloud.
What is data flow thinking?
Data flow thinking is the process of breaking down what happens to data and thinking about how it flows — so that it all moves in the most efficient way possible.
The exact term might be new to you, but you’re probably already using data flow thinking every day. It probably so natural that you don’t even think about it.
For example, if you’ve ever hooked up a device to your TV, you’ve likely done some basic data flow thinking. It begins by mentally picturing how the video information has to move from one source to another.
First, video data has to reach the device. This used to happen via a video cassette, game cartridge, or some kind of disc. Today, it’s just as likely data streaming over a wifi network (1).
Once the device acquires its video information and processes it (2), it outputs the video feed. These days, that means out an HDMI port and onto an HDMI cable.
The video data flows along the HDMI cable and into your TV through one of several HDMI input ports (3). To make it show up, though, you have to set the TV to the proper input.
If you add more devices, such as a video game console, A/V receiver, or surround sound speakers, the data flow grows more complicated, as you have additional pathways to consider.
Understanding how the data flows between devices can help you set up even the most complicated home theater in the most efficient way possible, and quickly diagnose, isolate, and fix any interruptions in those flows.
Harnessing your data flow powers
Plenty of us stumble into data flow thinking on a case-by-case basis. Hooking up a TV. Figuring out why a string of holiday lights won’t turn on. Setting up network attached storage to back up your photos.
But stumbling isn’t enough. It’s critical to bring data flow thinking to bear in a conscious way. Once you do, you can tap into this superpower on demand in your studies and your career.
Think back to Neo. In a moment of distress, he reflexively moves like the agents pursuing him. But it’s only when he “sees” the matrix that he’s able to understand and control everything. When he learns to consciously tap into that ability, he can put it to amazing practical applications like flying and stopping bullets.
Learning to apply data flow thinking in a conscious way allows you to harness its power and take your clouding to a whole new level of proficiency. You may not be able to fly or look cool in a trenchcoat, but you will be better able to ace your cloud certification exams and thrive in your career.
Unpacking data flow thinking
So data flow thinking is all about learning to identify and control data flows. But what does that mean? Well, if you think about data, you can generally do three types of things with it: 1) move it around, 2) process it in interesting ways, and 3) remember it.
- Moving data. The most obvious example of moving data is sending it from one computer to another, across a network. But this could also include loading stuff from a disc into memory or linking up services through some integrations that aren’t specifically networking.
- Processing data. This could involve how a database system responds to a query or the setting up of a messaging service to filter what passes through it. Of course, a really common way to process data is to run it through a program on the CPU--the Central Processing Unit.
- Remembering data. The obvious example here is storing data on a hard drive, but other examples could be caching data in RAM or at an edge location in a content distribution network.
Guru Tip: We generally associate moving, processing, and remembering with networking, compute, and storage, but they don’t always align themselves that way. Think of it as a good rule of thumb, but don’t let yourself get too boxed in with the idea.
Hustle and (data) flow
How can you develop your data flow thinking skills? Just being aware of the concept is a great start, but it’s only the start. Here are some ways that you can build this skill while you’re learning to cloud:
Take the time to understand each service. Don’t just fly through lessons. Stop and make sure you really understand each service. What does it do, and why? Where does it get data from? What does it do with it? And where does it send it when it’s done? How is it different from other services that do similar things?
Break things. Use labs to get hands-on experience navigating various services. Then take it further. Break them, then understand how they broke, and more importantly, why. Once you’ve broken a service, and know why, put it back together again.
Start with the leaf and zoom out. Start with truly understanding a single service, and then another, and then another. Then step back and work on understanding how data flows through different services when they’re linked into a small system. Then practice bringing those systems together to create a larger system, breaking and fixing and learning all the while.
Sketch out data flows. Grab some paper and sketch out a data flow between services or systems with the variables you have. Try to work out the most efficient flow. Then when you build it, you can see how well the mental model you sketched out lines up with the true behavior of the services you’re using.
If this sounds like a ton of work, it is! But it’s work that will pay itself off in spades as you continue to learn cloud (or pretty much any other subject — especially technical ones). By putting in the work up front, you are essentially teaching yourself how to learn and understand complex systems, rather than relying on the unreliable crutch of memorization.
Mastering multi-cloud with data flow thinking
One area where data flow thinking can really benefit you is moving from one cloud provider to another. In today’s market, multi-cloud is becoming more and more of a reality, and the more platforms you can navigate, the more valuable and sought after you’ll be. In fact, the currently-top-paying IT certification calls out multi-cloud abilities in a number of its requirements. (Incidentally, I’ve made this course to help people achieve this Professional Cloud Architect certification.)
If you managed to slip through your AWS Certified Solutions Architect - Associate exam by cramming and memorizing, you would have to try to do that all over again for your Google Certified Associate Cloud Engineer or Microsoft Azure AZ-103 exam.
But if you’ve got a handle on data flow thinking, you already have a massive leg up. Sure, AWS and GCP, for example, are two different animals. They use different names and their services work in slightly different ways. But the underlying concepts — the data flows — are always applicable.
Like driving in a new city or a new country, some of the details and rules might change, but the overall idea is the same.
The big takeaways of data flow thinking
Looking for a TL;DR? Here’s what we see as the most important aspects of data flow thinking:
- Data flows are the foundations of every system; developing the skill to understand and control these flows is critical for success in both exam and real world situations.
- Data flows involve moving, processing, and remembering data. This generally (but not always) matches up with networking, compute, and storage components.
- We may unconsciously engage in this, but the key is being able to summon and control it.
- You can hone your data flow thinking abilities by learning, breaking, and fixing services in various ways until you truly understand what they’re doing, how, and why.
→ Resource Note: Still feeling a bit fuzzy on data flow thinking? You may want to revisit my lecture on data flows in my Google Certified Associate Cloud Engineer course.
Learning to identify and control data flows is key to achieving success in the cloud. It ultimately makes learning, using, and mastering cloud services much easier. Actually, I would go as far as to say that data flow thinking is what makes this possible. Once you understand the flows, and what levers do what, you’ll be able to take on all kinds of tasks, from building well-architected frameworks to troubleshooting systems.
Developing your data flow thinking chops may seem like a lot of work, but it’s a worthy investment. Remember, Neo had to first (often painfully) unlearn limitations and reconstruct a new reality so that he could realize his new powers. So if you want to be successful in the cloud, you, too, will have to approach clouding in a new way.
Free your mind, Guru!