In April, I was given the opportunity to present An Executiveâs Cheat Sheet on Hadoop, the Enterprise Data Warehouse and the Data Lake at the SAS Global Forum Executive Conference in Dallas. During this standing-room only session, I addressed these five questions:
What can Hadoop do that my data warehouse canât?
Weâre not doing âbigâ data, so why do we need Hadoop?
Is Hadoop enterprise-ready?
Isnât a data lake just the data warehouse revisited?
What are some of the pros and cons of a data lake?
Following is a recap of my comments, along with a few screenshots. See what you think.
1. WHAT CAN HADOOP DO THAT MY DATA WAREHOUSE CANâT?
The short answer is: (1) Store any and all kinds of data more cheaply and (2) process all this data more quickly (and cheaply).
The longer answer is: I made reference to my opening âsoapboxâ statement – âBig data is not new.â They say that 20% of the data we deal with today is structured data (see examples in orange boxes below). I also call this traditional, relational data. The other 80% is semi-structured or unstructured data (examples in blue boxes), and this is what I call âbigâ data.
Are any of these example blue-box data types new? Of course not. Weâve been collecting, processing, storing, and analyzing all this data for decades. What we havenât been able to do very well, however, if at all, is mix the orange- and blue-box data together.
So hereâs whatâs new: We now have the technologies to collect, process, store, and analyze all this data together. In other words, we can now mix-&-match the orange- and blue-box data together â at a fraction of the cost and time of our traditional, relational systems.
2. WEâRE NOT DOING âBIGâ DATA, SO WHY DO WE NEED HADOOP?
I proposed six common Hadoop use casesâthree of which donât require âbigâ data at all to take full advantage of Hadoop. These use cases come from my white paper called The Non-Geekâs Big Data Playbook: Hadoop and the Enterprise Data Warehouse.
Hereâs a brief summary of each use case:
Stage structured data. Use Hadoop as a data staging platform for your data warehouse.
Process structured data. Use Hadoop to update data in your data warehouse and/or operational systems.
Archive all data. Use Hadoop to archive all your data on-premises or in the cloud.
Process any data. Use Hadoop to take advantage of non-integrated and unstructured data thatâs currently unavailable in your data warehouse.
Access any data (via data warehouse). Use Hadoop to extend your data warehouse and keep it at the center of your organizationâs data universe.
Access any data (via Hadoop). Use Hadoop as the landing platform for all data and exploit the strengths of both the data warehouse and Hadoop.
If youâd like to see these use cases further explained and demonstrated with some easy-to-understand visuals, I invite you to download the white paper.
3. IS HADOOP ENTERPRISE-READY?
I have two answers to this question:
For your organization: Maybe.
For all organizations: No.
It all depends on what and how you want to use Hadoop in your organization. If you simply want to use it as an additional (or alternative) storage repository and/or as a short-term data processor, then by all means, Apache Hadoop is ready for you.
However, if you want to go beyond data storage and processing and are looking for some of the same data management and analysis capabilities you currently have with your existing relational systems, you will first need to explore the vast ecosystem of Hadoop-related open source and proprietary projects and products. This will not be a small undertaking.
Because many of these newer Hadoop-related technologies are still maturingâquite rapidly, I might addâthatâs why I say Hadoopâas in the Hadoop ecosystemâisnât 100% ready for the enterprise.
4. ISNâT A DATA LAKE JUST THE DATA WAREHOUSE REVISITED?
Many of us have been learning more about the data lake, especially in the last 6 months. Some suggest that the data lake is just a reincarnation of the data warehouseâin the spirit of âbeen there, done that.â Others focus on how much better this âshiny, newâ data lake is, while others are standing on the shoreline screaming, âDonât go in! Itâs not a lakeâitâs a swamp!â
All kidding aside, the commonality I see is that they are both data storage repositories. Beyond that, the table below highlights some key differences. This is, by no means, an exhaustive list, but it does get us past this âbeen there, done thatâ mentality. A data lake is not a data warehouse.
5. WHAT ARE SOME OF THE PROS AND CONS OF A DATA LAKE?
Some of you may be aware of the Data Lake Debate blog series I recently participated in with my colleague, Anne Buff, on SmartData Collective. I took the Pro stance, Anne took the Con stance, and our boss, Jill DychÃ©, moderated.
It was an intense 8 weeks of discussionâloosely structured like a Lincoln-Douglas debateâand many key points about the data lake were addressed. During my presentation, I summed up these key points using a SWOT diagram:
Note: Read original post here.