It has become almost a cliché to start articles about data by acknowledging the enormous amount of data that modern economies create, and with the increasing prominence of AI, the demand for all this data will simply continue to increase.
Thanks to the rise of enterprise cloud storage platforms, organizations now also have a much easier way to store these massive amounts of data, but that comes at a price: this data is decentralized like never before. This means that it is not only difficult for a modern data consumer to find and access the data they are looking for; they can’t even be sure it exists at all.
Collecting, cataloging and managing the vast amounts of data stored by an organization is now critical to getting the most out of it. In an effort to accelerate this move toward ensuring data is both discoverable and usable, companies are increasingly opting for a one-size-fits-all approach through a data catalog alone. While this approach is being hailed as a panacea for businesses, could it actually be holding them back?
Unite your data
To overcome the challenges associated with large data lakes, which are becoming more and more like data oceans, many data professionals have adopted data catalogs to bring some order to their large amounts of data. But what exactly are data catalogs, and why couldn’t they be the only solution to the challenge of unifying an organization’s data?
Data catalogs provide the frameworks and interfaces for managing and collecting metadata, or information about the provenance, reliability, quality, and sensitivity of data and data products. Put more simply, they allow users to see all the information they need about data stored by an organization, which is an incredibly powerful tool for, for example, the data engineers who build their organization’s data products – think of the data catalog as a large inventory list on a construction site, allowing data product builders to find all the right materials for their next project at a glance.
Data catalogs are therefore incredibly important for an organization that seriously wants to get the most out of its data. However, they are not a one-size-fits-all solution to the challenge of making data findable, manageable and accessible.
The data marketplace
As data consumers become an increasingly important part of a modern data strategy, companies must look beyond a catalog-only approach to their data tools and start thinking about an internal data marketplace. But what exactly is a data marketplace, and what can it offer data consumers that they can’t achieve with a data catalog alone?
When it comes to marketplaces, many data professionals often think about selling data externally. However, there are just as many benefits for companies that use internal marketplaces to solve discoverability, accessibility and governance issues. These internal marketplaces deliver data products within a company and help drive corporate missions and data-driven initiatives. Some organizations use the term data exchange interchangeably with internal data marketplace, but in general these all refer to the same thing: a platform to make data products more easily available to data consumers.
When we talk about data marketplaces increasing the availability of data products, it doesn’t just mean discovery. Data product delivery, a process that allows consumers to use data platforms or BI tools, is key to a successful marketplace. Without the right facilities, a data marketplace would look very much like a catalog, but in turn only half complete; it would be a bit like using the App Store to find the perfect app, only to find that instead of being able to download it, you have to submit a ticket to the app creator and spend a week waiting to use it.
Note that the main difference between a catalog and a marketplace here is that they serve two different functions: catalogs are great tools for data engineers building your data products, while marketplaces are the best way to provide data consumers with the data products they need to have.
Combine catalogs and marketplaces
So, between a catalog and a marketplace, which one is right for a modern business? The answer is both.
A catalog-only strategy fails because it tries to serve two types of users instead of being allowed to focus on serving the builders, while leaving room for a marketplace that empowers data consumers. This is not only bad for underserved consumers, but can also leave builders in the lurch.
While relying solely on a data catalog helps data engineers identify the right data for their products, it can also create large amounts of additional work. In a scenario where data catalogs are used by consumers, not only would it be much more difficult to find what consumers are looking for, but they would also be exposed to all the intermediate data used to build the final data products, leading to a large amount of responsibility and pressure placed on the teams that own each data product.
Complementing a data catalog with an end-to-end data marketplace that provides the ability to publish, discover, and access products delivers the best of both worlds: two purpose-built solutions for both groups of data users, making data engineers can use catalogs without the need to document everything in the data platform, so they can focus on the products they own and publish. Consumers, meanwhile, will have a marketplace where they can find and learn about data products and benefit from automated access and governance features.
There is no one size fits all approach to collecting your data and making it useful to everyone in your company. However, by combining data catalogs and internal data marketplaces, organizations can take advantage of two powerful tools that give them the best of both worlds to get maximum value from the data they own.
We have listed the best cloud storage.
This article was produced as part of Ny BreakingPro’s Expert Insights channel, where we profile the best and brightest minds in today’s technology industry. The views expressed here are those of the author and are not necessarily those of Ny BreakingPro or Future plc. If you are interested in contributing, you can read more here: https://www.techradar.com/news/submit-your-story-to-techradar-pro