Data Mesh: Delivering Data-Driven Value at Scale

This book is the missing manual. If you're in any data driven organization &/or managing an organization through a transformation with the goal to manage distributed data as products, this is a must read.The book is technology agnostic. What it does very well is lay out a very thorough map for what shape data, technologies & teams should look like, what functions they should fulfill & how they should all interoperate as a system to manage data as product successfully. It provides a somewhat idealized vision, but I'm all in as I've seen some of this in action & helped put some of these systems in place.At MarkLogic we helped organizations implement solutions that managed data as product. (I no longer work at MarkLogic & have nothing to gain here in promoting them, just sharing my experience.) MarkLogic integrated data from various silos within large organizations to create data products. Above the silos of data were silos of people that also required integrating. Change management was always a challenge. I recommend How Stella Saved the Farm & The Phoenix Project to help others start to understand the types of changes they need to embrace to succeed. But what was missing was the architectural guide. We made up for it with slides, white papers & tribal knowledge, but this Data Mesh book truly captures some of the best practices I saw organziations put into place.It's all here: start with the business goal and work your way back, data product as self-contained unit, the sidecar pattern, embedding policy as code, start small & fast for a big win, iterate, progressively enhance products, govern with provenance & lineage, the changing roles that support this endeavor, operational & analytical systems & the desired intertwingling, + more..We didn't use the term Data Mesh & didn't do everything captured here. The challenge with any vendor is we tend to think of ourselves as the center of the universe. MarkLogic's way to integrate with a larger world of data was Semantics. But #DataMesh provides a higher level of abstraction.The book's just really well done. Technology vendors don't know how to document like this. Vendors focus on their own product's knobs & levers. What they fail to understand & illustrate for their customers is that their particular technology works within a system of technologies to achieve a business goal. It never works alone. So I HIGHLY RECOMMEND this book. If you can grok the patterns outlined here, you can fill in the gaps with the technology, people & process right for your solution.

This book elaborate the origins of data mesh which is a response to ancient arquitectures like warehouses or data lakes (at the writing of this, not so ancient) and promise a better way to create value from data where other approaches have failed to do so.It's a recommended reading for developers, architects who want to implement or know how to design a system for data analytics and suited for an high volatile environment that needs to scale through time.The concepts presented along this book are technology-agnostic but it is useful to know some technologies about databases, cloud providers, and agile methodologies in order to have a better context, again not a must but useful.By the way the coloured brushes used for pictures made the reading enjoyable.

Mesh advances the analytics domain in ways I'd not considered, but should have. My career has been in technology in small-cap public companies, and historically I've embraced the notion of a centralized analytics team creating and supporting: data pipelines, ELT/ETL, data lake, data marts, etc. Mesh blows up this mental model in favor of decentralized data elements surfaced via API's, and closer to their domain ownership.Dehghani posits a product model for analytic data, where functional teams surface data from operational systems, microservices, etc. directly. No longer reliant on a central team to provide a single source of truth; Rather, they expose data in a more raw form (my words), servicing both their own domain as well as others through defined data ports. She suggests leveraging developers for this work, avoiding the challenge of attracting and retaining resources in the analytics domain (data architects, etc.). Dehghani uses examples from a fictitious streaming music service that seems to mirror Spotify.There's certainly much more, and I recommend Data Mesh to anyone in a technology role. While I applaud her vision, and agree, directionally, with Mesh, there are three areas that I struggled with.First, the Mesh model seems much better suited to a larger tech company such as an Apple, Amazon, Spotify, etc. In my experience, SMB's just don't have the budget/headcount to support the expansive vision that Mesh espouses.Second, Mesh leans heavily toward "states rights" and away from central control. The success of this model hinges on knowledgeable, talented technical resources willingly collaborating for a greater good. That, in turn, presumes the existence/extension of global standards, the right culture of adherence to standards, adequate resources to serve the greater good (supporting deprecated API's for the benefit of other domains, for example), etc. In my experience, when the going gets tough, functions/domains tend to become more self-serving.Third, the vision of Mesh is fantastic, and fantastically broad. With data governance and privacy controlled through code. Consistent documentation, both human and computer, available for all data objects. Eschewing data lakes, warehouses, marts in favor of joins across domain API's (again, my words). Although Dehghani seeks to avoid monolithic solutions, I had to wonder if that isn't exactly where a fully-realized Mesh infrastructure ends up.Data Mesh is a bit redundant in pushing its vision for decentralization. It reads very much like a graduate software engineering text, with many references (many are excellent). Overall, very well written and a compelling vision that aligns perfectly with a microservices strategy, digital transformation and a product focus. Although it would be counter to Dehghani's specific vision of avoiding monolithic solutions, I have to wonder if a younger analytics company like Snowflake will try to run with it.

This book made me to realize that there actually aren't that many data books that provide a complete vision and not just focus on a limited set of functionality. Data Mesh is only partially about technology but has a lot to do with organizations, change management, empowering the right people and recognizing the domain specialists' needs. This book is not a blueprint of data platform but a blueprint of a data-driven organization which makes it mandatory read.

This book provides a comprehensive foundation for understanding data mesh from a technical and social context. It’s a must read for anyone who wants to learn more about how to manage data in collaboration with modern software development.