The Hadoop ecosystem is an eclectic mishmash of start-ups, mid-sized vendors and IT heavyweights with products and services up and down the Big Data stack. Inevitably the ecosystem will consolidate and thin itself out through mergers, acquisitions and – unfortunately for some of these start-ups – bankruptcies.
Consolidation is part of the natural evolution of any given technology market after an initial period of frenzied innovation, and the Big Data market is no exception. I believe we are witnessing the start of this consolidation today. It will take several years to play out, but the first phase of consolidation is manifesting itself in the form of strategic technical partnerships between vendors that play in different segments of the Hadoop market.
These are real partnerships in the sense that the vendors are doing (and will hopefully continue to do) the hard engineering work needed to integrate Hadoop with various related technologies in order to make the larger Big Data stack more consumable by enterprise practitioners.
Many Hadoop early adopters have successfully developed pilot applications and proofs-of-concept projects and are ready to move these deployments to full-scale production. In order to do so, they will need to integrate Hadoop and related Big Data technologies into the wider fabric of their existing IT infrastructure. The next wave of early majority adopters will soon be in the same position.
Hadoop deployments and related applications do not live in isolation – or at least they shouldn’t. They must connect to various data sources, make efficient use of existing infrastructure where possible, and be compatible with application development platforms, business intelligence tools and other end-user applications already in wide use.
Deep, technical partnerships help make this possible. One such partnership announced recently was between Red Hat and Hortonworks. Engineers from the two companies worked together (and with the open source community) for months to make Red Hat Storage a Hadoop-compatible file system and to integrate the Hortonworks Data Platform with Red Hat Enterprise Linux with OpenJDK. Another newly announced partnership is between HP’s Vertica division and MapR. Enterprise practitioners can now deploy Vertica’s massively parallel analytic database on the same cluster of hardware as MapR’s Hadoop distribution, which enhances the usefulness and efficacy of both products.
There are other examples of Hadoop-related technical partnerships worth pointing out, including Cloudera’s partnership with Oracle and Hortonworks’ partnership with Microsoft. But more tight collaboration is needed, particularly between Hadoop ecosystem players in the data integration, relational database and analytics market segments.
This is an important moment for the Hadoop market and I expect and hope we will see more such partnerships in the Hadoop ecosystem in the coming months. Ultimately, some of these partnerships will result in lucrative sales channels for many of the Hadoop-related emerging start-ups and a handful will evolve into acquisitions and mergers. But for practitioners, the important thing is that these technical partnerships continue and make production Hadoop more accessible to the enterprise.
