2015 Big Data Market Shares

Premise

The software, hardware, and professional services markets for big data are still in formative stages – with many less-than-optimal solutions forming an uncertain space for users and vendors.  Leadership is still evolving.  To deliver big data solutions that provide significant business impacts, users need to rely on specialized tools from specialized providers, employees with varying levels of tool and analytic competence, and vendor partners that don’t always share each other’s goals. While the market will grow rapidly over the next few years, current market conditions, including current market share, will help impact future winners. See the companion Wikibon report Wikibon 2016 – 2026 Worldwide Big Data Market Forecast for the market development roadmap for the next decade.

Big Data Market Share Overview

Table 1 shows that the overall Big data market grew 22% from $18.3 billion in 2014 to $22.6 billion in 2015. The overall growth rate of the top 10 vendors was 24%, and the remaining vendors grew slightly slower at 21%.

Figure 1 shows the overall 2015 big data revenue for the top 10 vendors, and their market shares. The overall market share for the top 10 vendors is 33%, which is lower than each of the hardware (71% for Top 10), software (55% for top 10), and professional services at 40%, indicating the significant breadth of options users must make to assemble big data solutions.

IBM maintained its lead of 9.3% market share in 2015, down slightly from 9.7% market share in 2014 (due in part to the sale of its Intel server business to Lenovo). Teradata and Microsoft are new entrants into Top 10 in 2015, displacing Cisco and EMC.

Big data offerings span software, hardware, and services domains.  Beyond a handful of market leaders with offerings across the board, most big data providers participate meaningfully in only one of these segments. In the near term, enterprises will be relying on professional services providers to enable their applications, but Wikibon expects that the big data software ecosystem will begin to solidify over the next few years, enabling users to bear down on solving more big data-related business problems themselves.  Application software will replace services as the key market driver in the 2020s.

Vendor leadership in big data is characterized by four classes of vendors: 1) traditional database and analytics services and software providers, 2) analytics and tools software providers, 3) traditional infrastructure hardware providers, and 4) a smattering of rapidly growing big data software pure plays.  The traditional suppliers are racing to leverage their vertical and solutions expertise by adapting their own offerings and practices to deliver big data tools and solutions to their customer base.  Analytics and tools software providers and traditional infrastructure hardware providers are both adapting their offerings to accommodate big data requirements.  Small big data software pure plays are growing faster than the market overall off of their small customer bases and are reinvesting to build out competitive ecosystems focused on a variety of functional, developer, and operational roles for solving big data problems.

Worldwide Big Data Hardware Market Shares, 2015

While the hardware, software, and services solution space is in flux, there are some clear leaders today in each.  Figure 2 shows the hardware leaders. The software-led infrastructure focus of Hadoop and other Apache projects has led to ODM’s domination of x86-based hardware with 40% of the big data hardware business, supplying both to cloud service providers (public cloud IaaS is included in the hardware numbers) and some enterprise customers.

Branded hardware is led by the usual server, storage, and networking vendors.  HPE, IBM, Dell, Cisco, EMC, Oracle, and NetApp each have an assortment of server , storage, network offerings, and/or appliances that provide the infrastructure for big data, especially for enterprises, at varying levels of convergence.  We are counting AWS’s IaaS offerings as hardware since they serve the same function as on-premise hardware services – but are delivered through an alternative channel.  A portion of IBM’s hardware market share is similarly accounted for.

ODMs (original design manufacturers) procure hardware to support their own big data applications like AWS (although most do not have native software capabilities).  This hardware will be present in web-based enterprises where big data is integral to their business operations.

Figure 2 shows the detail. HPE Enterprise has done well with 7%, and has introduced an excellent hybrid architecture for Edge devices for Internet of Things (IoT) management (an important requirement for IoT processing). IBM has reduced big data hardware sales in 2015 compared with 2014 as a result of the acquisition of IBM’s x86 product line by Lenovo.

Worldwide Big Data Software Market Shares, 2015

Figure 3 shows the top 10 software vendors, which account for 55% of the big data software market. While the top vendors have a lot of room for growth, most emerging big data software specialists experienced higher levels of growth in 2015 and will continue to do so in the near future (see Table 2 in Appendix A for the details).

The big data software space includes database providers, data management tools, core big data software (Hadoop, Spark, Streaming) , and applications, analytics, and related tools (i.e., visualization and query).  While the market is still very diffuse, database and application software providers SAP, Oracle, and IBM are in the top 4 (see Figure 3).  These vendors have a significant breadth of offerings that are applicable to big data problems – including products that address every segment of the big data software market.

Companies outside of the top 4 tend to focus on product strengths derived from the analytic capabilities of their tools and big data professional service skills, but are starting to provide tools for adjacent segments that alleviate integration costs in the zoo of tools that plague the big data software market today.  Companies that fit this mold include:

• For example, application specialist Splunk is formidable in the IT Operations segment.
•  Microsoft Excel and PowerBI is still a tool of choice for many big data users.
•  SAS Institute is the leading analytics software provider.
•  Palantir has software as well a large professional services focused on government and defense.
•  MarkLogic NoSQL is significant in the alternative database management space.
•  Informatica is a leader in data management tools in general, and has been gaining traction in big data with tools aimed at this market.  Informatica is the leading ETL provider  and is adapting their product sets to big data problems.
•  Cloudera is the sole pure play among the top vendors – although we expect other pure plays such as MapR and Hortonworks to continue their very rapid 2015 growth into 2016 and climb the market share ladder.
•  Scores of smaller (often private or venture capital funded) companies are growing rapidly by offering point solutions to the many gaps in the current big data software ecosystem.

Worldwide Big Data Professional Services Market Shares, 2015

Professional services is the largest segment of the big data market today and will remain so through 2021.  The professional services market is comprised of the following types of vendors:

1. 1st-tier general IT services providers with the widest scale and scope and most vertical and applications expertise where big data solutions would be applied. IBM and Accenture are prime examples.
2. Big data specialists, the largest being Palantir and Mu Sigma
3. the 2nd-tier of large IT services providers with the same strategy as tier 1 leaders, but with less breadth such as CSC, PWC, and Deloitte.
4. Specialists in adjacent  domains that are repositioning to big data. Teradata (analysis and data warehouses) and Pivotal (development) are representative.
5. The hundreds of small professional services with general data warehouse, analytics, business intelligence , and vertical industry expertise.

Figure 4 below shows the big data professional services market shares for the top ten vendors, which account for 40% of market share. IBM leads with $1.2 billion in revenue, and 13% market share. Accenture has $0.5 billion, with 5% market share.

Big Data Leader Overview

The big data market exploded into the analytics universe about ten years ago and has been expanding at an impressive rate since, filling up any space not occupied in the analytics market, pushing established analytics categories aside, and weaving it’s way into a number of critical transaction application domains. Generally, the market looked pretty much the same in all directions: open source Hadoop and a mishmash of analytic pipeline tools. Today, however, the big data community is gaining significant experience regarding what does and doesn’t work and proto-segments are starting to take shape:

• IBM, Palantir, Accenture, Teradata offer consulting-led template solutions. These vendors are focused on creating semi-custom templates with varying degrees of repeatability. Each engagement typically contributes additional intellectual property that makes future engagements more repeatable and less custom. Some of that intellectual property is contributed back to the big data community as open source artifacts.

• Microsoft, Oracle, SAP, Splunk offer big data tools and application-led platforms. The tools and platforms that support big data applications are undergoing the most rapid innovation the data management sector has seen in decades. Multi-product platforms that can handle different workloads and data types are growing at the expense of individual products. In addition, pricing models and levels are dropping dramatically under pressure of cloud metering and open source. Notably absent from this list? Cloudera, Hortonworks, and MapR, each of which continues to search for a business model that turns users into profitable revenue.

• Dell and HPE will offer converged infrastructure. Private cloud is going to be with us for many years, even as it starts to adopt some of the architectural elements of hyper-converged infrastructure rapidly emerging in the public cloud. However, private cloud won’t remain “vanilla.” HPE and Dell Tech, as leaders in this sector, will package their offerings to appeal to specific categories of buyers, including big data and IoT.

Service-led Plays

IBM is focusing most of its growth expectations on its analytics and machine learning businesses. Just like its pivot 16 years ago to embrace open source, when it migrated its middleware platforms and tools to Linux, today it is migrating most of its analytic tools and platforms to Spark. IBM uses its tools and platforms to build customizable industry-specific templates based on deep industry expertise. The expertise comes from years of serving large enterprises across many industries with its consulting organization. Each engagement deepens IBM’s domain expertise and makes the templates richer. While each new customer gets the benefit of the incremental functionality, IBM gets to carry the cumulative expertise forward. Expect to see IBM fill out its template solutions “library” with a portfolio of vertical and horizontal solutions with premium pricing for Big Blue accounts. One potential weakness is that its cloud strategy is built from top-to-bottom on IBM technology: from infrastructure through the application platform to the data management layer to analytics. IBM will need to keep the entire stack competitive with AWS, Azure, and Google or potentially put its differentiation in analytics at risk.

Palantir launched itself as the pioneer in solving extreme challenges in machine learning for customers in defense and intelligence, among others. Based on estimates that their revenues exceed $1bn primarily on solutions built on custom programming, their value proposition clearly has some merit. But its growth period preceded the explosion of independent machine learning tools. Putting these tools in the hands of vendors like IBM may make a services-only approach less competitive. The key question will be to what extent they can build their solutions on the rising tide of 3rd-party technology.

Accenture is like IBM, but unconstrained by the need to use in-house technologies. It can serve horizontal functions and vertical industries with solutions it assembles from a virtual bazaar of 3rd-party technology. The core of its custom templates is an analytic record, which defines the data required to support the analytics in its solutions. Its biggest challenge will be to keep up-leveling its solutions as the supply of 3rd-party technologies get more repeatable and requires less customization, integration, and change management. At some point Accenture may return just to the business of implementation and change management consulting, as it did for client server enterprise applications. But that day is likely far off since big data packaged applications are emerging very slowly.

Teradata gave birth to business intelligence and big data. Its business model of selling appliances delivered both maximum performance through tight integration with its database and high absolute gross margin dollars. Both powered its rise to prominence. Now, Hadoop and open source pricing are turning that business model inside out. Teradata offers a version of the database that can run on non-Teradata hardware, which means it can no longer expect to capture higher margins on its hardware. An explosion of open source Hadoop-based MPP SQL engines for business intelligence have brought pricing way down and started siphoning off workloads from the more carefully curated data warehouses. Teradata is answering the challenge by targeting three audiences, changing its delivery and pricing model, and transitioning from platforms to solutions. The three audiences include the traditional business analysts focused on business intelligence, data scientists originally targeted by the rich functionality in its Aster Data acquisition, and Hadoop for data scientists who want industry-standard functionality and can manage the complexity that comes with it. The delivery and pricing model is also going through a major transformation. Teradata will offer versions of its flagship database as well as Aster on AWS and Azure over the next several quarters with fully elastic scalability and metered pricing. And, finally, it is focused on going to market like IBM with consulting-led solutions, not platforms. This transition will be the hardest because changing the skills and relationships for any infrastructure sales force is extremely difficult. But in Teradata’s favor it has thousands of consultants who can lead the transition.

Tools and Application-based Platform-led Plays

SAP shares a unique position with Oracle in having a mission critical database capable of big data analytics, the HANA in-memory database, and the industry’s broadest suite of packaged enterprise applications. The combination of applications and a database should drive adoption synergistically. Two issues have slowed HANA’s adoption. First, a database can take many years to mature for mission critical uses and HANA is no exception. Second, and more unusual for an application vendor, SAP’s own core application suite, which hasn’t leveraged big data in the past, is slowly being rewritten over a period of years to take advantage of HANA. But SAP has also been slow to build new applications that leverage the in-memory speed and big data analytics capabilities in HANA. SAP has also been evangelizing HANA to 3rd-parties, but adoption has been slow. HANA’s pricing puts it out of the reach of buyers accustomed to the open source ecosystem, though it’s comparable to Oracle’s flagship database. Expect to see SAP emphasize its complementary HANA Vora database that works with data stored both in HANA and Hadoop in order to open more opportunities.

Oracle’s core 12c DBMS remains the gold standard for mission critical systems of record. However, it faces two challenges: (1) scalability on a shared-nothing cluster and (2) pricing. With respect to shared-nothing scalability for transaction processing, 12c has nothing to offer (nor do any of the OLTP database incumbents), and it’s not clear when it will. Oracle’s appliances stretch as much absolute performance from a shared-storage database as is possible. While the appliances deliver absolute scale, they also raise the second challenge, pricing. For those who want a price level closer to the open source ecosystem, Oracle offers the very low-end MySQL database. However, Oracle recently added a federated query capability to 12c that lets developers seamlessly access data in Hadoop, including their version of the Cloudera distribution, and NoSQL databases such as Cassandra. This query engine should allow Oracle to be competitive when their customers want to extend their solutions with open source databases. As far as Oracle’s enterprise applications, they seem to be adding big data functionality opportunistically, just like SAP.

Splunk is the anti-Hadoop. It is an application framework for analyzing large volumes of machine data that works out of the box. It is easy to collect, analyze, and present data without the complex setup, administration, and development associated with Hadoop. Splunk also sells several packaged solutions including security and operations management. While Splunk’s greatest strength is its “no muss, no fuss” simplicity, that same strength by necessity makes it less easy to stretch to unanticipated use cases. Splunk also has a relatively high price point, which might become problematic as data volumes continue to grow exponentially. Splunk is adding more general purpose capabilities, starting with the MongoDB database, to support a wider variety of applications. As long as it can continue to maintain its out of the box simplicity, expect to see it as the most direct competitor to Hadoop on-premises.

Microsoft has come from behind in big data and the cloud by leveraging its on-premise enterprise products. Unlike Amazon and Google, Microsoft can now build enterprise products on Azure first and then periodically “snapshot” them into packaged products for on-premises deployment. That is only the first step in their strategy of offering hybrid solutions. They are also building a PaaS layer where its many data management and analysis products offer a data management “fabric” where each component reinforces the rest. The hybrid-cloud equivalent of Visual Studio and Systems Center will greatly simplify developing for and managing this platform. Microsoft will be able to make the boundary between private and public cloud very hard to distinguish. In addition, more than any vendor, Microsoft will have great flexibility in pricing when bidding against either on-premises or public cloud competitors.

Converged Infrastructure Plays

HPE’s big data software offerings to date have largely centered on the highly regarded Vertica MPP SQL database it acquired. How the Autonomy technology that handles semi-structured data fits in has been less clear. Recently, HPE dropped its efforts to build an IaaS cloud and ported its HAVEn machine learning APIs to Microsoft’s Azure cloud. The API’s prospects aren’t clear since standalone machine learning tools and frameworks are widely available from dozens of vendors. Integrating the API’s with data sets that turn them into horizontal or vertical solutions would be a promising next step. After HPE dropped its plans to compete with AWS it has continued to sell infrastructure for private cloud deployments.

Dell Many people assume that x86 servers are low-margin commodity items. The reality is that Dell’s server business can take advantage of EMC’s higher-touch channel to sell sophisticated server networks to sophisticated customers. And as long as EMC’s products continue to be sophisticated, EMC will still need its high-touch channel. More intriguing, as systems continue to evolve from server-centric to storage-centric, the combined company will be able to design hyper-converged solutions better than either company on its own for the large portion of the market that still wants and needs to manage private clouds. EMC also has a targeted but vibrant presence in analytics that Dell could turn into something higher volume.

Worldwide Big Data Absolute Revenue Growth Leaders, 2014-2015

Figure 5 and Figure 6 below look at the vendors who are growing the fastest in big data. Figure 5 shows the top ten vendors by revenue growth. The Original Design Manufacturers (ODMs) show the most growth with $553 million, IBM is second with $333 million, and Spunk is third with $193 million. These ten vendors account for 46% of the market growth.

Figure 6 shows the percentage growth from 2014 to 2015 for the top vendors with 2015 revenues of $40 million or more. AWS has grown its public cloud IaaS revenues by 166% to $149 million. Hortonworks has grown 151% to $118 million, and Cloudera has growth 100% to $182 million. All the high growth companies are cloud and/or software vendors, with the exception of Teradata, who grew 50% to $432 million,

Action Items

Many different vendors provide the multitude of function, products, and services that make up the big data solution pipeline. Wikibon does not believe that the big data market is a “winner take all” market, if only because open source technology will moderate any specific vendor advantages.  Therefore, the  best fit for your enterprise’s big data entry point may not be a Top 10 provider.  So choose carefully and consider:

• Product offering quality – reliability, support, stability, breadth and depth
• Track record for applying it to the specific solution set in your industry
• Related business process compatibility, i.e. extensibility to related to adjacent applications and processes
• Incumbent relationships – especially where you have experienced success in related applications in the past
• Individual enterprise’s risk:reward tolerance – can you afford to take risk, make a mistake, and then recover?

Appendix A – 2014 & 2015 Worldwide Big Data Revenue ($M) by Vendor

Table 2 below gives the 2014 and 2015 big data revenues by vendor, and includes the market share and growth calculations. Overall, the big data market grew from from $18,319 in 2014 to $22,605 in 2015, a growth rate of 23%.