Evolving Organizational Dynamics for Cloud Native Applications

Premise: As more businesses refocus on building business-differentiating software applications, the need to hire and retain top talent has never been greater. For CIO and business leaders, creating an internal culture that supports rapid software development is critical to harnessing the power of Cloud Native application development. Wikibon looks at how leading companies are establishing the new best-practices for evolving organizational dynamics to hire, manage and retain the groups that are shaping the next generation of business.  

[In this three part series of research on Cloud Native Application Platforms, we will explore the Markets, Technology, Operations and Organizations that enable next-generation applications. Cloud Native applications are the modern, modular approach to building applications that are focused on web-scale, mobile-first, and real-time data.]

Part I of the research – “Cloud Native Application Platforms – Structured and Unstructured”

Part II of the research – “Technical Dive into Cloud Native Application Platforms”

While it’s often said that ‘every company must make software development a core competency’, this theory disrupts the traditional software supply chain that has been established over the past 20 years. IT leaders must determine if they accept this premise of software development, and if so, how do they create a culture that can succeed in this new model.

The core premise that every company must evolve into a technology company, or more specifically a software company is one that is being actively debated within our industry. The companies that have disrupted many traditional industries (e.g. Amazon, Netflix, Uber, AirBnB, Square, eSurance, Twitter, Facebook, etc.) have fundamentally been software-centric companies. As disruptive threats come into existing industries, incumbent companies are questioning how they respond.

In almost every instance of disruption, the disruptor is taking advantage of two core elements:

1. Lower cost models that bypass existing supply-chains to deliver services to customers. For example, using the Internet to deliver services and experiences rather than employing an expensive salesforce or physical facilities.
2. Shifting demographics and communication models that are more accepting of collaborating and community-centric approaches to B2C (Business to Consumer) engagement. For example, willingness to share assets (rooms, vehicles) or information more liberally through social means such as social networks or online communities.

But not every disruption is being driven by external competition. In many cases, existing companies are realizing that their core products are no longer the primary source of value to customers. Instead, the value is evolving to a set of systems and information that is associated with those core products. One example is the ability for farm equipment suppliers to gather, process and disseminate information about weather and planting patterns to farmers to increase crop yields. Another example is the shift of automobile manufacturers from marketing gas mileage or exterior styling to the in-car entertainment experience when selling vehicles. In both examples, the core products are not being eliminated, but the derived value for customers is significantly shifting from core product to overall experience. Existing companies must be able to adapt in order to capture revenues from these shifting customer needs before a disruptor moves into their markets.

Understanding the Scope of the Changes Ahead

Technology-driven organizational changes have occurred for decades. Over time, the core value a business provides to the market will get commoditized and they must look for new ways to drive revenues and differentiation. For example, manufacturing-centric companies began to realize that their source of differentiation was not the products that were created within the factory, but rather the speed and efficiency used to create those products. This type of thinking, where companies optimized for speed of “work flow”, became popular with the Toyota Management System (“The Toyota Way”) in the 1980s, disrupting the automotive industry. Dell Corporation executed a similar approach around optimized supply-chain-management in the 1990s to disrupt the Personal Computer industry. And recently, Amazon Web Services (AWS) has removed more elements of the IT supply-chain by delivering Cloud Computing services on-demand, over the Internet.

For companies attempting to leverage software in new ways, to remain competitive in their markets, there are several core focus areas:

1. Software Development and Software Operations become areas of Core Competence – These transformation are not targeting companies that primarily used packaged, commercial software. More and more companies are reversing the trends of the ’90s and ’00s to offshore technical skills, and are new focused on hiring top development talent to rebuild the internal skills needed to differentiate the business through software.
2. Organizing Teams and Processes around Outcomes, Not Functionals Areas – The best companies are shifting from PROJECT focus to digital PRODUCT focus. This focus is typically around external or internal facing digital products and process. The “products” must be focused on an end-to-end “customer” experience, not silo’d functions.
3. Understand the Importance of Collaboration and Communications – As these digital products tends to be more dynamic in nature due to the frequency of changes and built-in feedback-loops, the role of communications and collaborations within the teams is more important than ever.
4. Measurement is the Real-Time Feedback Loop – In the digital world, nearly everything can be captured, measured and analyzed. It is critical that these measurement systems are inherent in the digital products and that the information is freely shared across all aspects of the team, in order to find ways to improve the product and the end-user experience.

Case Study: TicketMaster – Team Alignment (DevOps Enterprise Summit 2015) 

Starting in 2011, Ticketmaster and LiveNation realized that they had two distinct problems. The first was that they had allowed too much technical talent to migrate to offshore resourcing. The second was that their growth in developers and development teams was overwhelming their limited number of operations people. This was a business that was focused on delivering a great digital experience to their customers, and their technology skills were not properly aligned to that business priority.

Their original applications in the 1970s were built on Mini Computers, with today’s environment consisting of ~130 core applications, with 1000s of elements within those applications. They had 110 development teams, with growth of +230% in developers and only +12% increase in Operations from 2011 to 2014. Ultimately they had to determine a way to better align the pace of resources, and get all teams coordinated on a common set of tasks.

From the human side of their transformation, they focused on three core areas: Empathy, Empowerment and Metrics. They internally interviewed 75 employees in 75 days to better understand their problems, and get buy-in for the changes they anticipated making. They made all groups spend time in roles that directly interacted with customers, to get direct perspective on the problems they were trying to solve. The transformation also focused on empowering individual groups, but also reinforcing the mantra of “One Team, One Message, One Goal”. This focus on empathy within teams and with end-users is an critical aspect of being able to grow the teams throughout the phases of the transformation.

Their organizational model was focus on the concept of  “4 in the box” – Product Lead, Engineering Lead, UX Lead, Process Lead. This brought together the multiple disciplines that were needed to successfully build the new applications that were needed for their business. This became the Ticketmaster variation on “DevOps”.

From a metrics perspective, Ticketmaster focused on Business Metrics above System Metrics. They wanted to make sure that all data could be translated back to impact on the business, not just IT goals. They focused on trying to measure and instrument everything possible, and made the data source open across the company.

The Flaws in Two-Mode Transformation Models

There is some sentiment, within the technology industry, that the nature of these Cloud Native application is so much different than existing IT that these projects should be isolated into their own category. In essence, create an entirely different mode of operations. Wikibon’s research has identified that this approach has several significant flaws, and there are few real-life implementations that have properly evolved beyond the first implementation stage.

There are three main problems with this approach:

1. The Old vs. The New – When transformation is classified between new and old, it creates a natural fragmentation about what is “interesting” and “non-interesting” to the business, which inherently leads to undesirable behaviors from both teams. The “new” group receives attention without generating much (initial) revenues, while the “old” group receives much less attention while supporting the bulk of the applications, budgets and revenues for the business. The realities are that legacy systems are rarely eliminated. This segmentation often gives the new group the false impression that there will not be a need for legacy integration.
2. Two Modes, One Management Chain – By implementing a two-mode system, the business is attempting to address the classic innovator’s dilemma, whereby the new model disrupts the old model. Some businesses address this challenge by spinning out a subsidiary, which is allowed to operate under different business model rules and different business goals. But when IT attempts to run both modes under the same IT management structure, they force the IT management to choose a blended set of goals and metrics that can often be harmful to both groups.
3. Failing to Plan for Scale – While many organizations can have a small group create a “lighthouse” model for small-scale success and transformation, that same group is often not capable of scaling the transformation to broader segments of the company. Still other groups are not always capable of institutionalizing the changes. There are different skills required for each of these three phases. Not understanding that these distinct phases, and feedback loops, are necessary will often leave companies with two highly fragmented and frustrated parts of the organization. [NOTE: This three layer concept has been discussed many times by Simon Wardley, and Robert Cringley].

• At the recent DevOps Enterprise Summit, this paradigm of three-mode transformation was evident over and over from Enterprise customers in all vertical markets.
• EMC’s President of Systems Engineering, Chad Sakac (@sakacc) highlighted this recently in a discussion about if the small group is a “revolution”, if they are successful and then integrated, they are now the thing that they are fighting against.
• Within the framework of DevOps evolution, Chef’s Michael Ducy does an excellent job of highlighting the journey of many Enterprise companies on The Goat Farm podcast.

It is critical for companies to understand that legacy integration and longer-term scaling of the transformation are important for the business to succeed. Unless a company is a start-up, these elements must be factored into any transformation plan.

Case Study: Sherwin Williams – Modes of Transformation (DevOps Enterprise Summit 2015) 

Over the past 3 years, 150 year old Sherwin Williams has attempted to migrate their operational model from Waterfall to Agile, using the Scaled Agile Framework (SAFe). Throughout the transformation process, they aligned themselves to many DevOps technical best practices – implementing Infrastructure-as-a-Code, using versioned Software Repositories for all code (software and infrastructure), implementing automated testing, versioning data, and building early prototypes.

The key to Sherwin Williams success is less about technology and more about their understanding of the organizational dynamics needed to guide the business through this transition. Key lessons learned:

• DevOps needs an internal salesman and a politician. Without these roles to drive cross-business collaboration, it’s a dead end.

• Internal communication and collaboration is key. Let the business be heavily involved in defining the roadmap, not the IT organization. Roadmaps should be clearly aligned to business strategies and forecasts.

• It’s critical for conversations to be defined as “pipelines” around products, not silo’d group projects. There needs to be an understanding of trying to make them bigger than the individuals involved. Keep them focused at the evolving business level.

• Internal partnerships and transition plans are more important than designating a “DevOps team”. There must be a win-win mentality that spreads across the company culture, not one of isolation or separation.

Driving Change from the Top and the Bottom

Successful Cloud Native organizational changes tend to follow several core principles:

1. Start from the Bottom Up – Not only are the application frameworks different (e.g. Microservices, 12-Factor Applications), but the operational models (e.g. DevOps) are also much different than traditional IT models. Successful implementations typically begin with small groups of people that are not only passionate about the changes in technology, but are willing to work through the challenges of more collaborative operations between Developers and Operations teams. This entrepreneurial mindset is highlighted in The Lean Startup and Jez Humble’s “Lean Enterprise”.
2. Start Small, but Expect to Scale – It is critical to identify ways to create change that are small and can be completed in very short periods of time (e.g. less than 1 month, often within 2 weeks). This scope will help the early groups see success (or fast failure) and avoid analysis paralysis. But it is also critical for the early team to not bypass all “scalability thinking”, as it can be very easy to design a small transformational process that becomes completely useless when the transformation eventually needs to scale across more groups within a business.
3. Choose Visible, Business-Impacting Applications – All too often, transformational groups will choose a new, greenfield application in which to highlight the possibilities of the “new model”. While simpler, these approaches often fail because they are novel and not business impactful. Successful transformation teams will typically target a visible business challenge and apply the new technology to that problem. The implementation may be a new application, but it often (also) requires integration with existing business systems to truly be impactful.
4. Measure Progress – While the symptoms of problems are often easy to identify, the causes are not always as simple. Successful teams not only measure for baselines, but they measure many variables throughout the change process to determine progress and areas for on-going improvement. Often times, the solution to problems will be different than a “gut feeling”, and only be revealed by examining the data.
5. Market Successes Internally – While early transformations may be simple at smaller scale, it is critical to begin getting broader support for change as the scope of the transformation grows. This requires an effort to internally “market” successes and progress to both adjacent teams and management groups. They not only need to be aware of the progress, but educated enough to buy into the changes. People inherently want to be associated with successful projects, so finding ways to be inclusive will help the transformation grow beyond the earliest stages.
6. Identify Executive and Technical Champions – Any Cloud Native transformation is an ongoing process that eventually needs both top-down executive support and broader team support. It is critical to enable these individuals with the data and success stories that will allow the details of the transformation to spread to more areas of the company.

CSC’s Paula Thrasher (@paula_thrasher) discusses how her team has been able to help large organizations within the US Government be able to balance the challenge of large goals with the need for smaller, faster project delivery models.

Reinforcing the Change Patterns

While many technologists and leaders will attend events throughout the year to learn about the latest trends, very few make this type of learning an on-going part of their role. The best companies leading these transformations are adopting models where continuous learning is part of their evolution.

Learn from Other Company’s Mistakes

While this type of transformation may be new to a specific company, there are many companies (in every industry) that have already gone through similar transformation. It is invaluable to seek their learnings, or seek the guidance of consulting groups that specialize in organizational transformation.

Rearrange the Chairs

One of the most common and simple techniques that teams follow when beginning a transformation is to physically co-locate all members of the team in the same room. Not only does this approach simplify collaboration with teammates, but it eliminates any barriers (elevators, bad weather, poor video collaboration tools, etc.) that can be used as an excuse for why people aren’t communicating.

Internal Events

As part of their internal communications and collaboration plans, companies such as Capital One are hosting internal events to showcase their technology to internal customers. These events, similar in format to a VMworld, AWS re:Invent or VelocityConf, are designed to help groups showcase their latest technology and to network with peer groups and prospective internal shareholders. These events force the technology teams to think about their work as a marketable service and to generate excitement about the possibilities that could evolve from their creations. In some companies, external speakers were brought in to reinforce best practices and on-going learnings.

Internal Hackathons

New ideas often come from unlikely places, but not enough groups give themselves the free time to experiment outside their day-to-day activities. More and more companies are beginning to take 1-3 days and host internal hackathons. These events give teams the opportunities to collaborate in new ways, as well as generate some internal competition between teams to come up with the most creative ideas. An interesting byproduct of the hackathons is that it will often force companies to evaluate how they engage with open-source technology or open-source communities, as the output of the hackathon might be best served as an open-source contribution. More and more Enterprise companies are finding that open-source contributions are not only valuable to the on-going momentum of their specific industry, but it also creates a more desirable working environment for their internal developers and future hires.

Internal “Dojos”

While events and hackathons are excellent short-term opportunities to drive cultural changes within a company, more advanced businesses such as Target have taken it to the next level by making transformation a permanent part of their on-going learning process. Many open community groups, such as Cloud Foundry Foundation and CoderDojo have also adopted the the dojo model of collaboration and learning.

Case Study: Target – Internal Dojos – (DevOps Enterprise Summit 2015)

The Target dojo is now 4-5 months old and they have had 200+ learners come through the process in that timeframe. The program takes 3-6 months of normal learning and condenses it into 1-2 months. The program is a fully immersive experience, whereby an entire team comes to the dojo and brings their actual work problem. This is not a simulation environment. After learning new skills and operating models, as well as solving their specific problem, the team returns to the original offices and is tasked with sharing their new knowledge the broader teams around them.

While the 30+ day “Challenges” are defined to create the biggest change and impact larger problems, Target has also created smaller programs called FlashBuilds (1-3 days) and OpenLabs (90 minute “office time”) to allow individuals to also experience the dojo environment and expert lessons.

Two of the critical success factors for Target were aligning dojo projects to the overall Target strategic roadmap and directly involving executive staff as well as technical staff. By involving the executive team members, they get a first-hand look at the types of changes that are coming, as well as the evolving collaboration model that drives success. This helps the executive team speak the same language as the technical teams driving the underlying transformation.

The dojo has a unique identity and is internally branded with a name and logo. Participants gets t-shirts and laptop stickers to help spread awareness of the dojo program throughout the company. As each team completes a session, their demos are made visible throughout the program on large displays and online video programs.

Other Key Learnings:

• Pushing the concept of MVPs (minimum viable products) vs. Monolithic roadmaps
• Successful challenges require a good charter
• Spread the focus to multiple areas – allow many groups to express creative ideas
• Find the right real-estate (large, open spaces)
• Expect the unexpected (e.g. there will occasionally be the loss of key people on a project)
• Get comfortable with uncomfortable – change is difficult, so prepare

Customers are Becoming the New Vendors

An interesting trend has been occurring for the past 12-18 months, as more business seek to reverse the outsourcing trends of the ’90s and ’00s, is a public push to re-hire software developers internally, rather than be so reliant on commercial software. This began as the demand for Data Scientists increased, with companies struggling to capitalize on the potential of Big Data Analytics projects.

In years past, large corporations were hesitant to be “public case study” customers. Now, these companies are speaking openly in high-profile keynotes. Companies such as GE, Capital One, Target, Nordstroms, Nationwide Insurance, NetFlix, WiPro are all using their environments as opportunities to attract the best software development talent.

In additional, more Enterprises are beginning to release internally developed code as open-source software, as a mechanism to show potential hires that their culture is open to using whatever technology will help solve business challenges. Some recent examples include Quintiles, Capital One, Walmart and EMC, along with hundreds of other examples. These contributions are radically changing the dynamics between “customers” and “vendors” and “communities”, where the definitions are each is beginning to blur.

Building Platforms vs. Building ON Platforms

As the previous research papers highlighted (Part I and Part II), companies have many options when it comes to deploying and operating Cloud Native applications. They can build their own platform, deploy on packaged platform (commercial or open-source), or consume platform services.

There is not yet a defined best practice for platform consumption, since there are examples of companies that have been successful with all three methods. The crucial factors are typically:

1. How much flexibility is needed from the platform, that the company needs to control? The greater the level of customization and control, the less structure should be defined by the platform.
2. How much time and resources can the company afford to spend building or maintaining the platform vs. developing applications or driving operational efficiency?
3. How much does the platform allow the company to dictate their internal organizational model, or does the platform dictate the model to the company? (see: Conway’s Law)

Companies must look at their maturity model and determine what trade offs will best fit their internal company culture. The culture of Netflix is different than the culture of an Insurance company in the Midwest of the United States, or a Telecom provider in Europe. Far too often, companies that fail in their transformation do not have a practical understanding of their existing maturity model. The typical Enterprise transformation can take many years, and will not occur just because a company is sold a “cloud in a box” solution from a vendor. It is critical to be realistic about existing technical skills, ability to change internal culture, and overall support from management to align the transformation to business goals.

Measure Everything – Use Data to Drive Improvements

As more products become digital and digital-enabled, companies begin to gain the benefit of direct platform-level visibility into customer usage patterns. This is a significant shift in how many companies interact with their customers and the overall marketplace, where it can often be difficult to collect feedback or understand usage due to several levels of “middlemen” between the company and their end-users. Digital platforms provide unprecedented insight into customer behavior, if the platform is instrumented to properly collect data and monitor interactions.

Many companies miss this opportunity because they are overly focused on feature lists, or UI nuances. While these are important, they will only be partially valuable if they can measure interaction and input. While this is a technical element of the Cloud Native applications or platforms, the need for data-centric feedback loops is also critical to how information is shared within a company. Leading companies  are adopting open data-sharing models regarding application monitoring, because the same data can be useful to product designers in different ways than sales, financial analysts or customer support groups.

In this interview, Wikibon speaks with Adrian Cockcroft (@adrianco, VC at Battery Ventures) about how important it was for Netflix to build monitoring into every aspect of their online video-streaming services.

Action Item: As more company engage on a Cloud Native organizational evolution, it is critical for them to understand the importance of talent and collaboration. Talent, both in impacting technology and managing change, is in high demand across the globe. Companies that hire and retain the best software development talent have a significantly better chance to excel in this transition to driving new value out of their businesses.