- March 31, 2021
- 3 minute read
With the acceleration of public cloud adoption among organizations in the past few years, developers and DevOps teams got used to the fast pace of trying new technologies, spinning up new instances and experimenting with cloud capabilities.
Developers and DevOps teams assume more budgetary power and often don't need to go through a long, bureaucratic approval process to utilize cloud services - they can swipe their corporate credit card or better yet - leverage an existing AWS, Azure or GCP account for their purposes.
The distributed nature of development teams in many places around the globe including Eastern Europe, Asia and South America add more complexity to track, monitor and control the pay-as-you-go SaaS-based cloud pricing model.
Cloud Cost Optimization or FinOps is a relatively new framework that addresses the need to optimize and control one of the largest cost line items in modern companies' budget. These companies rely heavily on the public cloud as part of their infrastructure, sometimes without the tools, visibility and processes to track, forecast and control cloud costs.
This guide describes FinOps, the current state of FinOps, FinOps adoption, how FinOps teams are built, common challenges FinOps practitioners face during the adoption of this field in their organizations, how to solve these challenges, FinOps Tooling, etc.
FinOps stands for Cloud Financial Management. It is the practice of bringing financial stability to the variable spend model of the cloud, enabling distributed teams to make business trade-offs between speed, cost and quality. FinOps is essentially a cultural practice that an organization follows, that enables them to manage their cloud costs using ownership, accountability, collaboration-based model.
FinOps team in any organization is generally a cross-functional team called Cloud Cost Center of Excellence (CCoE) that interacts with other components of the organization in order to manage the cloud strategy, governance and best practices that the organization needs to follow to maintain and transform their business. The stakeholders in FinOps are:
This category refers to the people who work in a company’s finance team or procurement team. Some roles that fall under this category are Financial Business Advisor, Technology Procurement Manager, Strategic Sourcing, etc.
This category refers to the people who are in the engineering team such as software developers, infrastructure architects, etc, and people belonging to the operations team of a company. They introduce cost as a metric similar to other performance-based metrics they consider. Some roles in this category are Lead Software Engineer, Principal System Engineer, Cloud Architect, Engineering Manager, Project Manager, etc.
This category refers to the executives or the head of a division whether it is related to engineering or some other correlated team. Some roles that fall under this category are VP/Head of Infrastructure, VP/Head of Cloud Management and Operations, CTO, CIO, etc.
This role includes a wide variety of people mainly in the leadership roles of a particular product or a business. Some roles that fall under this category are Director of Cloud Optimization, Cloud Analyst, Business Operations Manager.
The principles of FinOps are as follows:
1.Teams need to collaborate
FinOps is divided into three core phases, that signify its main stages:
Inform: This phase aims at generating visibility and accountability between teams so that the teams become accountable for their spending and realize how much they are spending and is it actually needed.
Optimize: The phase involves the empowerment of the teams to identify and measure the efficiency optimizations and then set the need and requirement of resources, primarily focusing on fixing/replacing unoptimized infrastructure with an optimized version that is cost-effective.
Operate: This is the execution phase of FinOps that enables the goals of Technology, Finance and Business Development.
The FinOps Foundation has defined the adoption of FinOps practices and tools as a three-stage evolution. The organization’s status within the process depends on the duration of practicing FinOps, but more importantly - on the extent to which the FinOps approach has spread into and across the organization.
This stage refers to the beginners who are just getting started with the FinOps adoption. Crawlers generally include newly formed FinOps teams in an organization or a small company/startup just getting started with FinOps practices.
Experience: Less than 1 year.
The stage refers to the people in the intermediate phase who have some experience with FinOps and have been using this practice for a while. This category mainly consists of the medium-sized organization or FinOps team in a large organization that has 1–2 years of experience using the FinOps practices.
Experience: 1–3 years.
This stage refers to the people who have become experts in the FinOps practice and adoption in the organization. Often the targets and goals of runners are very advanced as compared to the earlier two stages. The runners section mainly comprises people who belong to a very large organization with a large workforce and cloud infrastructure requirement.
Experience: +3 years.
Although FinOps practices are something the people belonging to various sizes of organizations can adopt whether it is a small startup or a large organization with thousands of employees. Still, it is observed that the adoption of FinOps practices is more prominent in a larger organization as compared to a small company or a startup. An open survey of this highlighted the fact that 70% of the people out of all the practitioners of FinOps belong to large organizations. 50% of FinOps practitioners belong to companies having 10,000 or more employees.
The distribution split of the number of employees in a company that follows FinOps practices.
The public cloud usage of an organization is quite diverse ranging from Public Cloud-First, Hybrid Infrastructure, On-prem Private Cloud, Traditional Data Center, or often a combination of a couple of these.
When it comes to FinOps, Spend and Need are directly proportional to each other, in simple words the higher the spend on the cloud the greater the need for FinOps in the organization. The need for FinOps was felt by practitioners when their spend on cloud usage reached around $1 million to $10 million annually, and naturally when it is higher than this. The majority of the organizations who have not been involved in FinOps practices feel a dire need for it when they reach this amount threshold.
It was found out that on average around 4 people were involved especially in the practice of FinOps in an organization that followed the practices of FinOps and this number is currently on the rise with them increasing the percentage of people involved in FinOps practices by around 47% from the year before and expected growth of 75% in the succeeding year.
FinOps advisory services by trusted external consultants can ease the adoption, especially for enterprises new to the cloud. Some train and certify your teams as FinOps Certified Practitioners. Some of these service providers are HCL, Tata, Accenture, Deloitte, KPMG, Capgemini, Wipro, etc.
Innovative, cloud-centric service providers like GlobalDots implement no-code tools to automate the trade of commitments and reserved instances. This allows the customer a great level of independence in day-to-day FinOps practices. Yet, the advisor should remain in the picture for planning and forecasting, especially when growth or architectural changes are nearing.
FinOps practitioners are from all over the world belonging to different countries. The practice of FinOps is often recognized by some other similar names in different countries. 25% of all the practitioners surveyed recognize this practice as FinOps, whereas 16% of them recognize this as Cloud Financial Management. Some other similar names are also found in some regions such as Cloud Economics, Cost Optimization, etc.
Image Source: FinOps Foundation
FinOps practitioners belong to various roles in an organization and are often a combination of people from different teams in a company. Some of them belong to the executive and leadership roles whereas some are working under a supervisor or technical executive.
Image Source: FinOps Foundation
It was found out that around 31.38% of the practitioners report to the CTO of an organization. 26.82% report to the CIO and 12.41% report to the CFO of an organization. The rest of the people respond to other superiors such as COO or Head of Procurement etc.
32.75% of people belong to the FinOps team of an organization whereas around 43.56% of people belong to the Cloud Center of Excellence team, this team includes the FinOps team in it as a sub-entity.
Involvement of the engineering team and software developers is the final step that needs to be accomplished in establishing FinOps practices in an organization. It is very important to encourage and incentivize software developers and engineers to participate in Cloud Cost Management. Getting the engineering team on board is a known challenge as they are the ones who have to implement it from a technical perspective and perform the optimizations.
There are various Key Performance Indicators (KPI’s) to track the successful outcomes of FinOps in an organization, such as Utilization Metrics, Usage of Saving Instruments such as AWS reserved instances and dialing up requirement engineering.
FinOps team can face a wide variety of challenges due to its vast scope in an organization. The collaboration of various teams is needed in order to make the practice successful. Even though the challenges faced can be a lot, still, some challenges are more prominent and it was found that some specific challenges are faced by the majority of FinOps teams.
Image Source: FinOps Foundation
The challenges faced by Runners, Walkers and Crawlers might seem to be very different from each other due to their the difference in their scale however this is not the case, in reality, a lot of challenges faced by them are common. Challenges faced by an organization that is spending $5 million per year are often very similar to those faced by a company spending $500 million per year. Hence FinOps adoption at an early stage and following best practices and performing optimization at an early stage become very beneficial for an organization in the long run.
FinOps practitioners refer to various sources for information regarding the practices. FinOps Foundation is the leading resource center. Apart from this, there are various other good sources such as Last Week in AWS, AWS’s Blog, Google Cloud Blog, etc.
The need to provide training for FinOps practices is ever increasing. Currently, the imminent need for training in FinOps practices is seen for these roles:
With setting up FinOps practices in your organization, the practitioners need tools to implement these practices. Some of the famous tolling used for this purpose are AWS Cost Explorer, Azure Cost Management, Cloudability, CloudHealth etc.
The adoption of FinOps tools by FinOps practitioners varies between them being Crawlers, Walkers or Runners.
Although the FinOps tools are very useful in implementing these practices. Practitioners often rely on other methods as well for forecasting their cloud budgets such as data collection, collation and data analysis using spreadsheet software such as Microsoft Excel and Google Sheets.
The average estimate of current numbers of servers and servers required in the succeeding year by the organizations that follow FinOps practices are
AWS has been the clear leader in cloud infrastructure. The popularity of other platforms is on the rise as well, Azure is gaining popularity in the Cloud space and Google Cloud Platform is another major player in the space, while some organizations use on-premise private cloud completely or partially.
It was found that compared to the other stages, Crawlers were the most flexible providing a variable percentage of variance in their forecasting and Runners the least. These stats make sense and generally the Runners consist of large organizations and even a couple of percentage of variance in their cost is a really large amount due to the difference in scale of the requirement of cloud resources between Crawlers, Walkers, and Runners. Crawlers are content with allowing 20% variance from the forecasted amount whereas Walker wants the variance to be 15% and Runners want it to be to 12%.
Automation in the allocation of resources to a service and measuring these metrics to scale up and down servers seems like a viable method of optimizing the cloud costs, however, implements this to your entire cloud infrastructure is a humongous task as it needs to be done from both engineering and financial perspective as some resources are pay as you go, some need to have a financial estimation upfront and often has the requirement of multiple teams, data collection regarding your automation for future price forecasting, etc. Due to various such factors automating your entire cloud infrastructure becomes a very complicated task as it might create its own sub-tasks at multiple levels and for various teams. It was found that around 50% of the practitioners have little to no automation in their infrastructure changes. Stage wise split of no automation is 70% of Walkers, 7% of Crawlers and 29% of Runners. The practitioners who have automated certain sections inside their infrastructure have to automate various tasks such as report sending to teams, tagging hygiene, automating rightsizing, etc.
The adoption of FinOps can be complex and disruptive, especially for legacy enterprises which have only recently undergone digital transformation or cloud migration. Receiving professional assistance from a trusted consultant, able to implement the latest automated tools and best practices can smoothen the transition to a great extent.
The benefits of FinOps are evident and immediate: The visibility and control of operational costs and the ability to continuously evaluate them against actual resource utilization can greatly reduce cloud spend, enable proper forecast and free up budgets. It will enable your teams to operate in a fast paced environment, free of resource shortage, thus deliver business impact faster and in a cost-effective manner.
Business vector created by vectorjuice - www.freepik.com