Speaker 1 (00:04):

Our division helps companies large and small modernize their analytics for their whole department. Some of it is technology-based, some of it is process and people-based. One I want to talk about today is an amalgamation of all the things I've learned, and services over the past 10 and 20 years, working with various companies and analytics leaders. These include multiple Fortune 500 companies. I have also done work and partnerships with some of the big four and five consulting firms. Redapt is based out of Seattle.

 

So I wanted to talk about Financial KPI Development in an Agile World. We've seen a lot of changes. I want to talk about this at a strategic level and address how we help our clients at a data maturity model. Typically that turns into some technology project, but the focus of this discussion is not necessarily on technology.

I want to start with an interesting factoid that leads into some of the points I’ll talk about later. Research shows that every time that the size of a city doubles, the innovation or productivity per resident increases by 15%. But, when companies get bigger, innovation or productivity per employee generally goes down. It seems an odd correlation that, when cities double, their output increases by 15%. Unfortunately that also includes crime, but innovation, productivity, and other aspects tend to be at around 15%. But, if we look at companies as they grow, these tend to decrease. I'm really interested in understanding why that is.

You might have heard of the marshmallow experiment. In this experiment, they created a team of different groups of demographics. They gave them some uncooked spaghetti, tape, and a marshmallow. They had to create the tallest tower and place the marshmallow on top. I believe they had a time limit of 30 minutes, or so. This was done with CEOs. It was done with lawyers, MBA students, and kindergartners. Which group do you think won?

The kindergartners whooped every group. They whooped the CEOs, the lawyers, and the MBA students. It's an interesting case study if you look at the interactions between the kindergartners and how adults tend to behave. Adults typically approach this project as a group—they immediately try to figure it out, plan, assess who's good at this and that, and then start doing the work. Kindergartners tend to go after the materials. They should start putting stuff together, trying things, and failing quickly. It comes down to the fact that the kindergartners didn't care. They kept trying. They started immediately. They failed immediately. 

 

The adults tended to plan and spend a lot of time trying to figure out the hierarchies of the group. “Who's better at this—who's better at that.” The key was, I think, the interaction. We’ll come back to this idea later.

 

At Redapt, as we work with companies of all different sizes, and as they move through their analytics journeys, we have this maturity matrix. There's technology and there are point solutions that do a lot of different things. Today it's AI, it's ML. That's the driving force and a lot of conversation and buzz is around it. But, I can tell you frankly that 90% of business can't even get the basics right.

I find that 90% of most business questions, most things you want to know, don't require AI or ML techniques. The value we present to businesses involves looking at where they are in their data maturity journeys. A lot of that includes helping managers and knowledge workers understand how to formulate business questions appropriately, how to experiment, how to do basic things like interpret results of AP tests and P scores. Doing this doesn't require a PhD in data science. On this matrix, most of the companies we work with will typically place a dot in the bottom left area. There are a lot of different aspects on this one, but we start with basic reporting. We have some dashboards and we do some really cool analytics. Then, we reach this predictive prescription mode where we can start to actually have an AI or ML model say, “Hey, you should probably refresh this product.” 

 

But, a lot of companies aren’t there yet. They haven’t mastered some of the basics. Look at the data maturity at the top. We go from spreadsheet, excel spreadsheet, and everybody doing their own thing. Then we move toward enterprise data warehouse models and managed models that provide insights on the right. Over on the scale, we move from individual to departmental, then start to bring in external data sources. For the culture, it’s how executives view analytics and BI. A lot of this starts with executives and how they view it. I think most executives realize that, to differentiate their companies, they need to invest here. They need to invest in technology, people, and processes, but it isn’t cheap. Data scientists are cheap—high performing analytics teams are not cheap.

 

At Redapt we worked to almost outsource that, because we do it all the time. Our augment teams are already there and we work across different technology stacks. Years ao, BI was different. “Hey, give me my reports, give me my spreadsheet, show me my graphs, and tell me where my numbers are.” It was seen as more of a call center. Companies are pushing this up more to a strategic resource, where all the work has been monetized and presented to their clients. 

 

For instance, I worked with a company called BigCommerce. They’re in the Shopify space.  If you're familiar with Shopify, they have thousands of e-commerce stores. We did a lot of internal work, and ultimately presented that work as value ads they could pay for. They wanted to add a purple skirt one day and the system would say, "This doesn't sell well this time of the year. You may want to rethink that." The ability to take all the internal work, move it out, and monetize it is important.

We help companies move up to this top right area, but there's a lot of work to be done there. I would break this down into three primary pillars we work with to get this done. This particular slide really highlights the three important pillars. 

 

I'm not going to talk a lot about the technology side. Most executives and most CIOs, CAOs, chief analytics officers, and knowledge executives probably agree that technology is not the biggest problem. There's a lot of technology that can do amazing things. The first are the biggest issues—Agile Culture and Organization. Our recommendation, and what we help companies mature in, is creating and rethinking how their teams are structured. 

 

When you look at the industrial revolution and how parent child hierarchy relationships were created, this is typical manager relationships. It doesn't work well in the analytics space. We hope companies embrace more modern hierarchy structures. This typically means cross-functional teams that tend to blur parent child relationships. One example of this is Zappos, the shoe company. They’re one of the bigger companies in the United States that uses more of a non-manager style hierarchy system.

 

It’s not the wild west and it’s not anarchy. We restructure the organization so it’s more agile. We use Agile, we use DevOps, we use scrum methodologies at the project level, but unfortunately companies don't do it at the higher structural levels. That's why organizations do these massive reorganizations. They get to a breaking point, but they do it so far between—they do it every year, or every two or three years. They didn't make micro changes in the organization up to that point. For analytics, we really look at, “How is this done at companies? Is there one department? Does it report to, is it cross-functional, does the financial group have the ability to influence? Is there a huge separation between tech and finance and is there friction there?”

We strive to create a cross-functional team that blurs those lines, because the most successful analytics teams don't live in a department. They're very ubiquitous, they'll move between departments, and they're able to respond to business challenges quickly. We treat tensions as fuel for innovation and positive change. In analytics, we fail often. We want to fail fast. So, we experiment constantly. Some of the most successful companies, such as Google and Netflix, are running thousands of experiments at any one time. If the historical data doesn't support a certain decision, we must run experiments against it. The experimental portfolio, software, and process to make that happen helps companies implement that. 

 

We treat tension and failure as a good thing. We say, "Hey, let's move on. Let's quickly figure out why that didn't work and let's find a solution." Tensions are the fuel for that change. We also change the analytical structure, because one year we might have three data scientists, a couple of data engineers, and different roles that are fine for a company. Two years later they’re in a different place and we need to quickly modify those roles and adapt. The financial industry needs the ability to adapt and change. COVID was a prime example of that. This includes data literacy training. One of the biggest pain points is that knowledge workers and managers don’t know how to structure questions or present data. They could do a lot of this themselves. I lay this at the feet of academia.

 

In most cases really brilliant data science comes out of universities. Brilliant MBA students and finance professionals come out. However, the merging of these two is very strange. There’s not much crossword and really smart finance professionals don’t know SQL. They can probably work in Excel. It doesn’t take a lot of training to elevate knowledge workers to where they need to be.

 

The second pillar has the modeling piece. Modeling with unstructured data, data lakes, and sort of the wild west, which—throw all your data on data lake—it sounds great. In reality, things need to be modeled correctly. Our business processes need to have the right model in place, so we can make good decisions. Agile Reality Modeling is hyper important. So gone are the days of taking 12 months to create an enterprise warehouse. We need results quickly and we need to do them with a scrum, agile methodology.

Buy where appropriate. There are industry model vendors who sell KPIs and models that are ready to go. Microsoft acquired one of the biggest and they are creating a product behind the scenes. What this will do is bring a lot of KPIs in finance, insurance, or whatever vertical you work in. There’s always that 20% or 30% that’s specific to your business and you want to differentiate. I would argue that modeling is the most important function of the analytics department. It sets up the data so the data scientists can do their work and knowledge workers can find answers. 

 

Those are the two I really wanted to talk about today. Cloud and a lot of different vendors are out there for visualization, AI, and ML, so I won't spend a lot of time there. I do want to mention there are other process areas at the bottom that we know are important. We can help companies take care of governance, operations, and data management strategies, but I won't touch on those today.

 

We see agile scrum methodologies in app development all the time, but we don't see them adapted to data ops, data, or AI too much. We help companies modernize here. We get enough designed up front. We say, "Hey, let's get what we need quickly and, let's do this in sprints. Instead of taking 12 months to create an enterprise data warehouse, let's do what we need to do to get the high priority questions answered.” We do our agile modeling quickly. We do our ETL development, or test-driven development prototyping, in sprint. So it's Scrum, it's Agile, but it's modernized towards what we call data ops. We help companies really get there. This is quite hard, because with data and analytics, there are so many different dependencies. There are many different things at play and it tends to be a difficult project to tackle.

 

We look at Scrum and Agile methodology. Some of this is product management, or how you manage your backlog. It is an example of something we would do, where we change features a little bit. Instead of epics, we call them analytics projects, business objectives, metrics, and value. These are what we use at the feature level. We help companies look at their backlogs, look at their Scrum or Agile methodologies and say, “How could we change this slightly so it works with analytics? Because it is not quite on target for analytics and data science work.” We do some modifications here, but the essence of the Scrum and Agile methodologies is still there. We also help companies when we ask business questions and leaders say, “Why did sales drop for this, a clear product?”

“Rob, why did loan volume decrease here? Why are financial services, why are these credit card processing events down for this particular month?” If the historicals aren't there, we have a tried and true process of quickly getting to the answers. We have what's called a Data Science Flow and a Decision Science Flow. We can use JIRA. We can use Azure DevOps. We can use whatever project management software the client wants, but it's about picking the right methodologies. I truly believe that 80 to 90% of all business questions can be answered with relatively simple math methodologies. It doesn't require AI and ML, but you need a solid process to get to it quickly and not spend a lot of time doing busy work.

 

Modeling is one of the most important things. Most businesses I ask, “Show me your dimensions and facts. Show me every discrete event and how all the dimensions intersect. You don't need to know what that is.” Over on the left is what we sometimes call the Bus Matrix in the analytics world. In this case, it's a high level view of every business event any leader would care about, as it intersects with the dimensions. Customer suppliers, what have you. That should be available to anybody, any knowledge worker. If you're at a company right now, and you can't look in an event matrix like this, and know how to analyze events against your dimensions? That's a problem.

We help companies create that legend and create a roadmap to get there. Each facet and dimension has stories we develop against. KPIs are important. They’re how we measure performance. We really strive to create a cohesive KPI matrix. 

For a simple use case of how we work, let’s look at Make-A-Wish. Make-A-Wish is a nonprofit that helps grant wishes to children. (We have permission to use this.)

 

We looked at the mission statement of the company and at the policy levels and asked, “What are the achievements?” A lot of this simplicity is quite deceptive. We’ve used Sigma. We use other performance enhancing methodologies, but they’re heavy handed and kind of going out of style. So, we use something that’s simple and that we can quickly turn around and create deliverables from. So, in the case of Make-A-Wish, we looked at, “What’s a top level organizational achievement?” In this case, they want to create happier children.

 

You can use this technique for anything, whether you’re weaving baskets, selling loans, or credit cards. It still applies. Ask, “What’s the achievement?” and don’t confuse it with behavior. When looking at a happier child, we go through workshops and identify three areas. We look at quality, quantity, and cost. Then, there are nine areas we ask the organization to come up with to help creating the KPI. So, we have accuracy class novel. Accuracy would be asking, “How accurate was that happier child? Did they get the wish they wanted? What class would be? What’s the quality and the novelty? Is it different from something else?” We go through a KPI process and visit all nine of these.

 

We come up with the most important KPIs and do some straightforward brainstorming. It might seem simple, but you’d be surprised at which KPIs are missing, which are wrong, or don’t cohesively line up with the mission statement. We typically find that KPIs at different departments don’t line up and they tend to create friction. We see people who are incentivized to do different things that don’t create a cohesive strategy for the business.

 

We want to create a KPI matrix that allows leaders to see everything rolled up, including all achievements and all KPIs against them. So, we start at the top level and then move down. Happier child would be broken down and a happier artifact created. This artifact could be that Brad Pitt said, "Hey, I'm going to have lunch with you.” It could be a trip to Disneyland. Whatever that artifact is, it's what they decided to go with. What does it take to create that artifact? Then we have to put that with an event. The event is, again, every achievement. We go through these nine areas and we create KPIs from them. Ultimately you get this beautiful cohesive KPI that makes sense and the business can agree on. This allows them to see the dark spots they currently want to see, but can't. Many times it exposes KPIs that are meaningless.

 

We helped one of the biggest shoe manufacturers in the world. They had a lot of issues in finance with understanding their total cost of ownership for cloud. They had Azure, they had AWS, they had Alibaba, GCP, and Rackspace. They used products like Aptio and Cloud Health, which are pretty popular, but they’re point solutions and somewhat limited. We had to fill in the gaps for them. We said, “Let's go ahead and enhance the GL.” We do this at a lot of companies. We start with the GL because it's a nice starting point. We remodel the GL. Many times companies are handcuffed with their accounting software, whether it's Hyperion, or financial management. There's a lot of customizability they don't have. Oracle or others can do that for them, but it requires millions of dollars and we can do it a lot cheaper.

Typically we use multiple products, or potentially one, based on wherever they're currently at. We can use GCP. We can use AWS, or Azure products to do this, but we typically enhance the GL. For this company, and specifically their finance department, to understand the total cost of ownership of all their workflows across all different cloud products, we had to really expand and enhance the GL. We remodeled the GL and we enhanced it to give them that ability. That's really the starting point for all of it. If you look at these, I call them hierarchy pillars, we have this concept of a total cost of ownership (TCO) object. One of their big questions was, “How much money do we spend on bot mitigation?”

 

A popular company can spend millions or tens of millions of dollars on bot mitigation. When the Xbox or PS4 came out, bots would buy them up. Those companies spent a significant amount of money mitigating and denying bots the ability to do that. They didn’t really know how much they spent. They amount of time it took for them to put together and analyze how much cloud resource spend was dedicated to bot mitigation was ridiculous. Analysts took weeks to even get to a number. With this new system and modeling technique, we can do value stream hierarchy pillar. You can look at costing. You can look at a lot of things from many different perspectives, but it requires the appropriate model.

Most point solutions don't have the customized ability to do this at this point. What will come in and do that for them? They still use CloudHealth. They still use FTO, for instance. But, we'll fill in the spots where they need help. 

As a quick example, here’s a case study involving United Federal Credit Union. They had a strategic initiative to help customer service. They had a lot of initiatives around customer service. Specifically, they wanted to analyze call volume. They had 40,000 calls per month using Cisco as their phone system and they really wanted to see how well they were performing. So, sentiment analysis. We used Azure Cognitive Services to do some quick sentiment. Now, we can use other cloud services, but their footprint was already within Azure. They did a lot of use cases.

We allowed sentiment change over the course of the call. We can pull up any agent and ask, “What is a KPI?” One of the KPIs was, “How can the agent move an angry customer to a happy customer?” Throughout the call, the sentiment would've been engaged. We have reports that could quickly see sentiment change over the call time period. To do this on 40,000 calls a month is ridiculous. A person could not do that manually. We used Azure Cognitive Services to quickly set this up. It analyzed the recordings and gave us sophisticated sentiment reporting around them. That was our customer service in the initiatives. Great success there.