At least 30% of generative AI projects will be abandoned after proof of concept (PoC) by the end of 2025, according to a Gartner press release.¹ This underscores a major challenge in harnessing the full potential of generative AI initiatives relative to the investment and confidence most leaders see in the technology. How can organizations effectively scale generative AI PoCs and achieve the expected impact? In this episode, our experts discuss the future of generative AI in the telecom sector, the role of AI agents in network operations, and the critical need for ethical AI practices. Tune in to discover the criteria and frameworks used to identify generative AI PoCs worth investing in and how to take these concepts from initial ideas to production-level implementation. Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025, Gartner, July 29, 2024. Featured experts Eugenia Ramirez, Director of New Trends and Emerging Technologies, Telefónica Priya Mehra, Equity Partner and Director, Altman Solon Iman Bendarkawi, Associate Director of Corporate Strategy & Development, Kyndryl This episode references a Kyndryl and Altman Solon survey that focuses on the challenges customers face with generative AI PoCs. The report will be available on Perspectives on Progress in January 2025. Stay tuned for more details.
At least 30% of generative AI projects will be abandoned after proof of concept (PoC) by the end of 2025, according to a Gartner press release.¹ This underscores a major challenge in harnessing the full potential of generative AI initiatives relative to the investment and confidence most leaders see in the technology. How can organizations effectively scale generative AI PoCs and achieve the expected impact?
In this episode, our experts discuss the future of generative AI in the telecom sector, the role of AI agents in network operations, and the critical need for ethical AI practices. Tune in to discover the criteria and frameworks used to identify generative AI PoCs worth investing in and how to take these concepts from initial ideas to production-level implementation.
Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025, Gartner, July 29, 2024.
Featured experts
Eugenia Ramirez, Director of New Trends and Emerging Technologies, Telefónica
Priya Mehra, Equity Partner and Director, Altman Solon
Iman Bendarkawi, Associate Director of Corporate Strategy & Development, Kyndryl
This episode references a Kyndryl and Altman Solon survey that focuses on the challenges customers face with generative AI PoCs. The report will be available on Perspectives on Progress in January 2025. Stay tuned for more details.
Sarah B. Nelson 00:00
Hello and welcome to The Progress Report. I'm your host, Sarah B. Nelson, Chief Design Officer for Kyndryl Vital, and this is going to be a really fantastic conversation today. As everybody I'm sure listening to this knows, it is a hot topic in every part of society, and obviously in the tech sector in particular, but it might not be as hot as it used to be. According to Gartner, a Gartner press release, at least 30% of Gen AI projects will be abandoned after proof of concept by the end of 2025. So there's this kind of tension. A lot of organizations are really struggling on how they can effectively leap and then scale from proof of concept to realizing that potential. So that's our job today. We're going to explore this and see what makes a Gen AI proof of concept worth investing in, and how do we take those concepts from the initial idea all the way to production level implementation? So we've got a great panel today. Three super brilliant people. We've got Eugenia Ramirez, Priya Mehra and Iman Bendarkawi. Eugenia is the director of new trends and emerging technologies at Telefónica. Priya is an equity partner and director at Altman Solon, and Iman is associate director of corporate strategy and development, and she's leading the generative AI strategy here at Kyndryl. So let's just jump right in. I'm going to start with Iman and Priya. I know we've been talking that Kyndryl and Altman Solon have been collaborating on some research focusing on some of the challenges that customers are facing with Gen AI proof of concepts. Can you talk a little bit about that study?
Iman Bendarkawi 01:34
Kyndryl has partnered with Altman Solon to bring this end-to-end view from the market standpoint on "What's the state of scaling Gen AI across telcos?" And looking at it from a tech and business leadership perspective, we've been studying how companies are scaling, where they're at on average, and what are the factors really driving enablement of Gen AI. Some of which being around budget, resources, business and IT alignment, ROI, tech readiness and business direction. In 2024, we have found that most of these companies are embracing Gen AI. They're excited. We have business leadership that's becoming more bought in and are starting to create really high expectations. However, we're noticing the tension, as you mentioned, Sarah, on how technology leaders can really get that to enablement and where Gen AI can really go in the enterprise. From the study, we found that actual expectations for benefits are incredibly high, with some leaders expecting 10%+ improvements in some of the most common success measures. The first and most common one being cost savings, where close to 80% of leaders are measuring success through cost reduction. Most leaders who are achieving an ROI are citing spectacular results, with majority of folks achieving an ROI between five to 15%.
Priya Mehra 02:50
I'll echo some of what Iman talked about. The survey tracks about 100 respondents or telco executives. Interestingly, we are covering about 3,000 on instances of use cases which these 100 executives noted that they are experimenting, exploring or are in production. Out of this, pretty much 30% are in production. So that 30% number seems to be interesting that everybody's quoting. However, I wouldn't say the rest of the 70% are in failure yet. So what they are saying is 30% have made into production. What we are also testing with them is how well these business use cases, which are in production, are meeting their stated business objectives, and that's where the answer gets a little muddy. Many of these use cases, it might be still too early, or they have been pushed into production too fast, or the business value is not clear, or the cost of scaling them is too high. So the ROI, for instance, they are not meeting the ROI. So there are multiple points of failure where our executives and survey respondents are indicating that many of these use cases are yet not meeting the stated business objectives. So there are some interesting perspectives on business objectives, and why they are not meeting business objectives, that we should explore.
Sarah B. Nelson 04:31
You know what strikes me as you're talking, just in the world of innovation, or ideation generally, or pipelines, it's a funnel. You always start with a lot of concepts and a lot of ideas, and as you learn, which is exactly what you're saying, you learn across all kinds of things. It doesn't really have the application we think of or it's not as technically efficient to implement, or to your point, it's the prediction of whether or not it will even meet the business needs the and ROI. Obviously, AI, and it has been a lot around a long time and Gen AI has become more mainstream, but it's still nascent. It feels natural to me that there would be so much experimentation, and that's how innovation works. I want to bring an actual telco executive. Eugenia, let's hear your point of view. Where are you making strides? Where are your challenges in AI and Gen AI at Telefónica?
Eugenia Ramirez 05:21
One thing, to give some perspective on this for Telefónica, but I think it could apply to the entire sector. Telefónica has been transforming itself for the last 10 years to make the most out of artificial intelligence in general, right? So, now we have seen this acceleration through Gen AI, but artificial intelligence is not something new for telcos. Specifically for Telefónica, we have become a fully digitized and data focused company, and I think these digital transformation efforts put us in a better position to leverage AI. So, that is a first consideration, right? And I think that, as with previous digital transformations, there are a lot of enablers that you have to consider in order to fully benefit from generative AI. And then, I guess a second consideration is that not all cases are equal. Our customer platform, actually our first cluster of use cases, we are already seeing impacts there. I mean, Gen AI is just potentially accelerating that impact, but we have been transforming those platforms a long time. Now we're starting to also focus on use cases related to the network and our systems using generative AI. In terms of our observability domain, we're starting to explore specific use cases around generative AI that could at least look promising.
Sarah B. Nelson 06:48
So, when we talk about scaling proof of concepts, what exactly do we mean?
Eugenia Ramirez 06:54
I think on our case, first of all, we work very closely with our operating units, because they are the ones close to our customers, of course, and close to both what they are expecting from us and the problems that we want to solve. So that's important. And I guess the first criteria for selecting those is what the impact is versus feasibility that each of these use cases could have. But later on, even to prioritize at a group level, what we're looking at is not only impact and feasibility, of course that would be the first consideration, but then also scalability of those in the sense that those are transversal to our different opcos, and the use case could be easily approachable and usable by other opcos. And then we do not stop there. We try to assess what the requirements for making these use cases a success are. We know the relevance of having the right data. We know the relevance of having the required skills and of having clear KPIs for each of these use cases.
Sarah B. Nelson 08:07
Can you give me an example of some of the kinds of KPIs that you might actually target?
Eugenia Ramirez 08:13
We have talked about the customer platforms, right? And for that, it would be a combination of both productivity of our agents, so average handling time, as well as, of course, customer satisfaction. But when we're talking about the network, it's around the degree of network autonomy and how much we spend regarding troubleshooting and how autonomous that could eventually become. Of course, if we're talking about some other more internal processes, the KPI would be different, such as time spent by our executives preparing certain documentation or even employee productivity. So, the use case will have specific KPIs associated to it, but I think it's fundamental to have them define very early on the process to make sure there's alignment on what these tools and solutions will bring us as benefit.
Sarah B. Nelson 08:13
So then Iman and Priya, I'm curious, what other kinds of patterns are you seeing across the telco industry in terms of the kinds of use cases people are investing in?
Priya Mehra 09:06
So, to the point of taking this POC, and when we are saying scaling right, it's basically what enables to get it into production. The data aspect has to address broadly a couple of things. It has to address, "Do I have the good data, data infrastructure, and data asset?" But it also has to address areas of privacy, security, compliance, and risk. So those are just the foundational capabilities. But then for the business, it's the top strategic things. Does it create value? Is there leadership alignment? Is the business and the technology leadership aligned, right? So within value creation, there are, I'll call them utility use cases. Which is, "Can you save me time? And can you save me cost?" Then there are the value use cases. The value use cases are stuff which will help you do recommendation, anomaly detection, root cause analysis. So the data and making sense of that data is a lot of effort and time. So one of the telco executives, for instance, who has implemented a very interesting generative AI use case in network operations, essentially had exactly the scenario where there were three or four different networks that they have acquired over time. They're using generative AI to essentially manage network configurations over these multiple networks, and it's enabling them to manage operations much better.
Sarah B. Nelson 10:50
Eugenia, what does a team look like that's working on these initiatives?
Eugenia Ramirez 10:54
I guess that question has two angles to it. First of all, we believe that in order to really prioritize and understand the benefit and the requirements of any of these use cases, you need an interdisciplinary team. I mean, it's not only the technical part, but we definitely think that the sponsor of these initiatives are the business people many times. And then you also need finance and human resources in that team to fully understand the entire transformation that needs to go together with the use case. So that's our first consideration. And then what we have seen is that specific technical skills are not very different to what we have seen before with big data, and as I said, with previous digital transformations, maybe some specific profiles are emerging. But we also believe that this should be very flexible. It should be flexible teams that are working on different use cases, as I say, hand-in-hand together with the people that really know either the business or the process that you are transforming.
Priya Mehra 12:08
I'll jump in here just looking at a broader landscape of telcos of various sizes. The bigger telcos are finding success in having a model which is a much more federated model, if I might call it. So, like a center of excellence team. And then the businesses or the various domains or functions are having their own data scientists or analysts who drive the initiatives. The responsibility of the business teams is really ensuring that the use cases are being adopted.
Iman Bendarkawi 12:46
Just to add on team. I think one thing that we found from our study, and also with conversations with Eugenia and other telco leaders, is that responsible AI in some capacity as part of the team or ethical AI is incredibly important. I think the former CTO of OpenAI had said the next version of GPT is going to be as smart as a PhD. So understanding how much of a model knows about a user, I think, will be incredibly important. And so responsible AI or ethical AI experts will be incredibly useful and important to the team on, "How do we make sure we're not overstepping or creating bias?"
Eugenia Ramirez 13:30
Just jumping on that, because it's something critical. I would say, a prerequisite for Telefónica, when we look at this and the guidelines we would give towards generative AI, of course, the first one is a recommitment to using AI in a responsible way, and that means it has a lot of implications. It's not only about security and privacy, which are the core of it, but also about fairness and non-bias and even sustainability, right? I mean, we know what it takes to run AI. We should do it in a sustainable way. So we have actually published our principles regarding responsible AI, and we have a commitment as an industry and as a sector, but specifically as Telefónica with all of our customers and employees, of course.
Sarah B. Nelson 14:25
One thing I'm wondering as we've talked a lot about operational efficiencies and cost savings and other kinds of efficiencies, I imagine a lot of the Gen AI agents that we're talking about would be invisible to an end user. But I am curious about you engaging humans in the loop. Maybe this is one for you, Iman.
Iman Bendarkawi 14:47
I'll give an example of something that Kyndryl has been tapped in to do as more of a transformation partner, and we've been asked by a large Fortune 50 telco to reimagine the in-store experience. So again, customers service, but to implement Gen AI as a real-time agent for store associates. So we're seeing store associates come in, especially during COVID, not getting the proper training and not having it be a well-established employee experience for the average store associate. I think we can all kind of relate to our average experiences as the customer when we go into an in-store telecom retail store. So we're partnering with this telco to try to understand, "What is it that the in-store associate feels is a gap? How do we understand what the common pitfalls are? How do we understand what the next best steps or actions are from their perspective?" And some of that can even be things like language or how much professional education they had before they came into the role. So the Gen AI piece is coming up with this recommendation engine to support what the next best step is. But at the same time, there is a lot of back-end work to analyze all of these experiences and get to common trends. And so one thing that we're seeing emerging is things around localization and language. Can a store associates speak Spanish or speak Russian or speak a different language? And so that has now become a new problem that we're solving, because we're seeing it from their point of view. It's not replacing anything. In fact, it's adding on to the role, and honestly, making the employee feel like they can better do their job. So there's a lot of work that the Kyndryl side is doing, not only on the direction where this goes and how to scale it, but on the security side, on the government side, and making sure that there are humans in the loop, protecting those experiences.
Sarah B. Nelson 16:36
It's interesting too, because you've got something that seems to have legs. How do you start looking at scaling?
Iman Bendarkawi 16:43
A lot of it is integration issues with existing systems. How do we integrate with existing systems without rehauling everything? And that's becoming very much a challenge in getting this to even, let's say, 20 stores.
Sarah B. Nelson 16:58
So I'm going to ask you all a big question. It feels a little weird in technology to put big time horizons on this, but I think it's useful. So when you think about 10 years from now, what is the vision for some kind of full-scale Gen AI? How might that look in the organization? This is a question for any of you.
Eugenia Ramirez 16:58
Going back to what you really need to scale. When we started looking at how generative AI is accelerating everything that's happening around artificial intelligence, we believe we have all the enabler elements ready. We mentioned data, we mentioned skills and interoperability of the solutions. We believe this is a key element as well, especially given that generative AI is a nascent technology. We believe we should be flexible and be ready to incorporate the best that is out there in the ecosystem, and then we can start thinking about really scaling these use cases in our case, globally. We have always looked at AI at least in two lenses, right? What AI can do for telco, but also what telco can do for AI, right? It's going to be much more business as usual, and we will see not only Gen AI, but AI agents in most of our processes, internal as well as external, as to what we can offer to our customers being B to C or B to B. But we also believe that telcos can have a bigger role to play within the AI ecosystem. Not only because connectivity is a big enablement of that, but because going forward, our network could have a decisive role in many of these use cases actually being deployed.
Priya Mehra 18:54
Gen AI, definitely, what you would want to see is that it's another AI technology in the arsenal of AI that if you, for example, look at business flows or workflows, there is a role to play in a workflow, both for what we call traditional AI and Gen AI. So that makes sense. The thing that needs to be ironed out about this technology is right now, you cannot just leave it and forget it. The autonomous nature of this technology, which is you completely let a workflow be automated on pure Gen AI, is still a question mark, and that's because you want to ensure model accuracy, performance, deterioration, and bias. The second thing which is also interesting is this technology, as of now, takes a lot of computational power. There is a pretty heavy cost and operational burden it can lead to, especially if some of the telcos, for example, are doing some of the custom development of the language models themselves. You cannot just take an open source or a closed source model and implement it off the shelf. There is going to be custom training. And as soon as you get into training, there is a question about computational resources power, and there is a lot of talk in the industry in terms of impact of generative AI on data center and power consumption, and how that plays out is also going to be interesting. And I'm sure there will be technology developments. For example, chip technology is improving, cooling technology is improving. All of those developments need to, in a way, go lock in step. So in 10 years, this thing can look a lot different than what we are seeing right now.
Iman Bendarkawi 20:49
I'll be the tech optimist in the room. So I think, like a famous pundit said, "Software is eating the world." I think Gen AI will be eating the world. And the only reason I think that this technology is slightly different than, let's say, cloud or your traditional AI, is because it's so user facing. I think most folks started adopting this because of the advent of chat GPT, and that's still being adopted. Some folks haven't even touched something like a copilot, but because it's so user facing, I see this being in more adoption. I see the role of a prompt engineer or an LLM fine tuner being almost like a suite role in 10 years, or being a software engineer, which is a very notoriously common role to go into now in tech. On the concepts of, how this affects leadership and orgs, I see Chief AI officers being a very common role in maybe the next five years, let alone 10 years. So I think there's a lot of kinks that with more technology resources who are inspired, maybe like myself, to go into the space, they might be figuring out some of those problems, and one of which is a really big one on sustainability that Priya harped on. I think 10 chat GPT prompts is actually equated to one usage of a water bottle in cooling a data center, which is a huge burden on sustainability. So I think the strong talent base that is really inspired. They'll go out and they'll learn how we solve and overcome these challenges.
Sarah B. Nelson 22:20
Excellent. It's a great way to end our fantastic discussion. Just want to thank you all for joining us and bringing your wisdom and insight to this really, really important topic. I hope that our listeners can really find some ways to think about how they can continue to look for new and different ways of really changing your businesses and finding new outlets. So again, thank you all for joining.
Eugenia Ramirez 22:47
Thank you, everybody.
Iman Bendarkawi 22:48
Thank you.
Priya Mehra 22:50
Thank you.
Sarah B. Nelson 22:52
Thank you for joining us on The Progress Report. If you enjoyed this content, please like, subscribe and share it with your friends and colleagues. You can find it on any of the podcast platforms.