This transcript was generated automatically by AI. If you find any mistakes, please email us.
Hello everyone, you're listening to Cloud for Cloud Innovation and Leaders' Insight brought to you by Global Dots
Ganesh (00:13)
Platform teams often operate behind the scenes, yet their choices can make or break how fast a company builds, ships and scales. Our guest today is Paweł Brodzki, former head of developer platform at Forte and now AI adoption lead.
What sets him apart more than anything is how he thinks about internal relationships, or as he calls them, internal customers. By treating engineers as customers, Pavel and his team helped turn the failed Kubernetes migration into a success, while reshaping the developer experience and team dynamics along the way.
I'm Ganesh the awesome Solutions Architect at GlobalDots where we research innovations every day so you don't have to. As always, we invite you to join the conversation on LinkedIn. Let us know your thoughts. Pavel, hello. Welcome to our show. Can you describe what it means to be Fortis developer platform, please? What are you responsible for day to day?
Pavel (01:10)
Sure. Thanks for having me. Great to be here. ⁓ First, probably a couple of words about Fortr because it'll make more sense then to talk about the platform and the scale we're operating in. So, Fortr, we've built a huge network of identities, over 1.5 billion in total. And we use this network to help huge enterprises, mostly in the e-commerce world, make trust-based decisions.
throughout the customer's journey, be it from the signup, login, obviously transactions and purchases, and even chargebacks later down the line in the process. we help these companies make those decisions throughout the entire process. And so we operate at very low latencies. We operate at very high throughputs. We're talking about thousands of decisions per second sometimes.
And we need vast quantities of data. So our secret sauce is in our data, in our pipelines, in our real time decisioning systems. so the engineering is Fortir is a very strong organization that leads a lot of the ⁓ kind of business outcomes for the company. So all of this to say is that Fortir really values its engineers, its R &D, obviously analysts as well, data scientists.
As the platform group or the developer platform subset of the larger platform engineering organization at Fortir, our job is to make sure that their jobs are as easy as they could be. So if we're talking about the metrics, the KPIs that we would use to measure how well we're doing, we would be talking about DORA metrics, so lead time, ⁓ the deployment frequency, MTTR, MTDD. ⁓
All of those are metrics that we deeply care about. Nowadays, we're also adding AI-centric metrics as well to measure how well we're adopting and using those tools. Yeah, but that's kind of the gist of what we do in the platform. And obviously, we can get into details on any front there.
Ganesh (03:30)
It would be very interesting to know because you mentioned 1.5 billion entities or like data points that you need to work very large.
Pavel (03:40)
Many, more data points. Over 1.5 billion identities, unique identities.
Ganesh (03:46)
unique identities. And just because it's a technical audience, and we don't have to go super deep on the tech stack, but what are some of your key components in there that allow you to do that?
Pavel (03:57)
Well, we're gathering huge quantities of information. So a couple orders of magnitude more than the number of identities ⁓ on each transaction that a user has on a website. if you go to one of our vendors that we work with and you want to purchase a of sneakers, we're getting a lot of information throughout this time. So we know quite a bit about you and we use this information to then link you to your
other purchases on different websites that are part of our network. And so we have this virtuous cycle where the more we know about you from one vendor, the more we can use it, this information, ⁓ when you ⁓ have interactions with other merchants. And so we build this really, really unique and sort of what we call those identities. And some of it is like
things that you would imagine like your IP address and your physical address and your details. And obviously a lot of the secrets or stuff that I can divulge here. But basically, even if you change some parts as fraudsters will try to do, if let's say they get your credit card information, we can still help the ⁓ vendor catch, rather not approve this transaction.
because we know that it doesn't follow the pattern by which we know that you are you.
Ganesh (05:30)
Sounds like a magic fingerprint if I had to give it a name.
Pavel (05:34)
Exactly.
Ganesh (05:35)
No, you can use that if you like. I don't mind if you want to put that in the product, that one's free to use. In the preamble and in our sort of pre-conversations, you mentioned a term, internal customers. What does it actually mean to you in practice? Like why are you calling them customers and not just colleagues or teammates? And can they turn down your ideas?
Pavel (05:58)
Sure. So they are customers because we behave with them as any product facing or user facing product person would with external customers, which obviously we have as well. And the idea is that we want to work with them. We don't want to just dictate blindly in the organization. This is how you should do CID. This is you should do CD. Even though we make some decisions, especially ones that have to do with compliance, security, safety.
There are some things that are non-negotiable, but within the parameters of the products that we give our users, our customers, we want to have as much feedback loops and as much conversation as we can get there. So for example, when we want to migrate from, let's say Jenkins to GitHub Actions, which is something that we're doing, we want to bring in the customers to understand what are your use cases.
We don't want to build a pure platform that would be amazing in a greenfield company, but will not be relevant to our actual users. So we want to know what they need and cater to them because at the end of the day, we are measured by how well they are doing. And so if we're doing just pure platform work that doesn't take into account the actual use cases that our teams in engineering and R &D have, we're not doing a job.
We're not doing a job well, at least. So I definitely see them as customers. I am working for them. I am providing a service to them. So let's say in our company, they need CI, they need CD. ⁓ What I provide should not be just, it's an excellent CI CD platform. It should be, it's an excellent CI CD platform for the customers, for the developers in Forter.
Ganesh (07:52)
Makes perfect sense. yeah, providing a CI-CD platform that's totally excellent, but not fit for your customers. You're not going to go down too well. You're talking about those feedback loops. Be interesting to know how you do that ⁓ in an organized way. mean, if I'm a product company, can obviously vote, you can upvote features that you want to appear in there.
internal customers, I guess they have louder voices and depending on their level in the company probably makes their request a bit more important than another person's request. But what are some of the costs of missing those signals in your organization?
Pavel (08:37)
Well, first to the point of whether they're louder or not as loud, consider that usually customers pay. Customers are paying customers. so their voice carries as much weight as their wallet does, right? Internally, customers don't really pay us. I don't go to a product team and say, like, hey, if you give me a thousand dollars, I'll create a tool for you. So you need to find other signals to understand how important something is to them.
And so we do several things. Obviously, we do surveys. So every six months, we send out a survey trying to understand some closed question and more open-ended questions. So closed question is we give a list of, let's say, 15 to 20 scenarios. So I'm debugging a problem in my code. And we ask the developers how easy it is for them to do so. So how easy is it for you to debug?
an issue in your production environment, how easy it is for you to launch a new service, how easy it is for you to get a code review and so on. So we have a list of those and we try not to tie them to specific technology or specific parts of the stack because again, if we want to go from Jenkins to GitHub Actions, we don't want to change the questions because the longer we run the same questions in the survey, the better our
longitudinal data is and the better we can understand the trends and how well we're doing over time and see improvements there, hopefully improvements. Besides surveys, in the past, we've done something called a cab, a customer advisory board. So we gathered, I think it was something like 15, 20 leading engineers from the whole of the organization. And we did a couple of multi-hour workshops where we did
whole facilitation with the whiteboard and stickies. And we try to understand what the most important ⁓ pain points were for the organization. And so it was a couple of years ago. And I think back then the number one issue was ⁓ scheduled jobs, crons, as we call them, which also were running on the different Jenkins. And so in the past 12 months, probably we've migrated them to Argo workflows running on Kubernetes now. so
⁓ This is now nearing its end ⁓ and we probably wouldn't have started this if we haven't asked the customers and try to understand what was the most important thing for them. But at the end of the day, some of the best signal that we can get is just embedding with the team for a couple of weeks and seeing their problems firsthand. So sitting with the team in their room, seeing how they operate, what they're doing and not doing, listening to them as they're struggling with
certain things that may already have been solved by the way. And they just haven't heard about it because it doesn't matter how many times you send it on Slack, may miss some people. And so the point is we see exactly what they're doing and they're like struggling with something we can intervene immediately. Obviously, oftentimes the issue would not be solved, but we would be able to see how impactful it is. Cause if it's something that they use once a month,
No big deal. If it's something that they use 10 times a day, we obviously want to address it more urgently. And so this is something that we've been doing for a while and we'll, we're ramping up even more so, and making sure that every quarter we have at least two, three teams with which we embed for a week or two and try to get kind of ethnographies that way.
Ganesh (12:23)
interesting to know what percentage of your team members time is spent embedded with other teams versus doing whatever else their other jobs are. Because in my experience from living and breathing this world in the past, everyone's always busy. Everything's always urgent. There's no time to make a cup of tea most days because everything is like a burning fire.
then to embed in a team for like a week and watch how someone's working seems like a very luxurious spend of time. But how do you equate that to normal working time? And again, there's obvious benefits there, but how do you weigh that up in your mind?
Pavel (13:06)
Yeah, great question. Cause it is a struggle. Everyone's busy and us as well. One of the strategies that work for us is to try to kill the birds with one stone and do it as part of another thing that we want to do with this team. Most usually it would be a rollout or a migration. So you alluded earlier that we're doing, migrating people to Kubernetes. And I also mentioned going from Jenkins to GitHub actions ⁓ and migrating from
and other Jenkins to Argo workflows. So we know that a team is bound to be migrated to one of those systems. We sit with them, we lead the migration, we try to do 80 % of the work for them, because we find that this is one of the best ways that we have to push migrations forward and not rely on kind of the kindness of the teams. So we try to withdraw those with migrations, do the vast majority of the work ourselves.
then use the opportunity, okay, we're doing it anyway. We still, we will need their input because okay, we get them to the final stage, let's say of the migration, but they still need to approve the fact that, okay, it's running in Kubernetes and the metrics look okay, and there are no error messages and so on. So we still need their approval. And while we're doing that, we're embedding and we're sitting there and we're listening to their feedback and trying to understand how they're working.
That's a strategy that seems to be working pretty well for us.
Ganesh (14:37)
Nice. You mentioned as well a couple of times Kubernetes and I candidly know that the migration didn't work out for you so well the first time around. What happened? What lessons can you give? know this is the kind of the real gold that the podcast listeners will be into. So what can you share from that?
Pavel (14:59)
Yeah, so it was a while ago, the first iteration. There were a few…
There setbacks, there a few issues that we encountered. Number one was unrealistic deadlines. We just didn't understand the severity, the magnitude of the change and that's on us. Obviously we did some research, but at the end of the day, we didn't have people with prior ⁓ meaningful knowledge of Kubernetes. So we were all learning on the job and the Dunning-Kruger effect was strong there. So we're like, okay, we know a little bit of Kubernetes, sounds easy enough, we got it.
And we didn't have it for sure. So we said like, yeah, in a year's time we'll have everyone running Kubernetes. And it was all tied into a larger issue, which was probably the root cause. were trying to go ⁓ multi-cloud back there. ⁓ We're running on AWS right now and we were trying to ⁓ work with Azure as well. And we said, okay, it'll be easier to migrate to Azure if all of our compute wasn't Kubernetes. Cause obviously it's like.
a layer that will make it easier to migrate. And ⁓ so we tied this project to that project and that was a huge mistake trying to combine the two. And on top of that, we also said, okay, might as well go with a service mesh, right? So let's bring in Istio as well. And that's a giant can of worms that we did not enjoy at all. We never got…
is there to work the way we wanted it with federated clusters and that just kind of blew up in our face. And so it was a little bit of just a hubris on our part trying to do it faster than it was reasonable. It was tying it to other projects with competing priorities and competing needs. And so doing multiple migrations at the same time is just the recipe for disaster. And it was. ⁓ We learned quite a bit from it, but
We scrapped the entire thing. And when we started again, a couple of years back, we basically started from scratch. ⁓ But we ⁓ reintroduced one of the issues from last time because we again wanted to do service mesh. This time we went with console connect and this one we got working, but not working well enough. And we ended up scrapping this part as well.
kept Kubernetes, ⁓ but we have a simpler networking setup right now. So this lesson we didn't learn well enough last time, but hopefully the others we've
Ganesh (17:48)
Well, there's ancient Hindi, Sanskrit proverbs of being human. And one of them is lessons will be repeated until learned. So you get there eventually. While you were talking, I also remembered a colleague of mine talking about Istio. He said, unless you're a huge enterprise, just say no to Istio. And that was like a little joke in the background. What did you end up with then as your service mesh?
Because you said you had two failed. What ended up working?
Pavel (18:22)
not having a service mesh.
Ganesh (18:23)
Just no having a service mesh, okay?
Pavel (18:25)
We're just relying on basically security groups and network rules within the cluster.
Ganesh (18:32)
Yeah, I think that's just a solid piece of advice right there. Like don't go building a service mesh. People are addicted to it. People are addicted to just using all of like, let's use the Google blueprint. It's like you're not Google. You don't have like 10 billion or whatever people every day coming onto your platform. It's totally…
Pavel (18:53)
We
may want to get back to it eventually. We do want the mutual TLS, right? MTLS. We do want the nice diagrams that it can paint you if everyone is running on the service mesh. But doing both together at the same time, definitely a bad idea. So start with one, make sure you're really, really comfortable with Kubernetes, which we're getting to that point now. And then you can start experimenting more.
Ganesh (19:19)
Yeah, I would agree. think the, I see more people worried about service mesh before they've done observability. just think if you, you haven't done like, if you haven't done open telemetry, you don't even dream about doing a service mesh because you don't even know what's going on in there. ⁓
Pavel (19:37)
Thankfully, that's one thing we did do from the get go. we do have ⁓ OTAIL installed on our own of our clusters and developers using Kubernetes get quite a bit of metrics, observability and logs out of the box in our stack.
Ganesh (19:55)
Awesome. like going back to this transition, what does it look like on the developer experience? What does it look like for the strength of the team or the dynamics and the performance? What's like kind of the, now that you're in the promised land, so to speak, what does that look like?
Pavel (20:14)
Well, the promise land is always an illusion, right? Cause there's always the next thing. Now it's, you know, Oh, you don't have a genetic workflows yet running in Kubernetes. What are you 2024? But, um, we, did, we were able to live up to some of the promises. So for example, we did see the average deploy time for some services go down, um, but well, not, not some, the average went down by something like, uh, 90%.
deploy times. So from 40 minutes to like 90 seconds, something like that. And that is something that is affecting users on a daily basis, right? So that this is something that they feel. And with costs, we also saw some significant improvements to the tune of I would say around 40 % reduction in compute for the same workloads. Just because you have auto scaling now and we're
Thinking further, obviously, we want to integrate Carpenter into the stack to get auto scaling for nodes and pods ⁓ and be better able to use spot instances in Kubernetes, which is not quite trivial out of the box. And I think one of the biggest things that we wanted to optimize for is ⁓ user developer ⁓ agency and autonomy.
So we want them to be able to say, Hey, here's something cool that I have. So for example, I have a Helm chart that I found under the cardboards. want to install it and we want them to allow to do it in a manner that would be safe, that wouldn't hurt other workloads, but that will allow them to experiment quickly. So to that end, we're actually only just this week or a week ago released the RGA version of ⁓
policy engine installed on all clusters. We use Kiverno, but there many other tools that you can use. And the idea is basically we block various cluster level resources. So you're not able to introduce those into the cluster, like cluster policies, cluster roles, without going through a review by security engineering or by the platform team. But other than that, within your namespace, do whatever you want. So you can write your own
Helm charts, you can write your own Kubernetes YAMLs and just go to town and experiment quickly. Because that's one of the things that we
that we are able to affect on the business level. So the faster developers, especially the closer they are to the product, ⁓ the faster they can release new features, the better it is for the business, right? The better for us to win the competition. so getting out of the way, not being a bottleneck is a crucial value for us, a crucial principle for us. one of the issues with, or rather one of the benefits of being on
Kubernetes is that there's the whole CNCF ecosystem. There's the whole ⁓ negative space that we weren't tapping into when we were on our custom bespoke ⁓ system that we had before with CloudFormation and some Python code on top of it. And every new engineer coming into Fortier had to learn this new system, whereas now it's ⁓ just plain YAML that you can learn online.
And by the way, if you're using co-pilot or whatever, it's very easy for it to integrate with it because it doesn't need to learn your whole code base and understand why you did certain things. You're just using plain Helm or YAML. So moving to industry standard tooling, moving to industry standard stack was a large part of our calculation with the new setup. I think this is paying dividends. So as an example, couple of quarters ago, one of our
data platform teams, wanted ⁓ to integrate a new streaming engine. We were using Storm, they wanted something a little bit more modern. And so they wanted to try out Flink in Fortir. And it was just, you know, a few days for us to launch a cluster for them. They installed with Helm whatever controller they wanted for Flink. They did some experiments and very quickly in a matter of a few days or maybe a couple of weeks.
They saw that it is working for them. And now it's ⁓ integrated in production. And a lot of the workloads has already moved to it. And we're moving other ones. The alternative before without Kubernetes, without this industry standard compute layer, would have been probably a month of work to integrate something like this.
Ganesh (25:08)
super cool. Yeah, and I had to laugh because when you said you dropped the release time from 40 minutes to 90 seconds, I'm sure everybody was like, you know, giving you high fives. I know the nature of humans and that within a couple of weeks, people would have been complaining about the 90 seconds waiting for you to get it down to 30 seconds or something.
Pavel (25:32)
You know people so well. That's absolutely what happened. Yeah.
Ganesh (25:36)
⁓ I knew it. And you touched on AI there. We can't have a conversation where we don't touch it a little bit just because it's contractually obligated. if we all want to get our next pay rise, we have to put AI in our titles and our jobs and roles somewhere. But how is Fordtru approaching this? Because you're ⁓ fairly leading, bleeding edge technology.
Pavel (25:45)
Contracts are obligated, yeah.
Ganesh (26:03)
How do you see that role, how that affecting your teams and your roles in the company?
Pavel (26:08)
There are many layers to it. The most basic one is on the human level. want basically everyone, not just in engineering or R &D using it, right? So everyone can get benefit from it. we have, know, Gemini for people that are working with Google Docs and emails and they're already ⁓ using it to improve communications that we send to customers and to answer RFPs and so on. But
in areas that are closer, ⁓ to, ⁓ to my domain of expertise, ⁓ obviously we're introducing copilot and all the other, ⁓ bells and whistles on the IDE level. So people don't want to use that. And we see something like 90 % adoption already. So very, very, very widespread and growing. ⁓ but besides that there's a whole stack and a whole set of processes and workflows that.
we're building now, for example, for approving new tools and new LLMs, right? Because new ones are being released on a daily basis. And you don't want to wait three months for approval if it's something that can genuinely affect how users work and behave. So we're creating new processes, sped up processes for compliance and legal and approval.
On the more technical side of things, we're now working out the architecture that will allow us to enable as many users as possible to tap into our data, for example. So that's not trivial, right? We want to build a layer on top of it with data contracts, probably utilizing MCP in some manner. So we already launched a little MCP server internally that taps into
those are the workflows that I mentioned into Asana into our observability stack. And it already allows you to ask questions in human readable way. Like, Hey, what are the alerts that are currently firing for my team? Then it tells you. then at least if it's on Kubernetes, you can go further and say like, Hey, find logs for this service from the last hour and tell me what you see there. And it goes and it reads them itself and it spits out.
⁓ answers that tend to be pretty decent and we're just getting started. So this is just early, early days and already it's something that is we see as being able to provide quite a bit of value. So we're to be investing heavily in this. the idea is not to improve existing workflows for engineers by 10, 15%. That's nice, but it's not Neil moving. We're talking about, ⁓ you know,
10 fold improvements where we can find them solving problems that we previously thought were either intractable or just impractical in terms of ROI. So we didn't want to invest a year in something, but we'll invest a week in something in anything basically, basically if it's, you know, meaningful enough, interesting enough. So this is something that we're exploring now and we're encouraging everyone in the company to come up with.
with ideas and last thing I mentioned is I want to give credit, but I don't remember where I saw it exactly. But someone was saying that in the early days of computing, the bottleneck was machine resources, basically, how much RAM you had, how much CPU you had. And so you had to write in assembly or C to be very, very efficient. Then with the cloud native revolution, the bottleneck became developer speed. How fast can you write?
great code because compute was cheap enough that it wasn't the bottleneck for the most part for most things. And now. LMS can speed out code way faster than you could ever write. And the new bottleneck is becoming just creativity, human imagination. Basically, if you, like, if you can think of problems to solve, the LMS are here and ready to help you solve them. They're not quite there exactly. And they make mistakes obviously, but
they're getting better at such a pace that I see no reason to think that they won't be able to overcome those difficulties in a year or two or three. And so we should all switch our mindsets to be less focused on, you know, what you can do to help me with this specific problem that I'm having and more towards, which other problems have I never even considered solving because they seem too difficult or not worth it.
Ganesh (30:55)
Very poetic in its description. And I particularly like it because ⁓ I was always a terrible programmer come DevOps engineer. I came from the technical world many years ago. One of the reasons I ended up exiting and becoming more of a sales engineer because I couldn't bear the thought of sitting there writing Terraform all day or something like that. ⁓
But now we enter the new era where it's just creativity. Maybe there's hope for me in a technical role yet. If all I need is ideas, I can definitely return ⁓ my big comeback. Yeah, really nice. Just a kind of a sobering thought, actually, that the bottlenecks will slowly become humans, which is also terrifying.
We always like to ask the DeLorean question to every guest that comes on and that would be if you could go back in time and give yourself one piece of professional advice, what would that be?
Pavel (32:02)
I think I kind of followed it to an extent, but basically take chances and get out of your comfort zone because this is where you grow the fastest and the most. So, ⁓ for example, if you're being offered a management role, even though you're quite enjoying just being a programmer, give it a shot because you'll probably learn more in a year than you would in
five years of being going from senior engineer to staff engineer. And if you are put in a position where you can present something either to a large audience or to leadership, to management, and your first instinct is to say like, don't know, I don't want to mess up. Think twice because again, just the preparation, just having to be ready for something like this will teach you.
so much more than any other any other situation. So just put yourself in a situation where you're at least a little bit uncomfortable and you'll learn very, quickly.
Ganesh (33:12)
Nice, very wise words and yeah, trial by fire or just going in at the deep end is definitely, ⁓ it's unfortunately how I learned everything because I was much more gung-ho, so it resonates highly with me.
Pavel (33:31)
⁓
No, no one wants it right like no one's enjoying the uncomfortable feeling of being outside of your comfort zone, but If you ask people to go back and see what actually pushed them to where they are. That's what they'll tell you listen my experience. Yeah
Ganesh (33:47)
I agree. If you're good at that and you manage to keep pulling the rabbit out of the hat, the problem is that people keep throwing that on top of you and then eventually you get the dreaded burnout or you have some sort of panic attack or mental instability. that piece of advice comes with some sort of, you know, know your limits as well. don't know. I was always terrible at knowing my
Pavel (34:11)
I agree, but just on that point, while I also conflate the two, they're completely different things, right? Like not saying no and saying right to the right things, very, very different things. So absolutely say no to the wrong things, just, ⁓ you know, no to stop. And when you're being presented with an opportunity, no, you recognize it as an opportunity and say yes to those. But obviously like it's difficult to know at the moment.
Ganesh (34:40)
I have to say it's a big sweeping generalization of IT people, but on the whole we're people pleasers. I think we just like to say yes, we don't want to let anyone down. We want to support the business or the customer or the whoever. I think it's somehow like built into the DNA of being a technical person that you want to say yes to things, is, yeah, that's a very tricky, very tricky thing to master.
and I'm 43 and I still haven't mastered it myself. So good luck following that piece of advice. Pavel, we've come to the end of the show. You've been totally wonderful. Really great having you on. Thanks for your insights and everything. Anything you'd like to say before we let you go.
Pavel (35:25)
No, thank you for having me. It was really fun. Appreciate it.
Ganesh (35:30)
Wonderful. Pleasure. We've been doing it for over 20 years, it's what we do, and if I don't say so myself, we do it pretty well. So have a word with the experts, don't be shy, and remember that conversations are always for free. I'm Ganesh Lee Awesome, and this episode was produced and edited by Toma Morvidson.
Sound Editing and Mix by Bren Russell. Stay tuned for more episodes.
