Keith Townsend on Modernizing Legacy Infrastructure (Scaling Tech Podcast Ep16)

Mar 21, 2023 | Developers, Project Management

What do you do when you’re told, “No”? We have all dealt with legacy applications that we want to upgrade or replace. Sometimes upper management does not approve of your modernization plans, and when that happens, what should you do next? David and Arin speak with The CTO Advisor, Keith Townsend, on the latest in moving from data centers to the cloud, modernization of legacy infrastructure, the future of AI in infrastructure management, the talent gap, and how to make the case to management for an upgrade.

Keith Townsend is Founder and Principal at The CTO Advisor LLC, which he founded in 2016 based on his career as a technology management consultant with more than 20 years of related experience in designing, implementing, and managing data center technologies. His areas of expertise include virtualization, networking, and storage solutions for Fortune 500 organizations. He holds a BA in computing and an MS in information technology from DePaul University. He is an IT infrastructure subject matter expert praised for transforming IT operations in verticals that include Pharma, Software, Manufacturing, Government and Financial Services. You can learn more about his work at TheCTOAdvisor.com as well as on his site AWSEveryday.com, and through the prolific content he publishes on LinkedIn and YouTube.

If you are leading teams of engineers working in the cloud, and you probably are if you are listening to this podcast, then you won’t want to miss this episode!

Listen on Spotify
Listen on Apple Podcasts

Watch the video:
Show notes with links to jump ahead are below

Show Notes from Episode 16 – Keith Townsend on Modernizing Legacy Infrastructure
Timestamp links will open that part of the show in YouTube in a new window

  • 00:00 Opening quote from Keith about the importance of knowing the value that your role contributes to the company’s bottom line and how to increase revenue. If you can describe how your IT infrastructure project will increase revenue, then your project is more likely to be approved.
  • 01:30 David and Arin talk about their interview with Keith Townsend. Arin notes how we have all worked with legacy systems that we want to upgrade or rebuild, but management is not always on board. What do you do now? That’s one of the topics they covered with Keith. David noted how Keith talked about how the talent scarcity problem and automation will affect cloud management. Arin notes that it’s all about risk mitigation, and that’s what a lot of the conversation is really about.
  • 03:52 Arin introduces Keith Townsend as Founder and Principal at The CTO Advisor LLC, which he founded in 2016 based on his career as a technology management consultant with more than 20 years of related experience in designing, implementing, and managing data center technologies. His areas of expertise include virtualization, networking, and storage solutions for Fortune 500 organizations. He holds a BA in computing and an MS in information technology from DePaul University. He is an IT infrastructure subject matter expert praised for transforming IT operations in verticals that include Pharma, Software, Manufacturing, Government and Financial Services. You can learn more about his work at TheCTOAdvisor.com as well as on his site AWSEveryday.com, and through the prolific content he publishes on LinkedIn and YouTube.
  • 04:45 Arin asks how Keith became involved in CTO Consulting and with producing AWSEveryday. Keith started as a management consultant, and back in 2012 he had an opportunity to move from Lockheed Martin to work with PwC, the major accounting consulting firm. It was a transformative experience for him. Keith also talks about his AWSEveryday.com website and more broadly his cloud management post. This started with advice from Kelsey Hightower to do more work beyond just data center, since his reputation was initially being in that area only. Keith noted that AWS has 238 products as of this podcast, and most of us only work with about 10 of them, so he thought a quick blurb each day would allow him to familiarize himself with a new product each day. It has been a lot of work to post that much content, so eventually Jo Peterson also joined him to share her expertise around cloud security in particular. As of our recording, Keith has only covered about 45 of those AWS products so he expects to keep doing this and other cloud provider content for a while! Many of these services also deserve a deeper dive over time.
  • 09:48 David brings up an article from Basecamp, where they noted that in their circumstances, it does not make sense to stay in the cloud. They are going back to self hosted. Keith has seen the argument before, but does not agree that a cloud-first company should move to a data center based architecture. That is much harder than you think. If a company is already a hybrid company however, and you want to move some workloads back and forth from a data center to the public cloud, that is more achievable. Some applications like SAP are hard to run in the cloud. Likewise, other cloud-first services like Lambda would be very hard to move to a data center. David asks, does this mean that the skillset of running a data center is in decline? Keith say it is in decline, many seasoned IT professionals now with a 10-12 year career have never actually touched a physical server, and only know the cloud.
  • 13:25 David asks what are the scenarios which justify having a multi-cloud architecture? Where one part of the system is on a certain cloud provider, and part is on another provider. Keith says it’s very complicated to make a single application span multiple clouds. You can run different applications in your company on different clouds, but having a single application span multiple clouds is very complicated. What Keith sees that is more typical is to choose a cloud provider based on specific licensing concerns. For instance, Microsoft does not allow some client licenses to run in any cloud other than Azure. The main reason that Keith sees a company with a multi-cloud architecture is mergers and acquisitions. The acquiring company typically does not require their newly purchased subsidiary to switch their architecture to a different cloud.
  • 16:00 Arin mentions a podcast he was listening to which said that the future is in hybrid clouds. Their view is that some key intellectual property should be kept in your own data center, and use the cloud for less valuable aspects of your systems. Keith notes this is a major area of research in his clients. He talked about a client whose SAP system is so mission critical that replicating it in the public cloud was nearly impossible to get the resiliency they wanted, even ignoring the cost. If an application was designed to run in a data center, like SAP, then it may simply be cheaper and better to run it that way. But you still are left with the problem of innovating around it, by putting other applications in the public cloud. Innovation is often going to stay in the public cloud, even if that is not acknowledged. For instance, Keith talks about how many retailers don’t like to acknowledge they use AWS because they compete with Amazon in their business. But they typically do anyways.
  • 22:00 Arin asks if you base your architecture on a service like TensorFlow, where those workloads are run in the cloud, should you be comfortable with all of that risk being in a single provider? Keith notes there is always a caveat, and you still have to figure out how to use AWS such that it is available all the time. You also must deal with realities like data sovereignty regulations in Europe which requires that European client data stays in Europe. Requirements like this might require you to find a more specialized cloud provider who can deal with edge cases and zones in the regions you need to work in.
  • 24:50 David goes back to the human element of cloud infrastructure. David suggests that using bots and automation will alleviate the talent gap in running cloud infrastructure. It will help with things like monitoring, but we still don’t have a comprehensive solution to the problem of talent scarcity in cloud engineering. Keith notes that the best cloud engineers he knows are a “little seasoned” and they cut their teeth in the data center. It is a difficult skill to grow from nothing, if you don’t know the OSI model or you don’t know the pain of deploying dedup or disaster recovery for the first time. But those are skills that a fully rounded cloud engineer can use. He draws a parallel to dealing with memory management in the application level 30 or 40 years ago. That knowledge is now lost because it’s handled at the OS and platform level. Keith agrees that AI and automation will solve some of these problems going forward. But gaps will still remain. He points out how you can write a cloud formation template using ChatGPT now, but it’s only going to get you 80% of the way there or work in the more generic cases. The other 20% though may need deep expertise, so Keith is worried about the future and how people get that deep knowledge required for the last 20%. Where are those engineers going to come from?
  • 30:17 Arin asks about a blog post that Keith wrote on “What to do when you’re told No.” When senior management tells you that they are not interested in doing an IT infrastructure project that you’ve recommended, what do you do? Keith notes how hard it is to get funding for infrastructure projects – management just doesn’t see the immediate value in infrastructure until it stops working. Well engineered systems don’t go down very much, so asking for millions of dollars to upgrade or replatform a system can be a hard sell to the CEO. Keith compares it to forklifts in a warehouse. If the warehouse manager goes to executives and says that forklifts are breaking down and it’s impacting customer deliveries, that has obvious value. So IT leaders need to adopt that “forklift mentality” and describe your infrastructure needs to the customer needs. You have to tie your needs to customer value. You can tie the expense to how it will create overall savings, or increase revenue, or create new opportunities. You need to figure out how your function improves the business metrics overall – doing that will help you get more projects approved.
  • 35:30 When you’re told no, Keith notes that you are not absolved of responsibility. You still need to make sure that business process won’t fail due to technology. It won’t be good enough to say later that “you told me no.” Keith uses a story of a CFO saying that you need to do a migration without impacting the business – and if you can’t justify modernizing the platform without breaking the business, then it will be hard to get it approved. What Keith’s team did instead, when they were initially told no, was that they started to practice the modernization. For each new project, they followed the new process and that allowed them to better understand and practice the modernization that they would have to to do eventually on the full system. This better prepared them for the day when they eventually got management to approve the upgrade.
  • 38:50 Arin asks what if the modernization that was rejected included crucial security upgrades? Keith says that doesn’t really change the situation, and CTO’s have peers who have been through similar situations. You can reach out to peers in other companies, maybe even competitors, and ask them how they are handling a key upgrade that is being delayed. Talking to vendors can be especially valuable since that vendor has dealt with upgrading their systems across many clients like you, so ask them about how they handle these upgrades in more complicated situations. They will likely have mitigation strategies they can recommend, and you can at least start to document that plan and prepare for it. If nothing else, that provides you with some cover if you get blamed later. Better yet though, try to get to a lower level of the system upgrade that management will approve budget for, so you can at least get incremental improvement.
  • 42:30 Arin asks if there are scenarios where a team should go rogue and do an upgrade anyways? Keith says he’s become better at asking for forgiveness. Sometimes you do just fix something without approval because it’s necessary. This is sometimes called “Shadow IT”, and IT leaders should have some tools in their belt that they can use without business approval, and which will not get them fired. For example, if you notice a server hasn’t’ been accessed in months, then you might just push out a firewall change that shuts off access to that server and see if someone complains. If you see a serious liability in your company, it’s your job to know what knob you can turn to fix it. It may start with simple things like upgrading a lower level application.
  • 47:00 Arin asks what Keith sees across his clients – what are the biggest issues they need to modernize today, and what are their blindspots? The biggest challenge is usually people. For example, when people do a “lift and shift” from the data center to the cloud, this rarely works as easily as predicted. That is a human challenge really, because transformation is hard. The way pricing works in the cloud can be a blind spot, because you are probably paying for storage even of things you don’t actively use, in a way that you might not be used to in the datacenter. How do you modernize an architecture that now has 1000 apps? Keith sees the biggest challenge that the industry faces is how to constantly modernize our application landscape.
  • 50:10 David asks about reactive architectures and reactive systems, and does Keith see the industry going that way? Keith says there is a trend for that in customer facing systems. Keith talks about how a small company doesn’t need to have its own manufacturing anymore for example, and they can just put an RFP out for the widgets they need built and get those widgets back so they can focus on selling them. In the same way, Keith sees more reactive architectures that support digital transformations over time.
  • 52:20 Arin’s last question is about Low-code/No-code applications, and Keith says that is huge and the ultimate desire of any application. We only develop in languages because that is how we started. But we really just want smart people to be able to put a business process into an application and get the application to come out the other side. But what companies don’t realize is that even with No-code applications, you still face all the problems that developers face in traditional code. Versioning, deployments, all those things still exist even when building a No-code application.
  • 54:30 Keith tells listeners they can reach him on Twitter (see below), where his DM’s are open, as well as on LinkedIn, and at TheCTOAdvisor.com. That ends another excellent conversation, thank you Keith for joining us!

Links from Episode 16 – Keith Townsend on Modernizing Legacy Infrastructure