Jeff McSchooler, EVP of Engineering and Operations at EchoStar, outlines his company’s progress with agentic AI so far, its step-by-step approach and immovable principles, to Contributing Editor Annie Turner.

Jeff McSchooler, Executive Vice President of Engineering and Operations at EchoStar, runs engineering and operations for EchoStar Technology, whose teams serve the whole company. This includes Boost Mobile, the Hughes brand for satellite governmental and residential services, plus DISH Technologies, which develops hardware and other tech for the TV delivery businesses, DISH Network and Sling.

He is on the keynote panel at FutureNet North America which will discuss agentic AI as the next frontier in telecoms. He’s well placed to comment, given his company’s progress with an initial 18 agentic AI use cases. “As a company, we decided the number one priority was to get all the data in a place where it’s good, clean, scrubbed so it’s usable by AI and the AI has a destination to put data pulled from everywhere,” he says.

Establishing the data foundation

Boost’s data architecture was a green field, built by McSchooler and his teams from the ground up. “We have this data platform where I can see every single piece of data across the entire network at once and that was intentional, to prepare us long term for correlation in AI and eventually agentic. We’re in a different world in that part of our company.”

Elsewhere, legacy tech can be challenging, as systems were not typically designed for use with AI. “People have to do a lot to some of [those systems] to get them to work and some will be easy…[until] you put the APIs in place and the calls go out, then it can turn into a different thing,” he explains. Some systems were designed to safeguard against network changes and cannot natively support network automation. “Now I’ve got to open them up and secure them in different ways,” McSchooler comments.

Data can be scraped from legacy systems, but this doesn’t always help. “If it’s dead the minute you pull it, that data is wrong and dangerous for AI to act on” he observes. “Whether it’s legacy or not, the biggest problem with data is its accuracy. I am not convinced anybody has yet solved this 100%.”

Even so, “We’re pretty far along with building the data foundation,” he says. “By the end of the year we will have 100% of our data from the company contained in a place where it can be observed via an API. That’s where we’ve been focusing most of our efforts.”

Picking the right PoCs

EchoStar is “pushing really hard” with its first 18 use cases for agentic AI, he states, They were chosen from hundreds of proofs of concept (PoCs) running across the entire company by a team set up for that purpose. The qualifying criteria for PoCs to progress were that they had to make or save money, and/or improve something that is important to the company.

McSchooler says valuable lesson were learned early from all those PoCs: “You need trials and tribulations to come up with great engineering and architecture. If you don’t learn those things, usually your architecture is not great because you didn’t think through what it’s going to be like for the customer or the operations team that has to manage it.”

EchoStar has already seen successes arising from the chosen PoCs. For example, its network operations centres are about 98% automated, run by around 35 staff instead of hundreds.

Coding PoCs led the company to choose 200 engineers who now only code using Anthropic’s large language model, Claude, and other available tools. “They code it that way, then QA [quality assure] it, monitor it, validate it and improve. They’re releasing tested code in 10 days; before it took six or seven months to write a new set of code,” according to McSchooler.

AI is also used to identify bugs in code. “It finds things you can’t believe humans miss,” he says.

Also, “We’ve been very carefully picking people to partner with on AI and cross pollinate outside our company on very, very selective [basis] so that we can hand data back and forth,” McSchooler explains. It ties its systems to that of one major customer, “so we can help on their systems with our data because our data is better. We give them data back so they can make better decisions in the call centre for their own customers,” he continues.

Governance centre stage

 “We’re moving deliberately, to make good decisions as we go, and not get ourselves in a place where we have to back up,” McSchooler stresses. In all agentic AI work, governance before scale is his watchword. “We have our own governance regulation team which we put in place as we started [work on agentic AI], and it includes regular audits of both prompts and system operations.”

“We’re constantly looking for drift or for odd behaviour where an agent is choosing to do something that it did originally but now it’s smarter and maybe linked, so it’s doing something else,” he explains.

“You need hard guardrails and setting up a primary path should never mean dropping the redundant path to somehow solve the problem. Hard work is needed to put this in place so that it can’t be moved,” he elaborates.

The only viable approach

McSchooler believes the only viable approach is to have disparate, specialist agents for different purposes, just like human teams, and they need a range of skills and knowledge. With agents, the first might look for packet loss on a cell path. The next checks the data from the first and correlates it against the RAN, and if there’s a problem, it determines if it’s in transport or the core.

A third agent might issue instructions to reroute, rearrange or reconfigure a system, based on what is learned. The next checks SLAs, if alarms are cleared and whether customers are still complaining. The last overlooks the whole process, logs it and records exactly what each agents did. “This means you can go back and see what happened, because the day it goes wrong, you need a crumb path to find your way back through it,” McSchooler says.

Another danger highlights the importance of rigorous checkpoints: the same input could generate a secondary output from one piece of data, and choose a different solution. “With millions of elements and other agents doing stuff, you must have some sort of gate or checkpoint that says, ‘Yes, that works or ‘No, that doesn’t, back it up and do it again’,” McSchooler continues. “We use this in our CI/CD process where we’re bringing up new things; there’s about 34 checkpoints in an automated light up of a site.”

Outstanding issues: roll back and orchestration

Rolling back an unsuccessful change to an established stable position is essential, but so far remains something of a grey area in the industry. “The key question is whether to allow the AI to roll back its own actions, rely on an external rollback mechanism, or deploy a secondary agentic system,” he says. “I don’t know the answer yet but you’ve got to have versioning of every network change to know where you were before each change.”

EchoStar achieves this by running a “the golden parameter list”, which does not involve agentic AI, and has minimal human oversight. It relies on the Basic Input/Output System (BIOS) and allocation of memory, CPU and storage in the server. McSchooler explains, “We constantly pull every single setting in the entire system, so we have every parameter in our data. Then we scrape it and ask if it matches what it’s supposed to be. If it doesn’t match, it reverts into automatic mode, which we do on things with a small blast radius. The big stuff still requires human interaction.”

He stresses that with agentic AI, “If anywhere along that path, you didn’t check and there was one bad decision, everybody else will keep going which could create cascading errors. That’s one of the biggest problems with orchestration. I’m not sure how well all our teams’ orchestrations are going to work long term [with agentic AI]. You may have an orchestrator of orchestrators, and maybe sections of orchestration for sections of your business to encapsulate just a few steps.”

This leads to another conundrum. When agentic AI orchestration is part of overall control, “You could drive latency into the network and if you’re trying to troubleshoot a real-time, live system, that could delay your decision making,” he muses. “It would cause less delay than with humans, but more than you might want. You need an observation layer that lets you look at everything that was chosen by all AI components across everything… and that’s going to be interesting.”

McSchooler concludes, “I’m excited about where the agentic AI future goes but a lot of people have a lot to do. There’s so much possibility, but we have to manage it carefully, with checks and balances everywhere.”