TNS
VOXPOP
Do You Resent AI?
If you’re a developer, do you resent generative AI’s ability to write code?
Yes, because I spent a lot of time learning how to code.
0%
Yes, because I fear that employers will replace me and/or my peers with it.
0%
Yes, because too much investment is going to AI at the expense of other needs.
0%
No, because it makes too many programming mistakes.
0%
No, because it can’t replace what I do.
0%
No, because it is a tool that will help me be more productive.
0%
No, I am a highly evolved being and resent nothing.
0%
I don’t think much about AI.
0%
AI / Edge Computing / Hardware

Chipmakers Putting a Laser Focus on Edge AI

Arm, Intel, and NVIDIA in recent weeks have rolled out new CPUs, GPUs, and NPUs to fuel compute and app development at the edge.
Apr 12th, 2024 10:06am by
Featued image for: Chipmakers Putting a Laser Focus on Edge AI

AI workloads are rapidly flowing out to the network edge, accelerating the demand for even more compute power that established chipmakers and startups alike are rushing to meet.

Silicon vendors over the past month have rolled out new CPUs, accelerators, and platforms designed to address the power, analytical processes, and security demands in a compute environment that by its nature is constrained in many of those areas. That said, the overriding need is to bring all these capabilities as close as possible to where the massive amounts of data are being generated, so for chipmakers, the race is on.

The result will be a range of new silicon options for AI workloads at the edge as the year unfolds. The same day this week that Arm unveiled its Ethos-U85 neural processing unit (NPU) and Corstone-320 Internet of Things (IoT) reference design platform for edge AI applications at the Embedded World 2024 show, rival Intel showed off its Gaudi 3 accelerators and Xeon 6 CPUs for the AI edge at the company’s Intel Vision 2024 event, with CEO Pat Gelsinger calling AI “the next killer app for the edge.”

“As the edge becomes increasingly important, it’s going to become the dominant AI workload resource,” Gelsinger said. “Research indicates that by 2026, 50% of edge computing deployments will involve machine learning and AI, compared to just 5% today. A killer use case.”

For its part, Qualcomm at Embedded World announced its RB3 Gen 2 Platform, a hardware and software offering aimed at embedded and IoT uses, including AI edge boxes. All this comes less than a month after NVIDIA took the covers off its upcoming Blackwell family of GPUs, maintaining its pole position for AI workload accelerators.

Startups Are in the Mix

It’s not only the established players but also startups looking to get a foothold in the edge AI space. Israeli startup Hailo this month raised $120 million in Series C funding — bringing to more than $340 million that total amount it’s brought in — and launched the Hailo-10 accelerators, designed to run generative AI applications locally without the need for cloud-based services. Another startup, SiMa.ai, which makes Systems on a Chip (SoCs) of edge AI systems, this month pulled in $70 million in funding — including from Dell Technologies’ investment arm — growing the amount it’s collected to $270 million.

It will be a boon for chip makers, with Omdia researchers expecting the market for AI processors at the edge to grow from $31 billion in 2022 to $60 billion in 2028.

The tech industry for several years has promoted the marriage of the edge and AI, but that’s accelerated over the last 16 months since OpenAI launched ChatGPT. The explosion of adoption and innovation of generative AI is quickly making its way to the edge.

Gartner has predicted that by next year, 75% of data will be created and processed at the edge. Bringing AI to the edge will accelerate data processing. It will deliver robust computing capabilities out to where the various IoT and sensors are and remove the need to move data to the cloud for processing, which can take seconds. Processing it at the edge can reduce that to milliseconds, a crucial difference when thinking of use cases like autonomous vehicles.

It also removes the bandwidth costs that come from moving the data to the cloud, as well as the cost of processing it in the cloud. There are data security and compliance benefits: not having to send data elsewhere makes it less vulnerable to mistakes or attacks.

A Nod to Developers

Not all of the focus by Intel, Arm, and NVIDIA was on the compute capacity, performance, and power efficiency of their chips. The companies’ executives also talked about meeting the needs of developers creating the applications and algorithms that will run on these AI-enabled edge devices.

The trend in AI and at the edge is toward more open and industry-standard approaches, according to Intel and Arm, with Paul Williamson, senior vice president and general manager of the IoT line of business at Arm, pointing to the PyTorch Foundation’s development of the ExecuTorch runtime for AI edge devices as an example. It will be important for the infrastructure and components at the edge to support the tools that developers use now as they build AI applications for the edge.

“The new Ethos U85 builds on previous generations and it offers that same consistent toolchain so partners can leverage their investment in Arm-based [machine learning] for a seamless developer experience,” Williamson told journalists and analysts at a press briefing.

He added that by using a chip design optimized for transformer models, “developers can enable new possibilities for AI at the edge applications that require faster inference, optimized models for better performance, and … improved scalability.”

During his keynote, Gelsinger several times pointed to Intel’s embrace of open systems as a competitive advantage over rival NVIDIA.

“Nearly all of the GenAI deployed developments today are moving to higher level environments, PyTorch frameworks and other community models from Hugging Face,” he said. “The industry is quickly moving away from proprietary CUDA models. Literally, a few lines of code and you’re able to be up and running with industry-standard frameworks on power-performant, efficient cloud infrastructure.”

During his own keynote at NVIDIA GTC, NVIDIA co-founder and CEO Jensen Huang said the company is developing its own AI-focused software development cycle that includes such generative AI features as chatbots and copilots and wants to expand the programming languages that its CUDA programming supports beyond C++, Fortran, and PyTorch.

Edge AI Use Cases Are Growing

Such capabilities will be important as the use cases for edge AI expand beyond myriad early examples, including image analysis, home security systems, secure banking, recommendation engines, and logistics. “All these require software,” Bob O’Donnell, principal analyst with TECHnalysis Research, told The New Stack. “That requires developers to create them. “The edge is fertile ground for apps.”

“Edge computing will play a pivotal role in the deployment of AI applications,” Dave McCarthy, research vice president of cloud and edge services at IDC, said last month when the research firm predicted worldwide spending on edge computing will jump to $232 billion this year, a 15%t increase over 2023. “To meet scalability and performance requirements, organizations will need to adopt the distributed approach to architecture that edge computing provides. OEMs, ISVs, and service providers are taking advantage of this market opportunity by extending feature sets to enable AI in edge locations.”

Group Created with Sketch.
TNS DAILY NEWSLETTER Receive a free roundup of the most recent TNS articles in your inbox each day.