The AI Inference Stack

As AI moves from model development to widespread production usage, inference is becoming its own infrastructure stack. Performance increasingly depends not only on access to GPUs, but on specialized chips, compilers, runtimes, serving systems, and routing layers that optimize latency, throughput, energy consumption, hardware utilization, and cost per request.

I broke this stack into four layers, moving from the hardware that runs the models, to the software that optimizes their execution, the platforms that deploy them into production, and the control layers that route and manage inference traffic.

Inference chips & systems

This category includes companies building specialized chips and complete systems designed to run AI models faster, more efficiently, and at a lower cost.

What are inference specific chips and systems, and why do we need them?

• Hardware designed specifically to run trained models. Most AI chips were initially built to support both model training and inference. Inference specific chips are optimized for the narrower task of running an already trained model repeatedly in production.

• A different performance problem from training. Training requires flexibility and large amounts of computing power over long periods. Inference is more focused on responding quickly to requests, processing as many requests as possible, and keeping the cost of each response under control.

• Better economics at scale. General purpose GPUs can run inference, but they are not always the most efficient option. Specialized systems aim to provide lower latency, higher throughput, better hardware utilization, and lower energy consumption.

• Memory is becoming as important as compute. Large models constantly move data between memory and processors. Companies such as d-Matrix and Fractile are designing architectures that bring memory and computation closer together, reducing one of the main bottlenecks in inference.

• The product is increasingly more than a chip. Customers do not only need a processor. They need complete systems that combine chips, memory, networking, racks, and software so that models can actually be deployed inside a data center.

What types of companies are building them?

• These are deeptech semiconductor companies. They need teams covering chip design, systems engineering, low-level software, hardware testing, manufacturing, and data center deployment.

• Some are building highly specialized processors. Etched is designing hardware optimized around a narrower set of frontier model workloads, betting that specialization can produce significantly better performance than more general purpose chips.

• Others focus on new memory and compute architectures. d-Matrix and Fractile are rethinking how model data moves through the system. Their products integrate memory and computation more closely to improve performance and energy efficiency.

• Some combine proprietary chips with their own inference cloud. Groq has built its LPU processor, but also operates GroqCloud so developers can use the hardware through an API without buying or deploying the systems themselves.

• Most are building complete systems rather than only selling chips. d-Matrix combines its processors with networking, software, and full data center racks, while Etched is developing the chips, racks, software, and manufacturing needed to deliver complete inference clusters.

• This makes the category extremely capital intensive. Developing a new chip requires large funding rounds, access to semiconductor manufacturing, long product cycles, and extensive testing before the product can reach customers. The companies therefore tend to raise much more capital and build larger technical teams earlier than the software companies in the rest of the landscape.

Category #2: Inference engines, compilers & runtimes

This category includes software that helps trained AI models run faster and more efficiently by optimizing how they use the available chips and infrastructure.

What problems do these companies solve?

• Once a model has been trained, running it efficiently in production is still very complex.

• Models need to work across different chips and infrastructure environments without being rebuilt for each one.

• Companies need to keep inference costs and latency under control, especially as usage scales.

• They also need to manage memory, batching, caching, sharding, and distributed execution.

• Basically, these companies sit between the model and the hardware. They make inference faster, cheaper, easier to scale, and less dependent on a specific chip or cloud provider.

Who are the customers?

• Cloud providers that want to improve the performance of their inference offering. DigitalOcean has worked with Inferact and vLLM to improve throughput and reduce inference costs for customers such as Character.ai. These providers use the technology to make their existing GPU infrastructure more competitive.

• AI model labs running large volumes of inference. These companies care about small improvements in throughput, latency, and hardware utilization because inference represents a significant part of their infrastructure costs. Gimlet Labs, for example, says it works with a frontier model lab and a hyperscaler.

• AI-native software companies whose product depends on real-time model performance. Coding assistants, voice agents, and other interactive AI products need very low latency and predictable performance. They may use these tools directly or access them through the cloud platforms that integrate them.

• Large enterprises operating models across different hardware environments. They use compiler and runtime infrastructure to avoid being locked into a single chip provider and to run the same models across NVIDIA, AMD, or their own infrastructure.

• Chip manufacturers and alternative hardware providers. These companies need a strong software layer to make their accelerators usable by developers. Modular’s partnership with AMD and Gimlet Labs’ work with d-Matrix illustrate how inference software can help new hardware platforms support real production workloads.

How do these products solve the problem?

• They solve the same problems, but with different product entry points. All these companies are trying to make inference faster, cheaper, and easier to run across different hardware environments. But they do not all solve the problem at the same layer. Some focus on how models are compiled and executed, others on the serving engine itself, while others orchestrate the full workload across different types of infrastructure.

• Approach #1: Compilers and hardware portability. Companies such as ZML and Modular focus on the layer between the model and the chip. Their products compile models into workloads that can run efficiently across different accelerators, reducing the need to rebuild the stack for each hardware provider. ZML is relatively focused on inference, while Modular is building a broader compute platform around its compiler, runtime, Mojo language, and cloud offering.

• Approach #2: Inference engines built around open source standards. Inferact and RadixArk take a different approach by building around widely adopted open source engines. Inferact is closely linked to vLLM, while RadixArk is built around SGLang. Their strategy is to improve the core engine, then add the infrastructure and enterprise tooling needed to run it reliably in production.

• Approach #3. Orchestrating inference across heterogeneous hardware. Gimlet Labs operates at a slightly higher level. Rather than only optimizing one engine or compiler, it breaks inference workloads into different components and assigns each part to the hardware where it can run most efficiently. Its thesis is that future inference workloads will increasingly run across a mix of chips rather than on a single hardware stack.

• Will these product scopes converge? These companies enter the market through different technical layers, but their products are gradually expanding toward the same destination. Compiler companies are adding serving infrastructure, engine companies are launching managed platforms, and orchestration companies are building their own optimization layers. The boundaries between these approaches are likely to become less clear over time.

Category #3: Model serving & deployment platforms

This category includes platforms that help companies deploy AI models into production and keep them running reliably as usage grows.

What problems do these companies solve?

• Once a company has selected or trained a model, turning it into a reliable production service is still complex.

• Teams need to provision the right infrastructure, keep latency under control, and scale capacity as usage changes.

• They also need to manage model versions, monitor performance, handle failures, and deploy updates without breaking the application.

• For larger companies, this often needs to happen inside their own cloud environment because of security, compliance, or data residency requirements.

• Basically, these platforms remove much of the infrastructure work required to turn a model into an API that can be used reliably in production.

Who are the customers?

• AI-native software companies running inference as part of their core product. Cursor uses Together AI to serve models with the low latency required for real-time coding workflows. These companies care directly about response time, reliability, and cost because inference is part of the user experience and their cost structure.

• Companies building AI agents and code-execution products. Ramp uses Modal Sandboxes to run internal coding agents, which reportedly generate around 30% of its production pull requests. Quora also uses Modal to provide isolated code execution inside Poe.

• Model labs that need to turn a model into a commercial service. They use platforms such as Baseten to deploy models behind their own API and domain, manage traffic across customers, and avoid building the serving infrastructure themselves. Baseten’s work around Mercury 2 is a good example of this model-lab-focused use case.

• Large companies standardizing model usage across different teams. Fireworks AI describes Trilogy using its platform to provide a common open-model infrastructure across internal teams and portfolio companies, rather than letting every team build and manage its own inference stack.

• Enterprises that need more control over where models run. Companies in regulated or security-sensitive industries use private-cloud, BYOC, or on-premise offerings from Baseten, BentoML, and FriendliAI to keep data and infrastructure inside their own environment.

• Cloud and infrastructure providers that want to offer a stronger inference layer. Nebius, for example, partnered with FriendliAI to integrate its inference optimization technology into its GPU cloud offering.

How do these products solve the problem?

• A common production infrastructure layer. Most of these products take a model and turn it into a production endpoint. They manage the underlying compute and handle scaling, reliability, monitoring, and deployment. The main promise is that companies can operate models without building their own inference infrastructure.

• Different ways to consume inference. The typical platform offers serverless endpoints for workloads that change frequently, dedicated deployments for predictable production usage, and batch inference for jobs that do not need an immediate response. Many are also adding private-cloud or self-hosted deployment options for larger enterprises.

• Inference platforms expanding across the model lifecycle. Baseten, Fireworks AI, and FriendliAI started from model serving but are gradually expanding into adjacent workflows. They now cover parts of model access, fine-tuning, evaluation, observability, and private deployment. Baseten is particularly focused on production infrastructure and enterprise deployments, while Fireworks and FriendliAI put more emphasis on optimized open-model inference.

• An open-source and bring-your-own-cloud approach. BentoML combines an open-source deployment framework with managed infrastructure. Its approach gives teams more control over how models are packaged and where they run, especially when they need to deploy inside their own cloud or on-premise environment.

• A broader serverless developer cloud. Modal and RunPod enter the market from a wider infrastructure angle. Model serving is one workload among several, alongside training, batch jobs, code execution, and other GPU-intensive tasks. Modal abstracts infrastructure through Python functions and Sandboxes, while RunPod combines direct GPU access with a growing serverless inference layer.

• A model catalog and API approach. Replicate focuses on making a large number of open-source and community models easy to discover and run through a common API. Its value is less about giving companies full control over the infrastructure and more about reducing the friction of experimenting with and integrating different models.

• A full-stack AI cloud approach. Together AI has the broadest scope. It combines model APIs, dedicated inference, fine-tuning, evaluation, and GPU clusters. It therefore sits somewhere between an inference platform and a neocloud, with the ambition of covering most of the infrastructure lifecycle.

• The products are gradually converging. The companies entered the market from different directions, but most are moving toward a similar destination. Model-serving platforms are adding training, serverless clouds are adding managed endpoints, and GPU providers are building their own inference optimization layers.

Category #4: Inference gateways, routing & control

This category includes software that sits between an application and different AI models, helping companies choose providers, route requests, monitor usage, and control costs and security.

What problems do these companies solve?

• Companies increasingly use several models and inference providers rather than relying on a single API. This creates a new infrastructure problem around how requests are routed, monitored, secured, and controlled.

• Different providers offer different levels of quality, latency, availability, and price. The best option can also change depending on the request, which makes static model selection inefficient.

• Applications need fallbacks when a provider is unavailable, caching to avoid unnecessary requests, and rate limits to prevent unexpected usage or costs.

• Once several teams and agents use models, companies also need a common layer to manage budgets, API keys, data policies, access rights, and security rules.

• Basically, these companies sit between the application and the model providers. They give companies one control layer through which inference traffic can be routed and managed.

Who are the customers?

• AI-native products using several models or providers. These companies use a gateway to access models through one API, switch providers without rebuilding their application, and automatically fall back when an endpoint fails. This is particularly useful for products whose quality or margins depend on continuously selecting the right model.

• Companies building agents that interact with several models and tools. Hermes Agent, for example, integrated Portkey to access thousands of models while tracking costs by session and applying budgets and governance rules to MCP usage. For these products, the gateway becomes the control layer around the agent rather than only a proxy for LLM calls.

• Large companies trying to standardize AI usage across teams. OpenRouter’s Workspaces and Portkey’s organization-level policies let platform teams give different departments access to models while keeping shared control over providers, budgets, rate limits, and data policies.

• Teams moving workloads from one model to another. Not Diamond helps them select the best model for each request, but also adapts prompts when they migrate between models. This matters because a prompt optimized for one model may perform poorly on another.

• Companies that need to understand what is happening in production. Helicone started from this use case by helping engineering teams inspect requests, costs, errors, sessions, and prompt performance. This is particularly useful when an AI application involves long or multi-step workflows that are difficult to debug with traditional observability tools.

• Security-sensitive and regulated organizations. These buyers use self-hosting, zero-data-retention rules, PII redaction, provider restrictions, and secret-management integrations to keep model usage within internal policies.

• Model and inference providers looking for distribution. OpenRouter also acts as a demand aggregation layer. Providers join the platform to make their models available through an API already used by a large developer base.

How do these products solve the problem?

• A shared gateway layer. At the most basic level, these products provide one API between the application and multiple model providers. They centralize credentials and make it easier to add, remove, or switch models without changing the application itself.

• A model marketplace and routing approach. OpenRouter has the broadest model and provider aggregation strategy. It combines unified access with routing, fallback, pricing, and provider-performance data. It is also expanding beyond routing into model discovery, multimodal APIs, web tools, caching, and agent infrastructure.

• An intelligent model-selection approach. Not Diamond focuses more specifically on deciding which model should process each request. Its router optimizes for quality, cost, and latency, while its prompt adaptation product rewrites prompts for the selected model. Its approach is therefore less about managing infrastructure and more about optimizing the model layer itself.

• An enterprise control-plane approach. Portkey combines routing with observability, security, and governance. Its product includes organization-wide policies, guardrails, audit logs, prompt management, and integrations with enterprise security systems. More recently, it extended this approach to agents through a dedicated Agent Gateway.

• An observability-first approach. Helicone originally focused on helping developers understand model usage through request logging, cost monitoring, sessions, and evaluations. It later added prompt management and an AI gateway, gradually moving from an analytics layer into the broader control path for inference.

• The category is expanding from models to agents. These products increasingly govern not only which model receives a request, but also which tools and MCP servers an agent can access, how much it can spend, and what data it is allowed to send.

• The category is also consolidating into larger platforms. Portkey was acquired by Palo Alto Networks and Helicone by Mintlify. This suggests that gateways may become a core component of broader AI security and developer infrastructure platforms rather than always remaining standalone products.