The AI Inference Stack

21 Companies 4 Categories

As AI moves from model development to widespread production usage, inference is becoming its own infrastructure stack. Performance increasingly depends not only on access to GPUs, but on specialized chips, compilers, runtimes, serving systems, and routing layers that optimize latency, throughput, energy consumption, hardware utilization, and cost per request.

Clement Vouillon

Filter and ordering

Order companies by Show category

Category

Inference chips & systems

4 companies

Companies designing specialized hardware and system architectures that make AI inference faster, more power-efficient, and more economical at scale.

d-Matrix

d-matrix.ai

d-Matrix builds AI inference compute hardware and software for data centers, including inference accelerators, networking, and an orchestration stack designed to make generative AI faster, more efficient, and cheaper to run at scale.

51-200 employees Employees Series C Stage 2019 Founded

Trajectory Signals Strength Strong

Confidence: Medium

Etched

etched.com

Etched designs frontier inference clusters—specialized AI chips, racks, software, and manufacturing methods intended to run frontier models with high throughput, low latency, and better power efficiency.

400+ employees Employees Series B Stage 2022 Founded

Trajectory Signals Strength Strong

Confidence: Medium

Fractile

fractile.ai

Fractile designs AI inference chips, systems, and software intended to make frontier-model inference faster, more power-efficient, and lower-cost at scale.

90-100+ employees Employees Series B Stage 2022 Founded

Trajectory Signals Strength Strong

Confidence: High

Groq

groq.com

Groq is a U.S.-based AI inference company that designs specialized LPU chips and builds GroqCloud and related systems to run AI models faster and more affordably at scale. ([groq.com](https://groq.com/newsroom/groq-raises-usd650m-to-scale-its-ai-inference-cloud-business?utm_source=openai))

51-200 employees Employees Unknown Stage 2016 Founded

Trajectory Signals Strength Strong

Confidence: High

Category

Inference engines, compilers & runtimes

5 companies

Software that turns trained models into efficient executable workloads by adapting them to the underlying hardware and improving how inference is scheduled and run.

ZML

zml.ai

ZML builds a high-performance AI inference stack that helps run open-source models efficiently across different chips and production environments.

2-10 employees Employees Seed Stage 2023 Founded

Trajectory Signals Strength Moderate

Confidence: Medium

Modular

modular.com

Modular is an AI infrastructure company that builds compiler-driven software, including the MAX inference platform and Mojo language, to make model inference faster and more portable across heterogeneous hardware.

51-200 employees Employees Unknown Stage 2022 Founded

Trajectory Signals Strength Strong

Confidence: High

Inferact

inferact.ai

Inferact is an AI infrastructure startup founded by the creators and core maintainers of vLLM, focused on making large-model inference cheaper, faster, and easier to serve at scale.

Unknown Employees Seed Stage 2025 Founded

Trajectory Signals Strength Strong

Confidence: High

RadixArk

radixark.com

RadixArk is an infrastructure-first AI company that builds large-scale inference and training systems, including open-source serving and reinforcement-learning tooling, for developers, startups, enterprises, and research labs. ([radixark.com](https://www.radixark.com/))

11-50 employees Employees Seed Stage 2025 Founded

Trajectory Signals Strength Strong

Confidence: High

Gimlet Labs

gimletlabs.ai

Gimlet Labs builds an inference cloud for agentic AI workloads, using heterogeneous hardware orchestration, a hardware-agnostic compiler, and kernel generation to make inference faster and more cost-efficient.

11-50 employees Employees Series A Stage 2025 Founded

Trajectory Signals Strength Strong

Confidence: High

Category

Model serving & deployment platforms

8 companies

Platforms that help teams deploy and operate models as reliable production services while handling scaling and infrastructure complexity.

Baseten

baseten.co

Baseten is an AI inference infrastructure platform that helps teams train, deploy, and serve models in production with autoscaling, observability, and optimized serving infrastructure.

201-500 employees Employees Series D+ Stage 2019 Founded

Trajectory Signals Strength Breakout

Confidence: High

Fireworks AI

fireworks.ai

Fireworks AI is an AI inference cloud and developer platform that helps teams deploy, serve, fine-tune, and optimize generative models in production with an emphasis on speed, cost efficiency, and control.

265 employees visible on LinkedIn Employees Series C Stage 2022 Founded

Trajectory Signals Strength Moderate

Confidence: Medium

BentoML

bentoml.com

BentoML is a unified inference platform and open-source framework for deploying, scaling, and operating AI models in production with infrastructure, autoscaling, observability, and cloud/on-prem deployment options. ([docs.bentoml.com](https://docs.bentoml.com/en/latest/?utm_source=openai))

51-200 employees Employees Unknown Stage 2019 Founded

Trajectory Signals Strength Moderate

Confidence: Medium

FriendliAI

friendli.ai

FriendliAI is a generative AI inference platform that helps teams deploy and operate large language and multimodal models in production with optimized speed, latency, and cost.

11-50 employees Employees Unknown Stage 2021 Founded

Trajectory Signals Strength Moderate

Confidence: Medium

Modal

modal.com

Modal is a serverless cloud platform for AI/ML workloads that helps teams deploy and operate code, GPU jobs, inference, fine-tuning, and other production compute without managing infrastructure. ([modal.com](https://modal.com/company?utm_source=openai))

About 192 employees Employees Series C Stage 2021 Founded

Trajectory Signals Strength Strong

Confidence: High

Replicate

replicate.com

A platform that lets developers run, fine-tune, and deploy machine-learning models through an API without managing underlying infrastructure.

26 employees Employees Acquired Stage 2019 Founded

Trajectory Signals Strength Moderate

Confidence: Medium

Together AI

together.ai

Together AI is a full-stack AI cloud platform that helps teams run, fine-tune, train, and deploy open-source models and AI agents with production-grade inference and infrastructure.

201-500 employees Employees Series C Stage 2022 Founded

Trajectory Signals Strength Strong

Confidence: High

Runpod

runpod.io

Runpod is an AI developer cloud that lets teams build, train, fine-tune, deploy, and scale AI workloads on GPU infrastructure from one platform.

107 employees Employees Series A Stage 2022 Founded

Trajectory Signals Strength Strong

Confidence: High

Category

Inference gateways, routing & control

4 companies

Infrastructure that connects applications with model providers and provides a shared control layer for production inference.

OpenRouter

openrouter.ai

OpenRouter is an AI gateway and model marketplace that lets developers and enterprises route requests across hundreds of LLMs through a single API with built-in reliability, pricing, and fallback controls.

11-50 employees Employees Series B Stage 2023 Founded

Trajectory Signals Strength Strong

Confidence: High

Not Diamond

notdiamond.ai

Not Diamond builds an intelligent AI model router and prompt optimization platform that helps developers route requests across multiple models to improve quality while reducing inference cost and latency.

11-50 employees Employees Seed Stage 2024 Founded

Trajectory Signals Strength Moderate

Confidence: Medium

Portkey

portkey.ai

Portkey is an AI gateway and control-plane platform that gives developers and enterprises one unified layer for routing, observability, guardrails, and governance across multiple model providers.

11-50 employees Employees Unknown Stage 2023 Founded

Trajectory Signals Strength Strong

Confidence: High

Helicone

helicone.ai

Helicone is an AI gateway and LLM observability platform for developers that provides a unified API plus logging, monitoring, routing, caching, and cost/performance analytics across model providers. ([ycombinator.com](https://www.ycombinator.com/companies/helicone?utm_source=openai))

2-10 employees Employees Acquired Stage 2023 Founded

Trajectory Signals Strength Weak

Confidence: Medium

The AI Inference Stack competitive landscape summary

The AI Inference Stack contains 21 companies across 4 categories. As AI moves from model development to widespread production usage, inference is becoming its own infrastructure stack. Performance increasingly depends not only on access to GPUs, but on specialized chips, compilers, runtimes, serving systems, and routing layers that optimize latency, throughput, energy consumption, hardware utilization, and cost per request.