Smart AI Router

Overview

Smart AI Router is a centralised routing and observability platform for AI requests. Teams used to call AI vendors directly in their own way. Now each request flows through one governed layer. That layer picks the right vendor and model, logs the call, measures cost and handles failures. It serves engineering organisations running AI across many features who need consistency, visibility and control over spend.

Challenges

Teams called AI vendors directly in ad hoc ways with no shared standard for how requests and responses should be handled. The same integration patterns were duplicated across the codebase. That multiplied maintenance effort and guaranteed the copies would eventually diverge. There was no centralised logging or correlation tracking. Nobody could say which AI requests had run or when or with what result. Token usage and the real cost of AI calls weren't measured at all. Spend was impossible to reason about or control. Vendor and model choices were static and disconnected from task complexity. Simple requests and hard ones were routed identically which wastes both latency and cost. Asynchronous and specialised operations were poorly supported and prone to integration errors. When a vendor call failed there was no consistent fallback behaviour so each request was exposed to a single point of failure. Migration and compatibility problems made it risky to evolve integrations or onboard a new vendor. And without enforceable standards each new AI feature reintroduced the same inconsistencies from scratch.

Solution highlights

We built one centralised routing system that all AI calls pass through. On top of it sits a vendor and model selection engine. It picks the right backend based on task complexity using configurable rule-based routing policies rather than hardcoded choices. Fallback and failover logic reroutes requests gracefully when a vendor is unavailable. The single point of failure is gone. We standardised the interface across image, audio, transcription, text-to-speech and vision workloads. Asynchronous request handling means long-running and specialised operations are supported alongside standard calls. Each request is logged with correlation tracking so it can be traced end to end. A token usage and cost calculation system quantifies consumption and spend across vendors and models. Usage analytics and administrative reporting turn that data into something teams can act on. To stop the inconsistency creeping back we added linting and enforcement for standardised integrations. Migration tooling and developer documentation let new vendors and use cases come on board without reinventing the pattern.

Key Features

• Centralised routing for all AI requests through one governed entry point
• Vendor and model selection based on task complexity
• Rule-based intelligent routing with configurable policies and constraints
• Routing fallback and failover when a vendor is unavailable
• Asynchronous AI request handling for long-running operations
• Standardised APIs for image, audio, transcription, text-to-speech and vision
• Request logging with correlation tracking for end-to-end tracing
• Token usage and cost calculation across vendors and models
• Usage analytics and administrative reporting dashboards
• Linting and enforcement to keep integrations standardised
• Migration tooling and developer documentation
• Structured request tracing and auditing

Our Role & Approach

Our role was to replace a pattern and not just write a service. The technical work was straightforward compared to the organisational problem. If the router isn't easier to use than calling a vendor directly teams will bypass it. So we built a standardised interface covering each workload type and documented it properly. We supplied migration tooling to move existing code across. We added linting that catches direct vendor usage before it reaches the codebase. Observability was a first-class feature rather than an afterthought. Cost visibility is usually what convinces an organisation that centralised routing was worth doing. Routing rules stayed configurable rather than hardcoded so the platform can adapt as models and prices change. And they change constantly.

Technology Stack

Python, Django, OpenAI APIs and other vendor integrations, asynchronous services, structured logging systems, administrative dashboards and a custom AI routing engine.

Outcomes

All AI requests now flow through one centralised router. The organisation has a single consistent way to integrate AI. Vendor and model selection adapts to task complexity so routing decisions are optimised for both latency and cost rather than fixed in advance. Full request logging with correlation tracking gives structured tracing and auditing across all AI activity. Token usage and cost are measurable for the first time so AI spend is visible. Fallback and failover logic keeps requests resilient. A vendor outage doesn't break the workflow any more. Asynchronous and specialised operations such as vision, audio and transcription are supported reliably through standardised APIs. Duplicated vendor patterns are gone and consistency is enforced rather than hoped for. Observability and monitoring coverage improved to the point where teams can see, audit and report on AI behaviour with confidence. Linting, migration tooling and documentation make the platform straightforward to maintain and extend as new vendors arrive.