Configure Advanced Inference Ingress on OCI Kubernetes Engine (OKE) with the Gateway API Inference Extension
Oracle, Monday, October 5th, 2026
Oracle walks through LLM-aware inference routing on OKE using Istio, the Gateway API Inference Extension and llm-d.
This Oracle walkthrough shows how to deploy inference-aware routing on OCI Kubernetes Engine using the OKE-managed Istio add-on for Gateway API together with the Gateway API Inference Extension's InferencePool.
The setup also uses the llm-d Router endpoint picker and the llm-d Inference Payload Processor, with a demo customer-support app whose Standard and Fast tiers route the same Gemma 4 model on vLLM to CPU-backed or NVIDIA A10 GPU pools based on the requested model field.
The configuration reports per-request metrics such as time to first token and tokens per second.