Lightbits Labs has officially launched its AI inference software engine, Infera, SiliconANGLE reported on Sept. 9 local time.

The company said Infera focuses on improving inference performance and cost efficiency by extending KV cache data beyond limited high-bandwidth memory connected to GPUs.

It targets large language model operating environments that use long context windows or have many concurrent sessions.

Infera manages KV cache across GPU high-bandwidth memory, DRAM and NVMe storage, using predictive prefetching that anticipates needed data and places it close to the GPU.

When a large language model (LLM) generates a sentence, it creates words one by one in sequence. To predict the next word, it must put all preceding words back into the calculation, and recalculating everything from the start each time takes a long time. AI therefore stores values it has already computed in memory and, when generating the next word, retrieves those stored values instead of recalculating them. The KV cache is the space that stores these values.

Infera does not load the entire context into GPU memory at once. It splits it into small blocks and supplies them when needed. It could be more effective when deployed in retrieval-augmented generation, AI agents, long prompts and GPU services shared by many users. The company explained that the effect may be limited in environments where it has ample spare capacity in its own GPU cluster, has few users and where the context already fits into available memory.