| Mobile Web

Search results for KV cache

AI & Enterprise

Lightbits launches KV cache engine to boost GPU performance

Lightbits Labs has launched its AI inference software engine, Infera, aimed at improving inference performance and cost efficiency. The company says Infera expands KV cache data beyond limited high-bandwidth memory attached to GPUs and targets large language model operations with long context windows or many concurrent sessions. It manages KV cache across GPU memory, DRAM and NVMe storage using predictive prefetching and supplies context in small blocks.

AI & Enterprise

Agent era accelerates multi-model use; small models set to take larger share

Multi-model use inside generative AI services is already common, with different-sized models dividing work, a Ravlub researcher said. Small models handle routine tasks such as search, information gathering and summarisation, while frontier models take key decisions and generate outputs. As agents spread, repeatedly calling top-performance models can push inference costs higher. He said small models will not replace frontier models, citing scaling, but role-based use of multiple model sizes will expand.