NextBigFuture

Defeating Nondeterminism in LLM Inference by Thinking Machines

A research article by Horace He and the Thinking Machines Lab (X-OpenAI CTO Mira Murati founded) addresses a long-standing issue in large language models (LLMs). Even with greedy decoding bu setting ...

DeepSeek open sources DSpark, a new framework to speed up LLM inference by up to 85%

DSpark can make decoding faster, but acceptance quality still determines how much speed the system actually realizes.
EDN

The hidden bottleneck in LLM inference and the impact on MLPerf benchmarking

Recent frontier LLM inference benchmarks have highlighted a recurring pattern. GPU-based systems deliver outstanding throughput when latency is not a concern, but their performance drops sharply once ...
InfoQ

NVIDIA Dynamo Addresses Multi-Node LLM Inference Challenges

Serving Large Language Models (LLMs) at scale is complex. Modern LLMs now exceed the memory and compute capacity of a single GPU or even a single multi-GPU node. As a result, inference workloads for ...
Semiconductor Engineering

Ultra-low-bit LLM Inference Allows AI-PC CPUs And Discrete Client GPUs To Approach High-end GPU-Level (Intel)

A new technical paper titled “Pushing the Envelope of LLM Inference on AI-PC and Intel GPUs” was published by researcher at Intel. “The advent of ultra-low-bit LLM models (1/1.58/2-bit), which match ...
Tech Times

NVIDIA’s New LLM Decodes 6x More Tokens Without an Auxiliary Draft Model

NVIDIA Nemotron-Labs-Diffusion is a new tri-mode language model that eliminates the separate draft model in speculative ...
Computer Weekly

Snowflake goes massive on Meta LLM for open source inference difference

The latest trends and issues around the use of open source software in the enterprise. Snowflake says it will now host the Llama 3.1 collection of multilingual open source large language models (LLMs) ...
InfoWorld

Multi-token prediction technique triples LLM inference speed without auxiliary draft models

With reported 3x speed gains and limited degradation in output quality, the method targets one of the biggest pain points in production AI systems: latency at scale. High inference latency and ...
techtimes

GitHub Copilot CLI Adds Pre-Commit Security Scanner: LLM Inference at the Detection Layer

GitHub shipped /security-review — a dedicated slash command for GitHub Copilot CLI — on Wednesday, putting AI-driven vulnerability scanning inside the terminal for the first time as an experimental ...
InfoWorld

Google targets AI inference bottlenecks with TurboQuant

Google says its new TurboQuant method could improve how efficiently AI models run by compressing the key-value cache used in LLM inference and supporting more efficient vector search. In tests on ...
Dataquest

OpenAI and Broadcom unveil LLM-optimized inference chip

OpenAI and Broadcom have unveiled Jalapeño, OpenAI’s first Intelligence Processor: an accelerator architected around OpenAI’s vision for the future of LLM inference, and the first AI accelerator in a ...