Meta opens up, robots offload their brains · Morning Gist
Sunday, October 185 stories · 16 min
Meta opens up, robots offload their brains
In today's stories the limit is rarely the model itself: it's a license, an image encoder, a robot's battery or the orchestration around it.
Picked from 120 new articles across 24 sources · arrow keys for other days
Today in 30 seconds
30s left
Meta's 816B-parameter Muse Spark Open is now the strongest American open-weight model, but its custom license limits how you can use it in products. story 1 ↓
Liquid AI's 280M drafter speeds decoding up to 3.13x on Apple silicon, but slow image processing holds total gains to 1.3–2.6x. story 2 ↓
Microsoft found onboard GPUs halved robot-arm model accuracy and cut battery life. Its open-source toolchain now offloads inference to remote GPUs. story 3 ↓
Google strings Veo clips into consistent minutes-long videos using planning, visual memory and self-correction. The video model is unchanged, and the post reports almost no numbers. story 4 ↓
NVIDIA's open NV-Reason-CT, a 4B Qwen model, reads all 13,824 tokens of a 3D CT scan and tops CT-RATE on macro-F1. story 5 ↓
1/5Interconnects (Nathan Lambert)· Nathan Lambert· Oct 17
Meta releases Muse Spark Open, now the strongest American open-weight model
Meta released the weights of Muse Spark Open, an 816-billion-parameter model that Nathan Lambert ranks as the best American open model by a wide margin. It still trails the top Chinese models, and its license is restrictive.
Why it matters · If you self-host or fine-tune open models, there is now a credible American option near the Chinese frontier, and its license terms affect whether you can use it in a product.
Know this first
Context from the editor, not the article · 29 sec
1 · Open-weight vs. open-source
An open-weight model publishes its trained parameters so anyone can run or fine-tune it. That is different from open source: the license can still restrict use, and training data and code usually stay private.
2 · Total vs. active parameters
Many large models are mixture-of-experts: each token passes through only a small subset of the network. "816B total / 42B active" means the model stores 816B weights but uses about 42B per token, so it runs much cheaper than its size suggests.
3 · The Chinese open-model lead
Over the past two years, Chinese labs such as DeepSeek, Zhipu (GLM), Moonshot (Kimi), Alibaba (Qwen) and Xiaomi have released most of the strongest open-weight models. American open models have lagged well behind them.
2/5Hugging Face· Sep 24
Liquid AI adds a speculative-decoding drafter to its 3B vision-language model
Liquid AI released a small "draft" model that guesses ahead for its LFM2.5-VL-3B image-and-text model. Output stays the same, and text generation runs 1.6–3.1x faster. The total response-time gain is smaller.
Why it matters · It's a clean, measured case of Amdahl's law in multimodal inference. Speeding up decoding doesn't help much on edge devices when image processing dominates latency.
Know this first
Context from the editor, not the article · 30 sec
1 · Speculative decoding
A small draft model proposes several next tokens cheaply. The large target model then checks all of them in one parallel pass and keeps the longest correct prefix. The output matches what the target would produce on its own; only the speed changes.
2 · Prefill versus decode
Inference has two phases. Prefill processes the whole prompt at once, which makes it compute-heavy. Decode then produces one token per step and is limited by memory bandwidth. Speculative decoding only speeds up the decode phase.
3 · Vision-language models
A VLM runs each image through a vision encoder, which turns it into hundreds of "visual tokens". A language model then reads those tokens alongside the text prompt and writes the answer token by token.
3/5Microsoft Research· Ganesh Ananthanarayanan, Matthew Balkwill, Xenofon Foukas, Sanjeev Mehrotra, Bozidar Radunovic, Connor Settle, Ankit Verma, David White, Shawn Cicoria, Mark Martin, Rachel Johnson, Mayur Patel· Sep 23
Microsoft finds robots work better when they run AI on remote GPUs
Microsoft Research ran robot AI models on onboard, nearby and cloud GPUs. Remote GPUs raised task success and roughly doubled battery life on one robot. Microsoft released open-source tools that move this processing off the robot.
Why it matters · Model size in robotics is growing faster than what a battery-powered robot can carry. This puts numbers on moving robot inference to remote GPUs, much as LLM serving already runs on remote GPUs.
Know this first
Context from the editor, not the article · 31 sec
1 · Physical AI and its models
"Physical AI" means models that control robots. A robot that can move around and handle objects (mobile manipulation) chains several models: mapping and planning, navigation, and vision-language-action models that turn camera images and instructions into motor commands.
2 · Where robot inference runs today
The usual design puts a GPU on the robot, often an NVIDIA Jetson module. Cloud servers sometimes handle high-level planning, but the moment-to-moment control loop runs on the robot so it doesn't depend on a network.
3 · Why latency breaks control
4/5Google Research· Sep 24
Google's agent stack strings Veo clips into consistent, minutes-long videos
Google Research built four agent frameworks that run on top of its Gemini and Veo models. They plan stories, track characters and sets, and fix defects by rewriting prompts, producing videos up to ten minutes long that stay visually consistent.
Why it matters · The underlying video model is left unchanged, so this is a test of how much orchestration, memory and model-as-judge loops can recover on their own, a pattern that carries over to any long-horizon generation task.
Know this first
Context from the editor, not the article · 29 sec
1 · Why long video breaks
Video diffusion models produce clips a few seconds long. Longer stories need many clips chained together, and each clip is generated with no memory of the others, so faces, costumes and room layouts slowly change from shot to shot.
2 · Agentic pipelines
Existing tools split the job across LLM-driven steps: script, then images, then video. Each step gets its own hand-written prompt, so a mistake early on carries into everything after it, and when the final video is bad it's hard to tell which step caused it.
3 · Multi-armed bandits
A multi-armed bandit is an algorithm that keeps choosing between several options, watches the rewards, and balances trying new options against reusing ones that have already scored well.
5/5NVIDIA Technical Blog· Tanya Lenz· Sep 23
NVIDIA releases NV-Reason-CT, an open model that reasons over full 3D CT scans
NVIDIA released NV-Reason-CT, an open model that pairs a native 3D image encoder with a small Qwen language model. It writes radiologist-style reasoning and reports for chest and abdomen CT, and leads published baselines on CT-RATE.
Why it matters · It is a concrete, open recipe for long-context multimodal input: uncompressed 3D tokens with 3D position encoding, plus SFT-then-GRPO training with a reward split by region.
Know this first
Context from the editor, not the article · 31 sec
1 · Why CT is hard for VLMs
A CT scan is a 3D volume, often 300–600 stacked slices. Most vision-language models take 2D images, so CT usually gets fed in as a handful of separate slices, which loses how structures continue from one slice to the next.
2 · CT-RATE benchmark
CT-RATE is a public dataset of chest CT volumes paired with radiology reports. It is commonly used to score models on detecting 18 abnormality labels, using F1 (precision and recall at a threshold) and AUROC (ranking quality).
3 · SFT then GRPO
Many reasoning models are first trained by supervised fine-tuning (SFT) on example answers. Reinforcement learning comes next. GRPO, a common choice, scores a group of sampled answers to each prompt and pushes the model toward the ones that score above the group's average.
Chart of the narrowing gap between the best American and Chinese open-weight models.1 / 3
Meta has released the weights of Muse Spark Open. Nathan Lambert of Interconnects calls it the strongest American 1 model "by quite some margin". (An open-weight model is one whose trained parameters are public, so anyone can run or fine-tune it.) Reflection also shipped its first model, Beam, this week. Lambert counts more than 20 Chinese open models that would beat the best Western one without these two. With them, he estimates the US–China gap in open models is about half what it was. American open models are now "behind but relevant."
The model
Muse Spark Open has 816B total parameters, but only 42B are active for each token. It is a 2 design, so it runs at a fraction of its full size. It has a 1M-token 3, takes images and other inputs natively, and is aimed at coding and agents. Lambert places it above GLM 5.2, near DeepSeek V4.1 Flash, and behind GLM 5.3 and Kimi K3, Chinese models released in July and August.
Meta added extra safety 4 for the open release. Some observers worried this had damaged the model. Lambert thinks it is more likely a safety tune of the older Muse Spark 1.2, costing about one point on the 5, an aggregate benchmark score.
Adoption vs. the license
Lambert's data shows Western open models get far more adoption than Chinese models of similar quality. Demand is also rising now that open models can run as agents. The main risk he sees is Meta's custom license:
Products must display "Built with Muse".
Businesses making over $20M a month need a separate deal with Meta.
Users must accept an 6.
The terms cover outputs and derivatives, which makes generating synthetic training data legally murky.
He expects most users to ignore the license. He still calls it an unnecessary strategic risk.
Where Meta stands
Days earlier, Meta's closed models, Muse Wonder and Muse Wonder Full, benchmarked behind only Anthropic's Opus 5.5. Lambert credits the turnaround to organisation. Alexandr Wang reportedly has more than 200 direct reports, a very flat structure, in contrast to the politics he blames for Llama's failure. His ranking: Anthropic > OpenAI > Meta ≈ Gemini > xAI ≈ Chinese open > US open.
He sees US frontier competition shrinking to five labs as a major model now costs billions. He also notes that labs still can't predict whether a model will land well in real use. Anthropic's run went from a hit (Fable 5) to a dud (Opus 5) to another hit (Opus 5.5).
Meta has released the weights of Muse Spark Open. Nathan Lambert of Interconnects calls it the strongest American-made 1 model "by quite some margin". The same week, Reflection shipped its first model, Beam. Mistral announced Mistral Large 4, which scores 38 on the 2 (an aggregate score built from many benchmarks), but has not yet released its weights. Lambert's count is that without Muse Spark Open and Beam, more than 20 open-weight models from Chinese companies would beat the best Western one. With them, he estimates the gap between the best American and Chinese open models has been "reduced roughly by half". American open models have gone from irrelevant to anyone who needs top performance to "behind but relevant."
What the model is
From Meta's model card on Hugging Face: Muse Spark Open has 816B total and 42B active parameters. It is a sparse model that touches only about 5% of its weights for each token. It has a 1M-token 3, it is natively multimodal, and it is built for coding and agentic work. Meta added extra safety 4 (the fine-tuning stage after pretraining) so that it resists misuse when run without the guardrails Meta's own systems provide. Meta still recommends deploying it inside a broader system with its own safeguards.
On capability, Lambert puts it above GLM 5.2, near DeepSeek V4.1 Flash, and behind GLM 5.3 and Kimi K3. Those Chinese models came out on August 18 and July 16. So Meta's best open model is competing with Chinese releases that are months old. Lambert asks what Zhipu and Moonshot have been doing since, and leaves that for another post.
Was it broken by safety training?
The open model arrived after Meta's closed Muse Spark 1.3. Some observers suspected the safety tuning had degraded a 1.3-class model. Lambert thinks it is more likely a safety tune of Muse Spark 1.2, which would cost only about one point on the Artificial Analysis index. The evidence he gives: when Meta released Muse Glimmer, a 30B agentic model, Alexandr Wang said Meta was preparing to release Muse Spark 1.2 as open weights. A chart in the post shows where the different Muse models land on the index.
Why it should catch on
Lambert's analysis of open models keeps finding that Western open models get "astonishingly more adoption" than Chinese models with the same performance for their size, even though the Chinese ones usually ship first. He gives three conditions that favour Muse Spark Open:
Demand is rising fast now that open models can work as autonomous agents. He places that threshold at roughly GLM 5.2-level capability.
5, the companies that host models behind an API, are scaling quickly.
An open post-training economy is forming, and many enterprises want to own their models instead of renting them.
He thinks it is "squarely good enough" to fit that demand.
Why it might not: the license
Lambert names the custom license as the main risk. It includes nearly every restrictive term he has criticised in open-model licenses:
Products built on the model must display "Built with Muse". Meta made the same demand for its earlier Llama models, and Kimi has tried it. Almost no one complies.
Companies making more than $20M per month across their whole business need a separate license from Meta. This is meant to set up revenue sharing with providers that resell inference.
Users must agree to an 6, which by open-source software standards makes the model "extremely not open-source."
These terms extend to model outputs and derivatives. That makes it legally unclear whether you can use the model to generate synthetic training data.
Internal research and internal applications are permitted. Lambert expects most people to ignore the terms and access the model through inference providers, with Meta pursuing only egregious violations. Licenses have not decided AI adoption so far. He still argues a non-standard license is an "unnecessary strategic risk" for a lab trying to win adoption back.
The closed side and the lab tiers
A few days earlier Meta launched closed models, Muse Wonder and Muse Wonder Full. On traditional benchmarks they trail only Anthropic's Opus 5.5 and roughly match OpenAI's GPT-6-Astra and Google's Gemini-4-Argon. Lambert doesn't know how the two differ. They may be separate pretraining runs, or "Full" may be a more looped version of the same model; a looped architecture reuses layers repeatedly to get more computation per token. Early impressions are solid, but nobody expected them to match the polish of the latest Anthropic and OpenAI models in real use. Meta is now the only American company with frontier models both open and closed. Alibaba's Qwen follows a similar but more open approach.
Lambert credits the turnaround, which comes about a year after Mark Zuckerberg's "Personal Superintelligence" redirection, to organisation more than money. Wang reportedly has more than 200 direct reports. That is a very flat structure, with high ownership and a lot of compute per researcher. Lambert argues that resources are necessary but organising them is the hard part. He describes Llama's failure as model decisions made "out of politics rather than performance." Gemini has a reputation (overstated, he says) for layers that block work. xAI may suffer from talent turnover. In his view, what sets OpenAI and Anthropic apart is consistency in staying at the frontier. His ranking: Anthropic > OpenAI > Meta ≈ Gemini > xAI ≈ Chinese open weights > US open weights, with Xiaomi V2.6 Pro the leading open-weight model overall.
Open questions
Lambert sees the long-predicted squeeze happening. Serious US frontier competition has narrowed to five labs: Anthropic, OpenAI, Meta, Gemini and xAI. Microsoft could join if it starts releasing models, given its recent scaling-science results and its access to OpenAI's IP. "Tier 3" labs such as Mistral, Nvidia's Nemotron, Reflection and Arcee can still build businesses, but they struggle to move either frontier when a major model costs billions. The models are also too close to each other to suggest a single winner through 7, where AI speeds up its own development.
He points to one weakness that money hasn't fixed: labs still can't reliably predict how a model will do in real use. Anthropic went from a hit (Claude Fable 5) to a dud (Opus 5) and back to a hit (Opus 5.5). Evals are a proxy, and the long tail remains "largely unknowable."
On money, Meta is valued under $2T, about where Anthropic is heading into its IPO. Meta has a very profitable core business and is half a step behind on the technology. Lambert argues one of the two valuations has to "correct massively." The longer Meta and others stay competitive, the more pressure there will be on AI IPO valuations.
Remember
Muse Spark Open runs 42B of its 816B parameters per token and still trails Chinese models from months ago.
Its license gates businesses over $20M per month and covers outputs, which clouds use for synthetic training data.
Lambert argues that organisation, not just compute, separates consistent frontier labs from one-off successes.
Diagram of how the drafter reads the VLM's hidden states to propose candidate tokens.1 / 3
Liquid AI has released LFM2.5-VL-3B-DSpark, an experimental "draft" model for its 3-billion-parameter 1 (VLM, a model that reads images plus text and writes text). It adds 2 to the model and comes with launch-day support in llama.cpp, MLX-VLM and SGLang.
How speculative decoding helps
Language models write one token per step, and each step reads every weight from memory, so generation is slow. In speculative decoding, a small draft model guesses a block of upcoming tokens. The full model checks the whole block in one pass and keeps the guesses it agrees with. Because the full model verifies every token, the output doesn't change; with greedy decoding (always taking the most likely token) it is identical. What changes is how many tokens each expensive pass produces.
Why images didn't need a new design
The drafter has the same architecture as Liquid's text-only DSpark drafters. It reads the main model's internal 3 at a few fixed layers and predicts a block of candidate tokens from them. Image patches and text are projected into the same representation before those layers. So the drafter sees vectors of the same size from either source, and the algorithm stays the same. The post doesn't explain the wider DSpark recipe.
The drafter has 4 attention-only layers. That depth was picked by testing 3, 4 and 5. It was trained for 10 epochs on vision-language fine-tuning data, and acceptance improved until returns diminished. It has 279.5M parameters, adding 8.9% to the 3B model.
Results
Speedups by setup, measured on six task types from the 4 benchmark (question answering, captioning, chart reading, reasoning, multi-turn chat):
MLX on M5 Max: decode 2.30–3.13x faster; end-to-end 1.56–2.62x
llama.cpp on M3 Ultra: decode 1.57–2.14x; end-to-end 1.30–1.77x
H100 GPU: decode up to 2.66x; end-to-end 1.64–2.27x. The post gives the low end as "20.4x", most likely a typo for 2.04x.
The post doesn't report acceptance rates.
The catch: 5
Before decoding starts, a VLM has to run the image through a vision encoder. It then has to "6", meaning process hundreds of image tokens plus the prompt in one pass. That work is limited by compute, and edge devices have little to spare, so these stages take a large share of response time. Speculative decoding speeds up none of it. By Amdahl's law, the part you don't speed up caps the overall gain. That's why end-to-end gains always trail decode gains and are smallest on the M3 Ultra. Long answers benefit most.
To try it, use a build of each runtime that includes Liquid's pull requests. Weights come in Safetensors and 7 formats.
Liquid AI has released an experimental draft model for LFM2.5-VL-3B, its 3-billion-parameter 1 (VLM, a model that takes images plus text and writes text). The drafter is called LFM2.5-VL-3B-DSpark. It adds a speculative-decoding path, which speeds up text generation without changing the output, for a small extra memory cost. It follows the text-only LFM2.5-DSpark drafters the company released recently. On launch day it works in three inference runtimes: llama.cpp, MLX-VLM and SGLang.
The problem it targets
Language models generate text one token at a time. Each step reads all the model's weights from memory, so the decode phase is limited by memory bandwidth rather than compute. In 2, a cheap draft model guesses a block of upcoming tokens. The full "target" model then checks the whole block in one forward pass and keeps the longest prefix it agrees with. Accepted tokens cost a fraction of a full step, and rejected ones cost little beyond the wasted draft work. Because the target verifies every token, the result is exact: with greedy decoding (always picking the most likely token), output matches the target running alone.
Here is an illustrative example, not from the article. Suppose the drafter proposes 8 tokens and the target accepts the first 5. The target's verification pass also yields the correct 6th token, so one expensive pass produces 6 tokens instead of 1. Speedup depends almost entirely on the acceptance rate, meaning how often the drafter guesses what the target would have written.
How the vision drafter works
The drafter reuses the architecture of Liquid's text DSpark drafters. It reads the target model's 3 at a fixed set of "tapped" layers and uses them to predict a block of k candidate tokens. Making this work for images needs no new machinery, because of where the modalities meet. Image patches and text tokens are projected into one shared representation before the tapped layers. So the drafter always sees hidden-state vectors of the same size, whether they came from pixels or words, and the inference algorithm is the same as for the text models.
The article doesn't explain the DSpark recipe itself, beyond that it follows the earlier drafters. It also doesn't say what some of its named parts do: the "Markov head" and the "confidence head". Going by the names, the confidence head probably estimates how far a draft can be trusted, but the post doesn't confirm this.
Training and size
Liquid trained the drafter on a mix of vision-language supervised fine-tuning (4) data, weighted toward the workloads it expects the model to serve. Ablations compared drafters with 3, 4 and 5 layers. The final model is a simplified attention-only drafter with 4 layers, trained with a block size of 9. Training ran for 10 epochs. Liquid measured acceptance after each one and saw it rise with more tokens until returns diminished.
The parameter breakdown:
Decoder stack (4 layers): 193.0M
Hidden-state projection: 21.0M
Markov head: 65.5M
Norms + confidence head: 6.4k
Total: 279.5M, which adds 8.9% on top of the 3B target
For inference, Liquid recommends a block size of 8 or 9 depending on hardware.
Results
Every benchmark used block size 8 and six task types taken from the 5 benchmark: general visual question answering (VQA), text VQA, image captioning, chart VQA, complex reasoning and multi-turn conversation. Ranges run across tasks.
Setup
Decode speedup
End-to-end speedup
MLX, M5 Max
2.30x–3.13x
1.56x–2.62x
llama.cpp, M3 Ultra
1.57x–2.14x
1.30x–1.77x
SGLang, H100
"20.4x"–2.66x
1.64x–2.27x
The H100 decode figure appears in the post as "20.4x to 2.66x". The order and the other numbers suggest the low end is 2.04x and the "20.4x" is a typo. The post doesn't give acceptance rates per task, and it doesn't compare against other drafting methods.
Why end-to-end gains are smaller
The most useful part of the post is its honest account of the limits. A VLM request has three stages: the vision encoder processes the image, the language backbone runs 6 over hundreds of visual tokens plus the text prompt, and then decode begins. Prefill is mostly compute-bound, and its cost grows with prompt length, up to quadratically. Edge devices have far less compute than datacenter GPUs, so on them the encoder and prefill take a larger share of total latency. Liquid notes that the per-core GPU neural accelerators in Apple's M5 narrow this gap somewhat.
Speculative decoding only shortens decode. This is 7: the overall speedup is capped by the share of work you didn't speed up. An illustrative case: if encoding and prefill take 40% of wall time and decode becomes 3x faster, total time falls from 1.0 to 0.4 + 0.6/3 = 0.6. That is a 1.67x end-to-end gain. The table fits this pattern. The M3 Ultra running llama.cpp shows the smallest end-to-end numbers, and on every setup the end-to-end gain trails the decode gain. Tasks with long answers, such as captioning or reasoning, should benefit most; short-answer VQA least.
Using it
The weights are open and come in Safetensors and 8 formats. Each runtime currently needs a build that includes a specific pull request: SGLang #40651, llama.cpp #29339, MLX-VLM #2280.
SGLang: launch the target with --speculative-algorithm DSPARK, the draft model path, --speculative-dspark-block-size 9 and --disable-radix-cache. This serves an OpenAI-compatible endpoint. Disabling the radix cache turns off prefix caching, which may cost something on workloads that share prompts.
llama.cpp:--spec-type draft-dspark --spec-draft-n-max 8, plus the multimodal projector file. Oddly, the example draft file is named LFM2.5-2.6B-DSpark, the text drafter's name. That may be a typo.
Per-response timings report drafted versus accepted tokens, so you can measure acceptance on your own workload.
What's left open
The release is labeled experimental. The post doesn't report acceptance rates, quality checks under sampling (it only guarantees identical output for greedy decoding), how batching affects GPU speedups, or how the drafter does on images unlike its training mix. For anyone deploying a small VLM on device, the practical lesson is to measure time-to-first-token before counting on decode speedups.
Remember
Shared image-and-text representations let a text drafter design carry over to a VLM unchanged.
End-to-end gains trail decode gains because image encoding and prefill aren't sped up.
An 8.9% parameter overhead bought up to 3.13x faster decoding on an M5 Max.
Robots act in a world that keeps moving. If a model takes too long to answer, the robot acts on a stale view: it misses an obstacle or grabs where an object used to be. Slow inference therefore causes wrong actions, not just slow ones.
The models tested for mapping and planning, navigation and manipulation in the study.1 / 4
The news
Microsoft Research tested a common robotics assumption: that a robot's AI models should run on a GPU the robot carries. The team ran the same workloads on small onboard GPUs, on nearby "edge" GPUs and in the cloud. Moving inference off the robot improved task success and battery life. Microsoft also released tooling that makes the move easier.
Why it's hard
Robots run a closed control loop, so inference that arrives late is effectively wrong: the robot acts on a scene that has already changed. That is why designers keep compute on board, accepting the extra power draw, weight and cost, and limits on model size. Sending data to a remote GPU adds network delay and needs bandwidth. The question is which cost is worse.
What they measured
The test task was 1: "check for rubbish in the kitchen and put it in the trash." That needs three kinds of model: mapping and planning, navigation, and manipulation by 2 (vision-language-action) models, which turn images and instructions into arm movements.
The smallest GPUs couldn't fit the full model stack at all.
Mapping and planning ran up to 383% slower than on an NVIDIA 3.
Navigation detected obstacles on time about 30% less often on lighter GPUs.
VLA models slowed only modestly, but their accuracy still fell 50%. Small delays matter a lot for control.
In a task where one arm hands tape to another, success was low with onboard-class 4 Nano and Orin GPUs and substantially higher when inference was offloaded to a Jetson Thor or A100. The article shows this only as a chart.
For the battery test, the team swapped the onboard GPU for a Raspberry Pi 5 that just forwards sensor data. A Jetson Thor on board cut battery life by up to 160%, a few hours. On the 5 robot, offloading more than doubled run time.
The tooling
Microsoft added an offloading feature to its open-source 6. It uses 7, the standard container scheduler, to treat the robot, edge GPUs and the cloud as one pool. You declare what to offload; the toolset puts GPU inference in its own container and places it. It works with (the standard robotics middleware) and (Hugging Face's robot-learning library), with examples for the SO-101 and UR10e arms.
Caveats
The authors call offloading a "complex tradeoff" involving network latency, bandwidth and GPU availability. The blog post doesn't quantify that tradeoff. It also doesn't say what network the tests used, how robots behave when the link drops, or how a shared edge GPU handles many robots. Hardware and per-model details are in a separate technical report.
The news
A Microsoft Research team has published what it calls the first systematic study of where robot AI inference should run. The robotics field mostly assumes that a robot's models run on a GPU carried by the robot. The team argues that this assumption is now holding robots back. They ran the same robot workloads on small onboard GPUs, on nearby "edge" GPUs and on cloud GPUs. Offloading improved task success, response time and battery life. Alongside the study, Microsoft added a Kubernetes-based offloading feature to its open-source 1.
The problem
Robots usually carry their own GPU, such as an NVIDIA 2 module. Cloud servers sometimes handle high-level planning, but the robot executes tasks with onboard compute. The authors list the costs: GPUs draw significant power, shorten battery life, add cost and weight, and may be too small to run the newest models. Models keep getting bigger, so the gap keeps widening.
Moving inference off the robot isn't free. A robot runs a closed control loop, so every round trip to a remote GPU adds network delay and needs bandwidth for camera and sensor data. Speed alone also isn't the goal. A model that answers late is effectively wrong, because the scene has changed while it was thinking.
What they measured
The study used 3, with a standard household task: "check for rubbish in the kitchen and put it in the trash." That task requires planning a route to the kitchen, perceiving the scene to find rubbish, navigating to it, picking it up and navigating back to the bin. The team picked representative state-of-the-art models for three capabilities:
Semantic mapping and planning: building a map labelled with objects and places, then planning on it.
Navigation: moving safely while detecting obstacles.
Manipulation: handled by 4 (vision-language-action) models, which map images and instructions to arm actions.
The blog lists the specific models in a table (Figure 2) but doesn't name them in the text. Exact test hardware is in a separate technical report. From the figures, the hardware ranged from onboard-class GPUs (Jetson Nano, Jetson Orin AGX) to offload targets (Jetson Thor, NVIDIA 5).
Results
Task performance. The smallest GPUs couldn't fit the full mobile manipulation stack at all. On GPUs with enough memory:
Mapping and planning ran up to 383% slower than on an A100. The authors say this limits the robot in spaces where things move around.
Navigation detected obstacles on time about 30% less often on lighter GPUs.
VLA models slowed down only modestly, but even that cut their accuracy by 50%.
The VLA result is the most interesting one. It shows that a fairly small latency increase can halve task accuracy. A rough analogy: catching a ball with a half-second delay in your vision. You can see perfectly well, but your hand arrives where the ball used to be.
A dual-arm task in which one arm hands a roll of tape to the other shows this concretely. With onboard-class GPUs (Jetson Nano, Orin AGX), handover success was low. With the same inference offloaded to a Jetson Thor or an A100, success rose substantially (Figure 4). The article gives the comparison only as a chart, with no exact percentages in the text.
Battery life. For this test the team replaced the onboard GPU with a Raspberry Pi 5, which only collects sensor data and sends it to the remote GPU. Large onboard GPUs such as Jetson Thor cut battery life by up to 160%, which the authors say means "a few hours," even on larger robots. On the 6 mobile robot, offloading more than doubled battery life (Figure 5).
Jetson Thor appears in both roles. It is a battery drain when mounted on the robot, and a useful offload target when it sits nearby on wall power. The argument is about where the GPU sits, not which chip it is.
The tooling
The second half of the announcement is a toolset that makes offloading routine. The team chose 7, the standard system for scheduling containers across machines, as one layer that treats the robot's own compute, an on-premises edge GPU and cloud overflow as a single pool. The toolset:
automatically packages robot workloads into containers from a declarative spec (you state what to offload, not how);
splits GPU inference into its own container and places it using what the authors call "smart policies" (details not given);
integrates with robot simulators, 8 (Hugging Face's open-source robot-learning library) and 9 (Robot Operating System 2, the standard robotics middleware).
The toolset ships inside Microsoft's Physical AI Toolchain, an open-source framework that connects Azure cloud services to NVIDIA's robotics software for data curation, evaluation, imitation learning and reinforcement learning. The release includes example projects for the SO-101, a low-cost open-source arm, and the UR10e, an industrial arm from Universal Robots. Demo videos show Microsoft's Rho model, built for dual-arm robots, running on a Jetson Thor and controlling a Mobile Aloha robot to press a button and turn a knob on a test board.
What's left open
The authors say themselves that offloading is "a complex tradeoff involving performance, network latency and bandwidth, and available GPU resources." The blog post doesn't quantify that tradeoff, and that gap matters most for practitioners. Missing details include:
what network setup (wired, Wi-Fi, 5G) the results assume, and how much latency it adds;
how a robot behaves safely when the link drops or slows down;
how one edge GPU is shared between many robots, and what happens under contention;
what the "smart policies" for placement actually decide.
The "industry-first" and "first systematic study" claims are Microsoft's own. Readers who want the hardware matrix and per-model numbers will need the technical report. The main point holds anyway: for current robot models, an onboard GPU pays for itself neither in accuracy nor in battery life.
Remember
A small inference slowdown halved vision-language-action model accuracy, so for robot control, latency directly affects accuracy.
An onboard Jetson Thor drains the battery; the same chip on wall power nearby works well as an offload target.
The article doesn't quantify the network side of the tradeoff (latency, bandwidth, link failures).
Google Research has released four connected frameworks for turning short AI video clips into coherent videos several minutes long. None of them changes the underlying models. They are agent layers running on top of Gemini, Google's multimodal LLM, and 1, its video generator. Google demonstrates the stack with a ten-minute film.
The problem
Video diffusion models make good clips lasting a few seconds. Chaining clips into a story breaks down: costumes and rooms drift between shots, an artifact in an early image spoils every clip generated from it, and when the final cut is bad it's hard to tell which of many prompts caused it. Google's approach is to treat quality as something optimized at generation time, through critique-and-retry loops, rather than by retraining.
The four pieces
AI video co-director picks the creative direction. A 2, an algorithm that balances trying new options against reusing proven ones, chooses a creative strategy, a story structure and a visual style. That choice is written into the prompts of every sub-agent: storyboard, keyframes, video, audio. A multimodal LLM judge then scores the finished cut on the same three axes. Because the score is split by axis, the system learns which choice was the weak one.
3 (Continuity-Aware Narratives via Visual Agentic Storyboarding) keeps a persistent visual memory of characters, locations and the state of objects. When the story returns to a place or person, it reuses the stored reference instead of regenerating it. In a museum-heist example, plain Gemini-3.1-Pro changed the stolen gem and the room layout, and the earlier AutoStudio system lost the thief's cap. CANVAS kept both consistent.
4 generates video segment by segment, with a memory of earlier segments. For each segment it decides between extrapolating, which moves the story somewhere new, and interpolating, which anchors the segment to characters and sets it has already stored.
5 (Video Quality Question Answering) fixes defects without touching pixels or model weights. It asks a 6 targeted questions about the video, turns the answers into feedback for rewriting the prompt, and generates again. To keep repeated rewrites from drifting off the brief, a rater scores every attempt against the original prompt and keeps the best one.
Evidence
The team built three benchmarks:
GenAD-Bench: 400 ad scenarios across 50 fictional brands
HardContinuityBench: long gaps before scenes and props reappear
LVBench-C: 120 scenarios where key assets vanish for at least 10 segments
The post's only hard number is a peak score of 81.4 on GenAD-Bench, given with no baseline. The other results are described only as "significant" or "notable" gains, with details left to the papers. The post also gives no figures for how many generations or how much time the loops take.
Google Research has published a suite of four frameworks for making long, coherent videos out of models that natively produce short clips. All four are orchestration layers: they sit on top of Gemini, Google's multimodal LLM, and 1, its video diffusion model, and change nothing inside those models. The pieces are AI video co-director (accepted at COLM 2026), 2 (accepted at EMNLP 2026), 3 and 4. Google says the combined stack produces minutes-long videos, including a ten-minute film, in which characters and settings stay consistent.
The problem
Current pipelines chain separate modules, each with its own hand-written prompt. Google names four ways this goes wrong:
Semantic drift: small changes in costume or scenery from shot to shot.
Cascading failures: an artifact in an early asset spoils every video generated from it.
Feature drift: entities slowly change over time.
Content collapse: the story stops moving forward.
The team frames the core difficulty as a credit-assignment problem. When the finished video is bad, it's hard to work out which of dozens of upstream prompts caused it. Its answer is to treat quality as a test-time objective: generate, critique, adjust and generate again, rather than retraining anything.
Pillar 1: AI video co-director, which picks the creative direction
The co-director treats storytelling as a global optimization problem. An Orchestrator Agent uses a 5 (MAB), an algorithm that balances trying new options against reusing ones that scored well, to choose a configuration along three axes:
creative strategy (the intent, e.g. informational)
narrative mode (the story structure, e.g. vignette)
aesthetic archetype (visual tone and cinematography)
That choice is injected into the system prompts of every sub-agent below it, so the whole pipeline works toward one vision. A Pre-Production Agent writes a scene-by-scene storyline and storyboard. A Production Agent then runs three specialists: a Keyframe Agent fixes how characters and scenes look, a Video Agent adds motion, and an Audio Agent adds voiceover and score. Finally, a multimodal LLM judge scores the finished cut on the same three axes. That factored reward goes back to the bandit, so the system can tell whether the strategy, the structure or the aesthetic was the weak part. This is how it attacks the credit-assignment problem. Google says the design is model-agnostic. Running on Veo means outputs carry 6, Google's invisible watermark. Google also notes that clips which are each safe on their own may still need a safety classifier run over the combined video.
Pillar 2: CANVAS, a persistent visual memory
CANVAS (Continuity-Aware Narratives via Visual Agentic Storyboarding) keeps structured records of characters, locations and object states as the story progresses. For each new shot it either retrieves an existing visual anchor from memory or creates a new one. The aim is that a character looks the same and a room keeps its layout when the story comes back to it, even after a detour.
The post illustrates this with a museum-heist sequence. Gemini-3.1-Pro on its own changes the stolen artifact and shifts the room layout. AutoStudio, an earlier multi-agent system, loses the thief's cap and lets the background drift. CANVAS keeps all of them consistent. The comparison is qualitative. Google doesn't give continuity scores in the post.
Pillar 3: A²RD, which generates video segment by segment
A²RD is an autoregressive architecture: it generates video one segment at a time, with each segment conditioned on what came before. It keeps a multimodal video memory of the context and motion of earlier segments. Each segment goes through a four-step loop: retrieve from memory, synthesize the segment, refine it, update the memory.
The key decision is the generation mode for each segment:
Extrapolation pushes the story into new material.
Interpolation pins the segment to entities and settings already in memory.
As an illustration (not from the article): a character leaves a café in segment 3 and walks into a new street, which is extrapolation. In segment 15 she comes back to the café, and A²RD switches to interpolation, pulling the café and her costume from memory instead of reinventing them. Google demonstrates the approach with a ten-minute film.
Pillar 4: VQQA, which fixes defects through the prompt
VQQA (Video Quality Question Answering) is a black-box prompt optimizer. It needs no access to model weights, and Google contrasts it with test-time methods that are either computationally expensive or need that white-box access. For a given prompt it generates targeted visual questions. A 7 (VLM) answers them about the generated video, and its answers become "8": natural-language feedback that the system uses to rewrite the prompt before generating again. It never edits pixels.
Iterative rewriting tends to drift away from what the user asked for. To guard against that, VQQA uses Global Selection: a VLM rater scores every candidate along the trajectory against the original, unedited prompt, and the best one wins, not simply the last. Google's examples fix attribute-binding errors, where a property attaches to the wrong object. One is a mylar-balloon texture on a strict cube shape. Another is a violinist and pianist who swapped instruments mid-performance.
Results
The team built three benchmarks:
GenAD-Bench: 50 fictional brands with four products each, 400 advertising scenarios. Using fictional brands avoids copyright problems and stops models leaning on what they already know.
HardContinuityBench: multi-shot storyboards generated with GPT-5.2, with long gaps before scenes reappear, frequent costume changes and props that change state.
LVBench-C: 120 text-only scenarios. Key assets must be absent for at least 10 segments, then return with realistic changes.
The only concrete number in the post is the co-director's peak quality score of 81.4 on GenAD-Bench, given without a baseline. Everything else is directional:
better story consistency on ViStoryBench
"significant" continuity gains on ST-Bench and HardContinuityBench (CANVAS)
better character and environment consistency on VBench-Long and LVBench-C (A²RD)
"notable absolute quality gains" on T2V-CompBench, VBench2 and VBench-I2V (VQQA)
Google points readers to the individual papers for numbers and baselines.
What to weigh
Two of the three new benchmarks were built partly with frontier LLMs (Gemini 3 Pro, GPT-5.2), and the post doesn't say how outputs were scored, how much the loops cost in generations or latency, or how the four systems perform when combined rather than separately. The approach also bets that critique from a judge model is reliable enough to optimize against. If the VLM judge has blind spots, the loop will optimize toward them. Next, Google plans human-in-the-loop workflows, where people steer the creative choices and the agents handle continuity.
Remember
The gains come from orchestration, memory and judge-driven retries, not from retraining Gemini or Veo.
Scoring each creative axis separately lets the system tell which upstream choice caused a bad result.
Scoring every candidate against the original prompt stops iterative prompt rewriting from drifting off the brief.
NVIDIA has released NV-Reason-CT, an open 1 for 3D computed tomography (CT) scans of the chest and abdomen. It writes structured reports, produces step-by-step 2 reasoning in a radiologist's style, and answers follow-up questions. Weights and training recipes are public. NVIDIA calls it a research foundation to fine-tune, not a clinical product.
The problem
A single CT study can contain 300–600 slices. Tumours and fluid collections only make sense as 3D shapes. Most VLMs see 2D token grids, so CT usually gets fed in as a few separate slices. MedGemma 1.5, for example, takes up to 85. That throws away the continuity between slices.
How it works
The model joins a 3D 3, adapted from Primus and initialised from Colipri, a pretrained CT encoder, to the small 4 language model. Each scan is resampled to a 192³ voxel cube at 2 mm resolution and cut into 8×8×8 patches. That gives 24³ = 13,824 vision tokens, all passed to the language model without pooling. Each token carries its (x, y, z) coordinate, and 5, a rotary position encoding over three axes, lets every layer know which tokens are spatial neighbours.
Training has two stages:
Supervised fine-tuning: about 550,000 question-answer examples, plus radiologist reasoning dictations, LLM-generated synthetic reasoning traces, and refusal examples, drawn from 6, NIH datasets and CancerVerse.
Reinforcement learning with 7: the reward scores diagnostic accuracy per anatomical region rather than as one global score. NVIDIA says this improves calibration across the 59 findings in its ontology.
Results
On CT-RATE, a public chest CT benchmark with 18 labels, the model was asked plain yes/no questions with no classification head. It reached 0.614 macro-F1 and 0.871 8. The best baseline, VoxelFM, scored 0.581 and 0.870, and MedGemma 1.5 scored 0.303 F1. So it leads clearly on F1, while its ranking quality (AUROC) is tied with a pure image classifier.
Caveats
NVIDIA says it is the first open model to do well at both classification and report generation. But it gives no report-generation metrics, only favourable reviews from NIH radiologists. CT-RATE is also part of its training data, and no results on other hospitals' data are shown. The clinical study cited for time savings tested the earlier chest X-ray model, NV-Reason-CXR, not this one. No figures for inference cost at about 14k vision tokens per scan are reported.
What was released
NVIDIA has released NV-Reason-CT, an open 1 for 3D computed tomography (CT) scans of the chest and abdomen. It does three jobs. It writes structured diagnostic reports. It produces step-by-step 2 reasoning in the style of a radiologist. And it answers follow-up questions over several turns. Weights are on Hugging Face, and inference scripts, training configs and post-training recipes are on GitHub. NVIDIA is clear that this is a research foundation for others to fine-tune, not a cleared clinical product or an autonomous diagnostic system.
It follows NV-Reason-CXR, NVIDIA's chest X-ray reasoning model, which uses the same training approach. The multireader clinical study the post cites, accepted at the RSNA 2026 radiology meeting, found time savings with no loss of accuracy. That study evaluated the X-ray model, not this one.
Why CT breaks standard VLMs
One abdominal CT study can contain 300–600 axial slices. Masses, effusions (fluid collections) and infiltrates only take on clinical meaning through their 3D shape, extent and density. A standard VLM treats an image as a 2D grid of tokens. Feeding it a CT as a stack of independent frames throws away the relationships between slices. For example, MedGemma 1.5 takes up to 85 axial slices.
The post names two further gaps. Models often output a label without saying why, which a radiologist can't audit. And most can't hold the iterative back-and-forth a radiologist uses to reconsider a finding.
The architecture
NV-Reason-CT pairs the 3 language model with a 3D 4. The ViT is adapted from Primus and initialised with weights from Colipri, a pretrained CT encoder. All weights are then retrained end to end.
The input pipeline works like this:
The CT volume is resampled to a 192×192×192 voxel cube at 2 mm per voxel in every direction. That covers about 38 cm per side.
It is cut into non-overlapping 8×8×8-voxel patches, giving 24×24×24 = 13,824 vision tokens.
Most VLMs merge or pool vision tokens to save context. This one passes every token to the language model, along with its (x, y, z) grid coordinate.
The language model uses 5, a multi-axis rotary position encoding. It encodes each token's position along three axes, so attention in every layer can tell that two tokens are vertical neighbours, not just adjacent in the sequence.
This is a deliberate trade. Each scan uses nearly 14k tokens of context before any text, in exchange for keeping spatial structure intact through the whole model. The post doesn't report memory use or latency.
Training recipe
Stage 1: supervised fine-tuning (SFT). Training mixes structured reports, chain-of-thought dictations from radiologists and general visual question answering (VQA). The dictations record what the radiologist checks in each anatomical region, which findings they consider significant and which alternative diagnoses they weigh. The curriculum has about 550,000 structured question-answer examples. They cover section-level anatomy, side-specific (left/right) and localised findings, severity, and yes/no abnormality questions. Refusal examples for invalid prompts and mismatched image-text pairs are included for robustness. Sources are 6, NIH CT datasets and CancerVerse. These are supplemented with synthetic reasoning traces generated by large language models, using expert annotations as grounding examples. A curated ontology of 30 chest and 29 abdominal abnormalities, such as lung nodules, pneumothorax, liver lesions and kidney cysts, guides both training and evaluation.
Stage 2: reinforcement learning (RL) with 7 (Group Relative Policy Optimization). The reward scores how accurately the model identifies abnormalities and diagnoses. Instead of one global score, the reward is computed per anatomical region. The intent is to stop the model from earning credit for getting common lung findings right while missing rarer abdominal ones, and NVIDIA says this improves calibration across the full set of findings. The idea is to learn the shape of the reasoning from a limited set of expert traces first, then reinforce correctness without needing annotated reasoning for every case.
Results
On CT-RATE classification (18 labels, one fixed threshold for all), NV-Reason-CT was prompted directly with yes/no questions, with no classification head or task-specific tuning:
Model
Type
Macro-F1
Macro-AUROC
NV-Reason-CT
Native 3D generative VLM
0.614
0.871
VoxelFM
3D image-only pretraining
0.581
0.870
Pillar-0
3D contrastive
0.544
0.861
ClinFusion-8B
Fused 2D/3D generative
0.442
n/r
CT-CLIP
3D contrastive
0.398
0.733
Merlin
3D contrastive
0.358
0.662
MedGemma 1.5
Up to 85 slices
0.303
n/r
The F1 lead over the strongest baseline is real: 0.614 against 0.581. On 8, which measures ranking quality independent of threshold, NV-Reason-CT is essentially tied with VoxelFM (0.871 vs 0.870), a model that only does image classification. The clearer gap is against slice-based and generative rivals. MedGemma 1.5 scores 0.303 F1, roughly half.
NVIDIA's distinctive claim is that this is the first single open model to be competitive at both CT classification and report generation. The post gives no numbers for report generation. The qualitative evidence is favourable reviews from NIH radiologists and a quote from Baris Turkbey, an NIH senior clinician, saying that seeing the model's thought process is what makes its findings trustworthy.
What the output looks like
The sample output reads like a dictation. It notes a left port catheter with its tip in the superior vena cava. It reviews the airways, flags a large right-lower-lobe mass with partial lung collapse plus enlarged lymph nodes, calls the pattern concerning for malignancy with nodal spread, and lists nonspecific ground-glass opacities and lytic spine lesions. Its <think> block, though, is a single generic sentence. The detailed reasoning appears in the visible answer. Practitioners trying the snippet should note that it builds text but then passes an undefined prompt to the processor.
Caveats and open questions
In-distribution evaluation. CT-RATE is both a training source and the main benchmark. The post doesn't say whether baselines saw similar data, or how the model does on external hospitals' scans.
Report quality is unquantified. There are no report-generation metrics and no reader study for CT, only reviews.
Faithfulness of reasoning. The post treats chain-of-thought as auditable. It doesn't test whether the stated reasoning actually drives the labels, or whether part of the reasoning style was learned from synthetic LLM-distilled traces.
Cost. There are no figures for inference time or memory at nearly 14k vision tokens per volume, and the 2 mm resampling may blur small lesions.
NVIDIA presents it as one part of a family that also includes NV-Generate-CTMR (synthetic CT/MRI) and NV-Segment-CTMR (organ and lesion segmentation).
Remember
It keeps all 13,824 3D vision tokens and tags each with a 3D position, rather than compressing them.
Its F1 lead on CT-RATE is clear, but its AUROC (0.871) only ties VoxelFM, an image-only classifier (0.870).
No report-generation metrics or CT reader study back the claim that it does both classification and reports well.
