In addition to the library bindings, we have a small library of tools (workspace for view/edit, scratchpad for persistence) and making your own is registering a Go function. And in recent weeks, I added the Vision and Qwen support, as ds4 added them.

Even if you don't use the Go library, the ds4go binary makes it really easy to download the libraries off of HuggingFace with a TUI available vie Homebrew.

Here's some TUI toy screenshots, sorry I still haven't released that code; it's of different quality than the others. [3]

EDIT: add ds4go TUI screenshot gist [4]

[1] https://github.com/NimbleMarkets/ds4/releases/tag/v0.8.20260...

[2] https://github.com/nimblemarkets/ds4go#install

[3] https://gist.github.com/neomantra/ae47422c8daf7a458212c93992...

[4] https://gist.github.com/neomantra/40180ade13df93290250ce8c6d...

The project GitHub page is a much better introduction for the hn crowd.

I'm also looking into expanding the protocol and the engine to support various steering techniques.

https://github.com/simoneiacomino/xenolith

Now Ive been running qwen 3.8 flash next for more than a week and it’s doing great, really fast and super long context windows. Sometimes the model is behaving stupidly by not remembering something I said earlier but it could be also a problem from the agentic AI harness. Im using oh my pi but Im wondering what people are using ds4 with here ?

https://youtube.com/@antirez

ds4 is referring to “dwarfstar” “4” and references DeepSeek V4 most of the time

but its model agnostic-ish

and benchmarks compared to what? what do these large MoE models typically get in tokens per second?

I’m garnering this is just an easier way to load large models per expert on consumer hardware? as opposed to the hackier solutions?

I’m intruiged. Note that the blogpost says 64gb Macs are good minimums while the github says 96gb is a minimum

  small native inference engine optimized first for DeepSeek V4 Flash (including the experimental vision model), DeepSeek V4.1 Flash (Metal, and text inference on CUDA), and additionally GLM 5.2 and 5.3, GLM 5.3 Flash and DeepSeek V4 PRO, and Qwen3.8 Flash Next (Metal and CUDA)

For our mere mortals that were kids not long ago and can't really believe we've got our hands on a x090 series targeting Qwen3.8 27b, https://github.com/noonghunna/club-3090 is the way to go.

I'm maintaining a web frontend for this, trying to at least. You can follow it here: https://github.com/gchamon/club-3090-server

Tensorfold is getting 100%+ speed increases on both prefill and decode for models like Qwen 27B. oMLX has followed them and have had similar improvements in the past week.

There's lots of different techniques like letting CPU help with prefill, DFlash specualtive decoding etc.

I'm really excited for this as I'll be receiving an M5U in about a month. Expect to be running Qwen 4 27B or Flash (it's a 96gb machine), and they may come close in performance to DS 4/4.1 Flash, and should be able to hit 100 tps. Local is really becoming viable, especially considering that GPT 6.1 has been running at 20ish tps the past week.

From the github repo it seems like you really don't need a big Mac with huge amounts of RAM but SSD is sufficient.

If this is anywhere near 50 TPS, that would be a game changer in the personal LLM space!

The other inference engine are also model by model with a huge switch statement deciding which part to load for which model or are they very generic?

    Metal, the primary target, on Macs with 96 GB or more. Smaller machines can use SSD streaming. SSD streaming is also needed in order to run very large models such as full GLM 5.x (not Flash) on 128GB systems

It must be more a working template for the biggest use cases, without trying to cover every possible setup.

this is such a great quote about building software in the AI age. given that everyone has a different use case, the best way for open source software to be built is to build it for the general use case and specialization could be done afterwards.

https://github.com/alainnothere/llama.cpp/commits/disk-cache...

And you know it's load bearing each of the load baerings parts that bear some load and load a bear... you fight a bear because it took a load... or something like that...

I mean, I've been using local models on vscode right next to frontier models with ollama for a few months. What's new?

What are we going to name the company, how about Dwarfism 2.0? What happened to 1.0 Jared?