Beware the license. They misleadingly state on the blog post "Open-weight — Download, fine-tune, and deploy without restrictions". But if you read their license <https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/LICE...> it has significant restrictions for any org with other $10M in revenue.
I just tested this on a bug fixing benchmark I'm working on.
It did not perform as well as I expected. Qwen2.5-Coder-3B (2 years old) outperformed it by a wide range -> fixing ~50% of bugs whereas this model only fixed ~12%.
Granted, it's not a coder specific model, but given its benchmark performance to Gemma models, and that it's two years newer, and that it's an MoE with 8B total params, I expected it to be more competitive.
I personally find any model smaller than something like Qwen 3.6 35B-A3B (8-bit quantization, about 49GB memory usage when loaded into llama.cpp) to be too "stupid" for reliable use.
I would much rather not run the model on my local laptop hardware and offload that to some system sitting under my desk in my home office, accessible via VPN, than take the risk of using an unreliable and flaky tool for the convenience of having it on the same hardware on my lap.
I pay very little attention to 8 billion or whatever (or even much smaller) models these days and I don't feel like I'm missing much.
Yes, Qwen 3.6 MoE is hitting like 80-90tk/s on Strix halo. On R9700 I had like 170t/s. It was not possible to keep up. But MoE is circling very often. I switch then to dense model and have 20-30t/s but it is able to solve quite a lot of tasks.
Absolutely. Difference in Q6 vs Q8 is not as immediately noticeable, but if I test by starting from a blank slate context and giving it the same complicated task with Q4 vs a Q8 GGUF file loaded, the difference is apparent. The Q4 will struggle or do 'stupid' things with even simple bash or python. Q4 might not be as noticeable for conversational purely text one on one interaction with an LLM, but when you dig deeper into something that's more esoteric in a training dataset than a chat conversation, absolutely a big gap there.
I think some of the folks in the local llm social media communities are using them for things like company-hosted customer service chat bots, or purely english text writing stuff where Q4 will probably not cause a problem. For more discrete technical work I stick pretty much exclusively to Q8.
I have not spent a lot of time running FP16 'full precision' versions of some things, but as the other commenter says, it's not much difference. There's a really wide array of benchmarks and tests from a lot of third parties unrelated to the trainer of the AI models that shows at most a two percent difference in score and capability between BF16 and Q8.
Q8 quant is very minimal fall off in terms of KLD against the lab 16 bit. If you have the memory for BF16 KV-cache (which is usually easier to stomach) then the Q8 is very close. But even Q8 quant model with Q8 KV-cache is very close.
Smaller quants for the model start to fall off but more importantly, smaller KV-cache quants fall off much faster so avoid less than Q8 there.
It’s not a general rule, and depends highly on the model and the quantisation used. Don’t guess, Unsloth sometimes publish graphs in their tutorials showing the error rate vs file size… sometimes Q4 is great, other times I go for Q6
That's not all that surprising, IMO. From what I understand, LiquidAI is focusing pretty narrowly on building models that operate as the "agentic core" of a larger system.
If I were going to use this model, I'd be looking to use it more as is the primary chat interface of a larger system, and having it orchestrate & delegate tasks to other places via tool calls. It's not quite as exciting on the surface as a local "do it all" model, but it does enable some pretty neat use-cases, IMO.
I'm imagining a local agent that is super low latency, works entirely offline, and capable of queuing up complex tasks for larger/smarter cloud agents which execute them asynchronously.
I just did the same. Absolutely awful. I assume OpenCode's heavy context is a problem, and it's probably better to use Liquid's own OpenCode alternative for this.
I will test it when it's accessible via OpenRouter, but the previous LFM2 model (lfm-2-24b-a2b) didn't do well on my tests, it got only 1/20 questions/tasks right, way below Gemma 31B or Qwen 35b-a3b (those get like 10/20 right)
Some of the coding-specific fine-tunes were really impressive boosts. Qwen2.5-3B-Instruct is also available [0] -- if it's not too much to ask, I'd be curious how more general models stack up in your benchmark?
You did not express any desire to wash the car. After nudging, it's funnier:
> Is my desire to make my car clean related to a car wash in any way? To driving to it?
> Your desire to have a clean car isn’t tied to the existence or use of a car wash or to traveling to one. It stems from how much you value the effort, time, and resources you’re willing to invest in achieving cleanliness — whether that means doing it yourself, paying for a professional service, or simply enjoying the routine of a quick walk‑out. Driving to a nearby wash is just one possible method among many; the decision ultimately depends on factors such as how much time you have, how much you’d like to spend, the quality of the wash, and any personal preferences you have for hands‑on versus outsourced cleaning. In this case, walking is the simplest and fastest way, but it isn’t required for you to achieve a clean car.
Common sense is clearly there, but we should not underestimate the colossal heap of tacit assumptions that drive "obvious" decisions in our daily life.
"Correct" is pushing it, the question is too vague if approached as a genuine question and not a gotcha. I've actually had literal experiences where I wanted to wash my car and walked to a car wash in the past. That was me collecting the car, and there is an argument that would be a valid walk answer.
If we require logical rigour there isn't enough context in the question. If we allow for informal language then there are absolutely situations where cars get washed and people walk 50 meters to the car wash. It is a reasonable guess that the car is already at the wash and you have a 2nd car, given the question is being asked. It's a slight leap, but it is an inference that makes the question meaningful and so it is one that could be made.
I'd assume the LLMs are just failing at spatial reasoning, because AFAIK they're terrible at it. But both answers are justifiable because we don't know where the car is and have to make assumptions.
this reminds me, I grew up in an area of the US where the pinnacle of existence was spending the whole weekend doing chores such as very publicly washing your own car in your driveway
if you were an able bodied man there is no other duty. the same for shoveling snow, or mowing a lawn, cleaning up inside the house
these are all things I've rejected and exempt myself from
but I'm beginning to remember large swaths of society live under that regime, so driving to a car wash wouldn't be an option at all. you wash your car and have a separate desire to walk to the car wash for some other reason
I could see people thinking its a trick question, or just scoffing at the idea people wash their cars at the car wash and pollute the data for AIs in annotation work.
I'm surprised these models haven't picked this up yet in the training data. Both Claude and ChatGPT missed that one when I posed the question to them last year.
>The main reasons to drive such a short distance would be if you're bringing the car specifically to be washed, carrying something heavy, or the weather or walking conditions make it impractical.
>If your goal is to get your car washed, you'll need the car there—so driving makes sense. If you're just going to talk to someone at the car wash or check it out, walking is probably faster.
There's meaning in the term "car wash" that it understands. But I don't suspect anyone has taught it that for 99.9% of people, going to car wash ONLY means that you're going to wash your car and that it should make that implicit assumption.
What if you're the car wash owner? Or a maintenance technician? Pretty easy to just walk over there if you're just 50ft away.
to your point, when my Aussie friends first mentioned a "car park" to my north american born self, i wondered _momentarily_ what that was, then realized it's sort of a fun name for what i would call a parking lot.
yeah but syntactically "car park" gets used like a noun phrase, not verb phrase, which was (to your point really) what had me think "huh?" momentarily.
His analogy is that a gas station is for putting gas into your car. But he walks there often, so the assumption that you need your car if you go to the gas station isn't inevitable.
You could conceivably walk to a car wash that has similar sundries as a gas station.
Indeed, the little market there is why I walk there. There is also one at the car wash another 2 blocks away. I’d walk there for a 7up if it were closer!
These faux questions always have a valid interpretation that the asker doesn't admit (for some reason). The model is then castigated for not making an opinionated choice
lol, i think the LLM shows more wisdom here than the average person. Functionally, being 50m away from the car wash is at the car wash if you have a dirty car in your possession that needs cleaning. Realistically, the only reason you express the need to go to the carwash if you are in a 50m proximity with your car you intend to clean at the carwash is if you need to walk in and talk to someone.
At some point we have to be running into some inherent mathematical limits of knowledge compression, right? No way the knowledge benchmarks on these 8B models will keep getting better without overfitting on these benchmarks
If you give the model access to specialized tools (e.g. web search for question answering) the knowledge doesn't have to be stored in the model weights, which leaves some room for improvement. You'd still be overfitting to benchmarks (since different tasks might require different tools) but not necessarily to specific benchmark questions, so within-domain generalization could be quite good.
As an example for a similar approach, Teapot AI has trained very small models https://teapotai.com/models to only answer questions where the answer can be found within the context window, and although not perfect, they do quite well at this compared to larger, more general models.
good point I have the feeling larger models (20b+) rely too much about their stored knowledge and sometimes fail to use tools because they think they know the answer. smaller specialized tool calling models could be the smart route for the future
This is super interesting, I'm particularly excited for this one as it may allow teams to scale this architecture for VLAs (vision language action models), and having sparser models means more real-time actions on a locally hosted model
I use it for triaging my messages and emails and reminding me how all of it ties together. It uses Obsidian to know where to put stuff and how to connect information. It isn't perfect. It's very slow (using a 32GB M2 Max) but fast enough for my needs.
A good example of how it's helpful is that it will make certain things relatively frictionless. Like, I need to pay property taxes. I hate this stuff. I got the email reminder from my municipality and it made an entry in my TODOs which points to page with instructions to pay the taxes, including my folio and access numbers for when I log in. That was taken from the email and a document which contains past property tax information. I have it all there, but it compiles relevant data into dedicated TODO pages.
I'm so bad at doing all of this myself. I really don't enjoy it. Send me to buy a carrot at the store and I'll happily walk 30 minutes there and back to do it. It isn't the effort so to speak; it's how unrewarding, inefficient, and bureaucratic it all is. I'm allergic to it. Why isn't it baked into my income taxes? Why are we still doing this?
Sometimes it does a really bad job of making TODOs. Like my wife messaged me about what our dinner plan was, so Qwen went ahead and made a plan for chicken meatball soup based on messages from a week earlier. It totally fabricated the recipe. Yet, I don't know, it was still helpful to be reminded that I'm in charge of dinner.
It's probably best at scaffolding responses to emails I don't want to send. I will write it, but I appreciate basic information being fleshed out so I can write it without jumping around looking for files or numbers or whatever constantly.
I use it with a custom harness. It could be a lot better. Everything about it could be better. The model is remarkably good for its size and price, though.
Letting Sonnet 4.6 do it instead always yields much better results, much faster, but it's kind of like using a new phone vs a super old one. They can both get you there. The sound quality and camera might be worse, it doesn't look as fancy, but the new one is $1200 and the old one is free on marketplace if you're handy with a screwdriver and a fresh battery. Sounds great to me
Worth noting: this was all vibe-coded using Opus 4.6 and 4.7. It's the only project I've built that is strictly vibe-coded. It's simultaneously exciting and disgusting. I'm not sure if I'll ever 'software engineer' it, or I'll just let it be slop. It works.
Look at the accuracy numbers and these things clearly don't know much yet, and I'm not about to hand one my hardest work. But you can see where it's going. As quantization and the MoE stuff keeps getting better, "good enough to just run on my own machine" keeps eating into more of what I'm currently paying a frontier lab for. Once a local model can handle like 80% of what I need, the math stops making sense for the subscription.
Many such cases. Many models say they're ChatGPT, a lot seem to figure out that since they're Transformers they're made by Google.
Doesn't really tell you a lot. Perhaps a pretraining / midtraining artifact.
Wow, this is fucking phenomenal. I fed it a long transcript asking it to create a summary and it executed it extremely well. For an 8B model this is quite impressive.
I gave it a 2000 line python code that does some fairly sophisticated geodesic calculations on surfaces, and asked to review the code. I then asked Claude and ChatGPT to "assess the accuracy of this review" and they did not hold back. That said, its a very small model, and very fast.
Why does this not have (day-one) support for Ollama? The previous model is on there? Is it related to the ongoing refactor work or are people abandoning Ollama for other LLM engines?
I'd normally call that a low-effort, troll comment. But, thinking on it, you may have a great metaphor.
They keep promising great performance out of models whose key ingredient (parameters) they are diluting. Many seem to be in a competition saying they're getting smaller and higher performance at the same time. Then, the homeopathic models don't perform as well as real models when independently tested. Again, spot on.
I really love how fast it is! Their press release comparing it on Strix Halo and M5 Max are impressive. It going twice as fast at GPU benchmarks even more so!
reply