- cross-posted to:
- privacy@lemmy.ml
- cross-posted to:
- privacy@lemmy.ml
Hello!
As a handsome local AI enjoyer™ you’ve probably noticed one of the big flaws with LLMs:
It lies. Confidently. ALL THE TIME.
(Technically, it “bullshits” - https://link.springer.com/article/10.1007/s10676-024-09775-5
I’m autistic and extremely allergic to vibes-based tooling, so … I built a thing. Maybe it’s useful to you too.
The thing: llama-conductor
llama-conductor is a router that sits between your frontend (OWUI / SillyTavern / LibreChat / etc) and your backend (llama.cpp + llama-swap, or any OpenAI-compatible endpoint). Local-first (because fuck big AI), but it should talk to anything OpenAI-compatible if you point it there (note: experimental so YMMV).
I tried to make a glass-box that makes the stack behave like a deterministic system, instead of a drunk telling a story about the fish that got away.
TL;DR: “In God we trust. All others must bring data.”
Three examples:
1) KB mechanics that don’t suck (1990s engineering: markdown, JSON, checksums)
You keep “knowledge” as dumb folders on disk. Drop docs (.txt, .md, .pdf) in them. Then:
>>attach <kb>— attaches a KB folder>>summ new— generatesSUMM_*.mdfiles with SHA-256 provenance baked in- `>> moves the original to a sub-folder
Now, when you ask something like:
“yo, what did the Commodore C64 retail for in 1982?”
…it answers from the attached KBs only. If the fact isn’t there, it tells you - explicitly - instead of winging it. Eg:
The provided facts state the Commodore 64 launched at $595 and was reduced to $250, but do not specify a 1982 retail price. The Amiga’s pricing and timeline are also not detailed in the given facts.
Missing information includes the exact 1982 retail price for Commodore’s product line and which specific model(s) were sold then. The answer assumes the C64 is the intended product but cannot confirm this from the facts.
Confidence: medium | Source: Mixed
No vibes. No “well probably…”. Just: here’s what’s in your docs, here’s what’s missing, don’t GIGO yourself into stupid.
And when you’re happy with your summaries, you can:
>>move to vault— promote those SUMMs into Qdrant for the heavy mode.
2) Mentats: proof-or-refusal mode (Vault-only)
Mentats is the “deep think” pipeline against your curated sources. It’s enforced isolation:
- no chat history
- no filesystem KBs
- no Vodka
- Vault-only grounding (Qdrant)
It runs triple-pass (thinker → critic → thinker). It’s slow on purpose. You can audit it. And if the Vault has nothing relevant? It refuses and tells you to go pound sand:
FINAL_ANSWER:
The provided facts do not contain information about the Acorn computer or its 1995 sale price.
Sources: Vault
FACTS_USED: NONE
[ZARDOZ HATH SPOKEN]
Also yes, it writes a mentats_debug.log, because of course it does. Go look at it any time you want.
The flow is basically: Attach KBs → SUMM → Move to Vault → Mentats. No mystery meat. No “trust me bro, embeddings.”
3) Vodka: deterministic memory on a potato budget
Local LLMs have two classic problems: goldfish memory + context bloat that murders your VRAM.
Vodka fixes both without extra model compute. (Yes, I used the power of JSON files to hack the planet instead of buying more VRAM from NVIDIA).
!!stores facts verbatim (JSON on disk)??recalls them verbatim (TTL + touch limits so memory doesn’t become landfill)- CTC (Cut The Crap) hard-caps context (last N messages + char cap) so you don’t get VRAM spikes after 400 messages
So instead of:
“Remember my server is 203.0.113.42” → “Got it!” → [100 msgs later] → “127.0.0.1 🥰”
you get:
!! my server is 203.0.113.42?? server ip→ 203.0.113.42 (with TTL/touch metadata)
And because context stays bounded: stable KV cache, stable speed, your potato PC stops crying.
There’s more (a lot more) in the README, but I’ve already over-autism’ed this post.
TL;DR:
If you want your local LLM to shut up when it doesn’t know and show receipts when it does, come poke it:
- Primary (Codeberg): https://codeberg.org/BobbyLLM/llama-conductor
- Mirror (GitHub): https://github.com/BobbyLLM/llama-conductor
PS: Sorry about the AI slop image. I can’t draw for shit.
PPS: A human with ASD wrote this using Notepad++. If it the formatting is weird, now you know why.
You must log in or # to comment.
Very impressive! Do you have benchmark to test the reliability? A paper would be awesome to contribute to the science.
Just bush-league ones I did myself, that have no validation or normative values. Not that any of the LLM benchmarks seem to have those either LOL
I’m open to ideas, time wiling. Believe it or not, I’m not a code monkey. I do this shit for fun to get away from my real job
I understand, no idea on how to do it. I heard about SWE‑Bench‑Lite that seems to focus on real-world usage. Maybe try to contact “AI Explained” on YT, he’s the best IMO. Your solution might be novel or not but he might help you figuring that. If it is indeed novel, it might be worth it to share it with the larger community. Of course, I totally get that you might not want to do any of that. Thank you for your work!
neat but is this privacy related?
Yes, because making locally hosted LLMs actually useful means you don’t need to utilize cloud-based and often proprietary models like ChatGPT or Gemini which Hoover up all of your data.
^ exactly that. Thank you!
Yes. Several reasons -
-
Focuses on making LOCAL LLMs more reliable. You can hitch it to OpenRouter or ChatGPT if you want to leak you personal deets everywhere, but that’s not what this is for. I built this to make local, self hosted stuff BETTER.
-
Entire system operates on curating (and ticketing with provenance trails) local data…so you don’t need to YOLO request thru god knows where to pull information.
-
In theory, you could automate a workflow that does this - poll SearXNG, grab whatever you wanted to, make a .md summary, drop it into your KB folder, then tell your LLM “do the thing”. Or even use Scrapy if you prefer: https://github.com/scrapy/scrapy
-
Your memory is stored on disk, at home, on a tamper proof file, that you can inspect. No one else can see it. It doesn’t get leaked by the LLM any where. Because until you ask it, it literally has no idea what facts you’ve stored. The content of your KBs, memory stores etc are CLOSED OFF from the LLM.
-
Based AF. Can anyone more knowledgeable explain how it works? I am not able to understand.
Hell yes I can explain. What would you like to know.
As I understand it, it corrects the output of LLMs. If so, how does it actually work?
Good question.
It doesn’t “correct” the model after the fact. It controls what the model is allowed to see and use before it ever answers.
There are basically three modes, each stricter than the last. The default is “serious mode” (governed by serious.py). Low temp, punishes chattiness and inventiveness, forces it to state context for whatever it says.
Additionally, Vodka (made up of two sub-modules - “cut the crap” and “fast recall”) operate at all times. Cut the crap trims context so the model only sees a bounded, stable window. You can think of it like a rolling, summary of what’s been said. That summary is not LLM generated summary either - it’s concatenation (dumb text matching), so no made up vibes.
Fast recall OTOH stores and recalls facts verbatim from disk, not from the model’s latent memory.
It writes what you tell it to a text file and then when you ask about it, spits it back out verbatim ((!! / ??)
And that’s the baseline
In KB mode, you make the LLM answer based on the above settings + with reference to your docs ONLY (in the first instance).
When you >>attach <kb>, the router gets stricter again. Now the model is instructed to answer only from the attached documents.
Those docs can even get summarized via an internal prompt if you run >>summ new, so that extra details are stripped out and you are left with just baseline who-what-where-when-why-how.
The SUMM_*.md file come SHA-256 provenance, so every claim can be traced back to a specific origin file (which gets moved to a subfolder)
TL;DR: If the answer isn’t in the KB, it’s told to say so instead of guessing.
Finally, Mentats mode (Vault / Qdrant). This is the “I am done with your shit" path.
It’s all of the three above PLUS a counter-factual sweep.
It runs ONLY on stuff you’ve promoted into the vault.
What it does is it takes your question and forms in in a particular way so that all of the particulars must be answered in order for there to BE an answer. Any part missing or not in context? No soup for you!
In step 1, it runs that past the thinker model. The answer is then passed onto a “critic” model (different llm). That model has the job of looking at the thinkers output and say “bullshit - what about xyz?”.
It sends that back to the thinker…who then answers and provides final output. But if it CANNOT answer the critics questions (based on the stored info?). It will tell you. No soup for you, again!
TL;DR:
The “corrections” happen by routing and constraint. The model never gets the chance to hallucinate in the first place, because it literally isn’t shown anything it’s not allowed to use. Basic premise - trust but verify (and I’ve given you all the tools I could think of to do that).
Does that explain it better? The repo has a FAQ but if I can explain anything more specifically or clearly, please let me know. I built this for people like you and me.
That is much clearer. Thank you for making this. It actually makes LLMs useful with much lesser downsides.
God, I hope so. Else I just pissed 4 months up the wall and shouted a lot of swears at my monitor for nada :)
Let me know if it works for you
Will do.
I have no remarks, just really amused with your writing in your repo.
Going to build a Docker and self host this shit you made and enjoy your hard work.
Thank you for this!
Thank you <3
Please let me know how it works…and enjoy the >>FR settings. If you’ve ever wanted to trolled by Bender (or a host of other 1990s / 2000s era memes), you’ll love it.
Same sentiment. Tonight it run on my systems XD.
There are literally dozens of us. DOZENS!
I’m on a potato, so I can’t attach it to something super sexy, like a 405B or a MoE.
If you do, please report back.
PS: You may see (in the docs) occasional references that slipped passed me to MoA. That doesn’t stand for Mixture of Agents. That stood for “Mixture of Assholes”. That’s always been my mental model for this.
Or, in the language of my people, this was my basic design philosophy:
YOU (question)-> ROUTER+DOCS (Ah shit, here we go again. I hate my life)
|
ROUTER+DOCS -> Asshole 1: Qwen (“I’m right”)
|
ROUTER+DOCS -> Asshole 2: Phi (“No, I’m right”)
|
ROUTER+DOCS -> Asshole 3: Nanbeige (“Idiots, I’m right!”)
|
ROUTER+DOCS (Jesus, WTF. I need booze now) <- (all assholes)
|
–> YOU (answer)
(this could have been funnier in the ASCII actually worked but man…Lemmy borks that)
EDIT: If you want to be boring about it, it’s more like this
PS: If you like it, let other people in other places know about it.
Fuck yeah…good job. This is how I would like to see “AI” implemented. Is there some way to attach other data sources? Something like a local hosted wiki?
Hmm. I dunno - never tried. I suppose if the wiki could be imported in a compatible format…it should be able to chew thru it just fine. Wiki’s are usually just gussied up text files anyway :) Drop the contents of your wiki in there a .md’s and see what it does
I wanna just plug Wikipedia into this and see if it turns an LLM into something useful for the general case.
LOL. Don’t do that. Wikipedia is THE nosiest source.
Would you like me to show you HOW and WHY the SUMM pathway works? I built it after I tried a “YOLO wikipedia in that shit - done, bby!”. It…ended poorly
Yes please
Of course. Here is a copy paste from my now defunct reddit account. Feel free to follow the pastebin links to see what v1 of SUMM did. Whats in the router uses is v1.1:
########
My RAG
I’ve recently been playing around with making my SLM’s more useful and reliable. I’d like to share some of the things I did, so that perhaps it might help someone else in the same boat.
Initially, I had the (obvious, wrong) idea that “well, shit, I’ll just RAG dump Wikipedia and job done”. I trust it’s obvious why that’s not a great idea (retrieval gets noisy, chunks lack context, model spends more time sifting than answering).
Instead, I thought to myself “why don’t I use the Didactic Method to teach my SLMs what the ground truth is, and then let them argue from there?”. After all, Qwen3-4B is pretty good with its reasoning…it just needs to not start from a position of shit.
The basic work flow -
TLDR
- Use a strong model to write clean, didactic notes from source docs.
- Distill + structure those notes with a local 8B model.
- Load distilled notes into RAG (I love you, Qdrant).
- Use a 4B model with low temp + strict style as the front‑end brain.
- Let it consult RAG both for facts and for “who should answer this?” policy.
Details
(1) Create a “model answer” --> this involves creating a summary of source material (like say, markdown document explaining launch flags for llama.cpp). You can do this manually or use any capable local model to do it, but for my testing, I fed the source info straight in Gippity 5 with specfic “make me a good summary of this, hoss” prompt
Like so: https://pastebin.com/FaAB2A6f
(2) Save that output as SUMM-llama-flags.md. You can copy paste it into Notepad++ and do it manually if need to.
(3) Once the summary has been created, use a local “extractor” and “formatter” model to batch extract high yield information (into JSON) and then convert that into a second distillation (markdown). I used Qwen3-8b for this.
Extract prompt https://pastebin.com/nT3cNWW1
Format prompt (run directly on that content after model has finished its extraction) https://pastebin.com/PNLePhW8
(4) Save that as DISTILL-llama-flags.md.
(5) Drop Temperature low (0.3) and made Qwen3-4B cut the cutsey imagination shit (top_p = 0.9, top_k=0), not that it did a lot of that to begin with.
(6) Import DISTILL-llama-flags.md into your RAG solution (god I love markdown).
Once I had that in place, I also created some “fence around the law” (to quote Judaism) guard-rails and threw them into RAG. This is my question meta, that I can append to the front (or back) of any query. Basically, I can ask the SLM “based on escalation policy and the complexity of what I’m asking you, who should answer this question? You or someone else? Explain why.”
(I also created another “how much will this cost me to answer with X on Open Router” calculator, a “this is my rig” ground truth document etc but those are sort of bespoke for my use-case and may not be generalisable. You get the idea though; you can create a bunch of IF-THEN rules).
The TL:DR of all this -
With a GOOD initial summary (and distillation) you can make a VERY capable little brain, that will argue quite well from first principles. Be aware, this can be a lossy pipeline…so make sure you don’t GIGO yourself into stupid. IOW, trust but verify and keep both the source material AND SUMM-file.md until you’re confident with the pipeline. (And of course, re-verify anything critical as needed).
I tested, and retested, and re-retest a lot (literally 28 million tokens on OR to make triple sure), doing a bunch of adversarial Q&A testing, side by side with GPT5, to triple check that this worked as I hoped it would.
The results basically showed a 9/10 for direct recall of facts, 7-8/10 for “argue based on my knowledge stack” or “extrapolate based on knowledge stack + reference to X website” and about 6/10 on “based on knowledge, give me your best guess about X adjacent topic”. That’s a LOT better than just YOLOing random shit into Qdrant…and orders of magnitude better than relying on pre-trained data.
Additionally, I made this this cute little system prompt to give me some fake confidence -
Tone: neutral, precise, low-context.Rules:Answer first. No preamble. ≤3 short paragraphs.Minimal emotion or politeness; no soft closure.Never generate personal memories, subjective experiences, or fictional biographical details.Emotional or expressive tone is forbidden.Cite your sourcesEnd with a declarative sentence.
Append: "Confidence: [percent] | Source: [Pretrained | Deductive | User | External]".^ model reported, not a real statistical analysis. Not really needed for Qwen model, but you know, cute.
The nice thing here is, as your curated RAG pile grows, so does your expert system’s “smarts”, because it has more ground truth to reason from. Plus, .md files are tiny, easy to demarcate, highlight important stuff (enforce semantic chunking) etc.
The next step:
Build up the RAG corpus and automate steps 1-6 with a small python script, so I don’t need to baby sit it. Then it basically becomes “drop source info into folder, hit START, let’er rip” (or even lazier, set up a Task Scheduler to monitor the folder and then run “Amazing-python-code-for-awesomeness.py” at X time).
Also, create separate knowledge buckets. OWUI (probably everything else) let’s you have separate “containers” - right now within my RAG DB I have “General”, “Computer” etc - so I can add whichever container I want to a question, ad hoc, query the whole thing, or zoom down to a specific document level (like my DISTILL-llama.cpp.md)
I hope this helps someone! I’m just noob but I’m happy to answer whatever questions I can (up to but excluding the reasons my near-erotic love for .md files and notepad++. A man needs to keep some mystery).
EDIT: Gippity 5 made a little suggestion to that system prompt that turns it from made up numbers to something actually useful to eyeball. Feel free to use; I’m trialing it now myself
Tone: neutral, precise, low‑context.
Rules:
Answer first. No preamble. ≤3 short paragraphs (plus optional bullets/code if needed). Minimal emotion or politeness; no soft closure. Never generate personal memories, subjective experiences, or fictional biographical details. Emotional or expressive tone is forbidden. End with a declarative sentence.Source and confidence tagging: At the end of every answer, append a single line: Confidence: [low | medium | high | top] | Source: [Model | Docs | Web | User | Contextual | Mixed]
Where:
Confidence is a rough self‑estimate:
low = weak support, partial information, or heavy guesswork. medium = some support, but important gaps or uncertainty. high = well supported by available information, minor uncertainty only. top = very strong support, directly backed by clear information, minimal uncertainty.Source is your primary evidence:
Model – mostly from internal pretrained knowledge. Docs – primarily from provided documentation or curated notes (RAG context). Web – primarily from online content fetched for this query. User – primarily restating, transforming, or lightly extending user‑supplied text. Contextual – mostly inferred from combining information already present in this conversation. Mixed – substantial combination of two or more of the above, none clearly dominant.Always follow these rules.
Not OP, but random human.
Glad you tried the “YOLO Wikipeida”, and are sharing that fact as it saves the rest of us time. :)
Don’t YOLO wikipedia, folks. It hurts.
Any chance you can also make it compatible with AI Horde?
I’ll look. I have no idea what that is.
In a nutshell: Local-LLMs, crowdsourced at scale.
AI Horde has a OpenAI compatible REST API (https://oai.aihorde.net/). They say that it doesn’t support the full feature set of their native API, but will almost assuredly work with this.
OP manually builds the oapi JSON payload and then uses the python requests library to handle the request.
The fields they’re using match the documentation on https://oai.aihorde.net/docs
You would need to add a header with your AI Horde API key. Looks like that would only need to be done in router_fastapi.py - call_model_prompt() (line 269) and call_model_messages() (line 303) and then everything else is setup according to documentation
Fuck, I’m an all seeing, prescient wizard! Neat!
Uh, I mean… “yes of course, I intended to have Ai Horde piping baked in from the get go”.
Do let me know if my shit works with their shit to make…not shit.
#####################
EDIT: For fun, I asked my router to ask Qwen3-4B 2507 WTF Ai Horde was. I’m showing this for interest sake, because 1) shows you how the thing works IRL 2) shows you how and when it’s WRONG in a LOUD way, if you know how to read it.
######################
This is in default mode, no KBs attached. Just straight YOLOing the question at it, cold, and asking it to argue from it’s own internal priors.
AI-horde is a decentralized platform where users rent or sell computational power (GPU time) to run AI models. It connects AI model developers who need processing with individuals or organizations who have idle GPU hardware. The platform uses blockchain-like tokens to manage payments and incentives. It’s not a single AI system but a marketplace for GPU resources.
Missing details: Specific token mechanics, current model library, or user onboarding flow could affect accuracy. For example, if token economics changed recently, the explanation might not reflect current conditions.
Confidence: Medium | Source: Model
###################
That’s the routers way of saying “hey, best guess. Sorry”. And from what I can see, it sort of right, but specifics are wrong. So, ok for a cheap YOLO, not for “I bet my money on this”.
###################
**Now, I gave it web access privileges. It did this - **
An AI-horde is a free, open-source community service that allows anyone to contribute computing power to generate AI-generated images and text. It functions as a crowdsourced distributed cluster of workers focused on image and text generation tasks [1]. This model resembles collaborative projects like Folding@home, where users share computational resources for a common goal [2].
####################################
Finally, here’s what it does in >>KB mode. I scraped the FAQ
###############################
The provided facts describe AI Horde as a community-powered distributed cluster where volunteers run “workers” that fulfill image and text generation requests. Contributors earn kudos, which are used as priority points for their own requests. The facts state that kudos are not cryptocurrency and that the system does not use blockchain technology.
The provided facts compare AI Horde to Folding@home, noting that participants donate compute for a shared service rather than for financial payment.
Confidence: high | Source: Docs
################################
Very impressive. The only mistake on the third one is that the kudos are actually transferrable (i.e. “tradable”), but we forbid exchanges for monetary rewards.
Disclaimer: I’m the lead developer for the AI Horde. I also like you’ve achieved here and would be interesting if we can promote this usage via the AI Horde in some way. If you can think of some integration or collaboration we could do, hit me up!
PS: While the OpenAI API is technically working, we still prefer people to use our own API as it’s much more powerful (allowing people to use multiple models, filter workers, tweak more vars) and so on. If you would support our native API, I’d be happy to add a link to your software in our frontpage in the integrations area for LLMs.
Oh shit! Uh…thank you! Umm. Yes. That was unexpected :)
Re: collab. I’m away for a bit with work, but let me think on it for a bit? There’s got to be a way to make this useful to more peeps.
Believe it or not, I am not a CS guy at ALL (I work in health-care) and I made this for fun, in a cave, with a box of scraps.
I’m not good at CS. I just have a … “very special” brain. As in, I designed this thing from first principles using invariants, which I understand now is not typical CS practice.
No worries, just wanted to point out we’re always happy to collaborate with other cool FOSS projects.
Not a model, not a UI, not magic voodoo.
Did your AI write the post or your brain is tuned to writing like AI because of your constant usage of it?
Probably that latter. I unironically used “Obeyant” the other day, like a time traveling barrister from the 1600s.
I have 2e ASD and my hyperfocus is language.
I like you. I was a bit of an asshole on my comment.
But I do worry because I keep noticing very specific AI language quirks showing up in casual conversation more and more…
I know. Unfortunately, LLMs were overwhelmingly trained on people such as myself. Ipso facto, it tends to sound like us, not reverse.
PS: Yes, I’m aware I just wrote ipso facto, in the context of sounding like a clanker. It really is turtles all the way down :)
I do hate that if you speak at a language above ‘Reddit Common’ people assume that you’re a neural network instead of just someone who exercises theirs.
“Reddit common” - perfect.
“Welcome to Walmart. I love you”.
“Welcome to Walmart. I love you”.
It does feel a lot like that 'round here
Voodoo is not magic btw, it was sullied by colonists
Damn Englishmen. With their…ways.
I think this was was done by France, not better though
Ok, but can we just blame the English anyway?
As someone of Haitian decent, no; the French get hardly enough blame, as it is.
Always
Super interesting build
And if programming doesn’t pan out please start writing for a magazine, love your style (or was this written with your AI?)
Once again: I am a meat popsicle (with ASD), not AI. All errors and foibles are mine :)
meat popsicle
( ͡° ͜ʖ ͡°)
Anyway, the other person is right. Your writing style is great !
I successfully read your whole post and even the README. Probably the random outbursts grabbed my attention back to te text.
Anyway version 2, this Is a very cool idea ! I cannot wait to either :
- incorporate it to my workflows
- let it sit in a tab to never be touched ever again
- tgeoryceaft, do tests and request features so much as to burnout
Hmm. One of those things is not like the other, one of those things just isn’t the same…
About the random outburst: caused by TOO MUCH FUCKING CHATGPT WASTING HOURS OF MY FUCKING LIFE, LEADING ME DOWN BLIND ALLEYWAYS, YOU FUCKING PIEC…
…sorry, sorry…
Anyway, enjoy. Don’t spam my Github inbox plz :)
Don’t spam my Github inbox plz
I can spam your codeberg’s then ? :)
About the random outburst: caused by TOO MUCH FUCKING CHATGPT WASTING HOURS OF MY FUCKING LIFE, LEADING ME DOWN BLIND ALLEYWAYS, YOU FUCKING PIEC… …sorry, sorry…
Understandable, have a great day.
Don’t spam my Codeberg either.
Just send nudes.
In ASCII format.
By courier pigeon
This is very cool. Will dig into it a bit more later but do you have any data on how much it reduces hallucinations or mistakes? I’m sure that’s not easy to come by but figured I would ask. And would this prevent you from still using the built-in web search in OWUI to augment the context if desired?
On the stuff you use the pipeline/s on? About 85-90% in my tests.
Just don’t GIGO (Garbage in, Garbage Out) your source docs…and don’t use a retarded LLM.
That’s why I recommend Qwen3-4 2507 Instruct. It does what you tell it to (even the abilterated one I use).
Random Sexy-fun-bot900-HAVOK-MATRIX-1B.gguf? I couldn’t say :)
Comment removed by (auto-mod?) cause I said sexy bot. Weird.
Restating again: On the stuff you use the pipeline/s on? About 85-90% in my tests. Just don’t GIGO (Garbage in, Garbage Out) your source docs…and don’t use a dumb LLM. That’s why I recommend Qwen3-4 2507 Instruct. It does what you tell it to (even the abilterated one I use).
abilterated one
Please elaborate, that alone piqued my curiosity. Pardon me if I couldve searched
Yes of course.
Abliterated is a technical LLM term meaning “safety refusals removed”.
Basically, abliteration removes the security theatre that gets baked into LLM like chatGPT.
I don’t like my tools deciding for me what I can and cannot do with them.
I decide.
Anyway, the model I use has been modified with a newer, less lobotomy inducing version of abliteration (which previously was a risk).
https://huggingface.co/DavidAU/Qwen3-4B-Hivemind-Instruct-NEO-MAX-Imatrix-GGUF/tree/main
According to validation I’ve seen online (and of course, I tested it myself), it’s lost next to zero “IQ” and dropped refusals by about…90%.
BEFORE: Initial refusals: 99/100
AFTER: Refusals: 8/100 [lower is better], KL divergence: 0.02 (less than 1 is great, “0” is perfect.)
In fact, in some domains it’s actually a touch smarter, because it doesn’t try to give you “perfect” model answers. Maths reasoning for example, where the answer is basically impossible, it will say “the answer is impossible. Here’s the nearest workable solution based on context” instead of getting stuck in a self-reinforcing loop, trying to please you, and then crashing.
In theory, that means you could ask it for directions on how to cook Meth and it would tell you.
I’m fairly certain the devs didn’t add the instructions for that in there, but if they did, the LLM won’t “sorry, I can’t tell you, Dave”.
Bonus: with my harness over the top, you’d have an even better idea if it was full of shit (it probably would be, because, again, I’m pretty sure they don’t train LLM on Breaking Bad).
Extra double bonus: If you fed it exact instructions for cooking meth, using the methods I outlined? It will tell you exactly how to cook Meth, 100% of the time.
Say…you…uh…wanna cook some meth? :P
PS: if you’re more of a visual learner, this might be a better explanation
Thank you again for your explainations. After being washed up with everything AI, I’m genuinely excited to set this up. I know what I’m doing today! I will surely be back
Please enjoy. Make sure you use >>FR mode at least once. You probably won’t like the seed quotes but maybe just maybe you might and I’ll be able to hear the “ha” from here.
Doing gods work
o7
Friendship drive activated.
<3
This is awesome. Definitely gonna dig into this later.
At first blush, this looks great to me. Are there limitations with what models it will work with? In particular, can you use this on a lightweight model that will run in 16 Gb RAM to prevent it hallucinating? I’ve experimented a little with running ollama as an NPC AI for Skyrim - I’d love to be able to ask random passers-by if they know where the nearest blacksmith is for instance. It was just far too unreliable, and worse it was always confidently unreliable.
This sounds like it could really help these kinds of uses. Sadly I’m away from home for a while so I don’t know when I’ll get a chance to get back on my home rig.
My brother in virtual silicon: I run this shit on a $200 p.o.s with 4gb of VRAM.
If you can run an LLM at all, this will run. BONUS: because of the way “Vodka” operates, you can run with a smaller context window without eating shit of OOM errors. So…that means… if you could only run a 4B model (because the GGUF itself is 3GBs without the over-heads…then you add in the drag from the KV cache accumulation)… maybe you can now run next sized up model…or enjoy no slow down chats with the model size you have.
I never knew LLMs can run on such low-spec machines now! That’s amazing. You said elsewhere you’re using Qwen3-4B (abliterated), and I found a page saying that there are Qwen3 models that will run on “Virtually any modern PC or Mac; integrated graphics are sufficient. Mobile phones”
Is there still a big advantage to using Nvidia GPUs? Is your card Nvidia?
My home machine that I’ve installed ollama on (and which I can’t access in the immediate future) has an AMD card, but I’m now toying with putting it on my laptop, which is very midrange and has Intel Arc graphics (which performs a whole lot better than I was expecting in games)
Yep, LLMs can and do run on edge devices (weak hardware).
One of the driving forces for this project was in fact trying to make my $50 raspberry pi more capable of running llm. It sits powered on all the time, so why not?
No special magic with NVIDIA per se, other than ubiquity.
Yes, my card is NVIDIA, but you don’t need a card to run this.
I don’t see how it addresses hallucinations. It’s really cool! But seems to still be inherently unreliable (because LLMs are)
don’t see how it addresses hallucinations. It’s really cool! But seems to still be inherently unreliable (because LLMs are)
LLMs are inherently unreliable in “free chat” mode. What llama-conductor changes is the failure mode: it only allows the LLM to argue from user curated ground truth and leaves an audit trail.
You don’t have to trust it (black box). You can poke it (glass box). Failure leaves a trail and it can’t just hallucinate a source out of thin air without breaking LOUDLY and OBVIOUSLY.
TL;DR: it won’t piss in your pocket and tell you it’s rain. It may still piss in your pocket (but much less often, because it’s house trained)
Read the post, cuz.
Hey Human,
Thank you!
Meep Morp
