Rendered at 22:14:13 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
puttycat 60 minutes ago [-]
This is really impressive. Can you say a bit about the underlying process? I'm guessing this is post-training qantization? Isn't PTQ also resource-intensive? (Ie might not work on any machine)
LLMFit tells you what can run on something. I built something quite similar to their search into Shoehorn now.
sscarduzio 2 hours ago [-]
The project name is perfect!
jedbrooke 3 hours ago [-]
I gotta laugh at some of the models it suggests, for example:
> AnkitAI/Parable-Qwen3-4B-Claude-Fable-5-GGUF
you’re telling me you managed to fit Fable 5 into just 4B?
chompychop 3 hours ago [-]
I gotta laugh at your thought process: knowing Fable 5 is a large frontier model, you're telling me that the first thing that came to your mind on seeing that model name is that it's a quantized version of Fable? As opposed to a distillation/fine-tuning on Fable responses?
metalliqaz 1 hours ago [-]
Well to be fair here... the title of this post doesn't mention fine tuning, it mentions quantization.
unrented7977 2 hours ago [-]
Don't make fun of people you think are ignorant, it's a pretty shitty look
chompychop 2 hours ago [-]
Well, then don't get all snarky and dismissive of things you might not be knowledgeable about ("you" here referring to OP).
does this work similar to airllm? i am wondering how it would handle something like quantizing kimi k3 on a budget of 8 gbs, or is that something you are not attempting to solve yet?
rhgraysonii 3 days ago [-]
Yes that is exactly what this does.
kennywinker 3 days ago [-]
Could you explain what happens when you try to shoehorn a 2.4T parameter model into a 24gb m4 mac?
metalliqaz 1 hours ago [-]
extreme divergence would be my guess
akshay_akula 2 days ago [-]
Wondering the same thing but for 48gb M5 Max.
jaylane 3 days ago [-]
tried it out but based on the model sizing result i got i got an insufficient memory error when the server started running
rhgraysonii 3 days ago [-]
If you could post an issue if you still have the error around that would be awesome.
> AnkitAI/Parable-Qwen3-4B-Claude-Fable-5-GGUF
you’re telling me you managed to fit Fable 5 into just 4B?