LocalLLaMA

2244 readers

1 users here now

Community to discuss about LLaMA, the large language model created by Meta AI.

This is intended to be a replacement for r/LocalLLaMA on Reddit.

founded 1 year ago

MODERATORS

SkySyrup@sh.itjust.works

pax@sh.itjust.works

noneabove1182@sh.itjust.works

Discovering Locally Run Language Models: Share Your Favorites/Not So Favorites! (beehaw.org)

submitted 1 year ago* (last edited 1 year ago) by dtlnx@beehaw.org to c/localllama@sh.itjust.works

14 comments fedilink hide all child comments

Let's talk about our experiences working with different models, either known or lesser-known.

Which locally run language models have you tried out? Share your insights, challenges, or anything you found interesting during your encounters with those models.

top 14 comments

sorted by: hot top controversial new old

[–] actuallyacat@sh.itjust.works 7 points 1 year ago* (last edited 1 year ago) (1 children)

The wizard-vicuna family is my favorite, they successfully combine lucidity with creativity. Wizard-vicuna-30b is competitive with guanaco-65b in most cases while being subjectively more fun. I hope we get a 65b version, or a Falcon 40B one

I've been generally unimpressed with models advertised as good for storytelling or roleplay, they tend to be incoherent. It's much easier to get wizard-vicuna to write fluent prose than it is to get one of those to stop mixing up characters or rules. I think there might be some sort of poison pill in the Pygmalion dataset, it's the common factor in all the models that didn't work well for me.

[–] theterrasque@infosec.pub 2 points 1 year ago (1 children)

What setup do you have? Prompt / instruct formatting?

[–] actuallyacat@sh.itjust.works 6 points 1 year ago

W-V is supposedly trained for "USER:/ASSISTANT:" but I've found it flexible and able to work with anything that's consistent. For creative writing I'll often do "USER:/STORY:". More than two such tags also work, e.g. I did a rpg-style thing with three characters plus an omniscient narrator, by just describing each of them with their tag in the prompt, and it worked nearly flawlessly. Very impressive actually.

[–] Kerfuffle@sh.itjust.works 2 points 1 year ago (1 children)

guanaco-65B is my favorite. It's pretty hard to go back to 33B models after you've tried a 65B.

It's slow and requires a lot of resources to run though. Also, not like there are a lot of 65B model choices.

[–] planish@sh.itjust.works 2 points 1 year ago* (last edited 1 year ago) (1 children)

What do you even run a 65b model on?

[–] Kerfuffle@sh.itjust.works 4 points 1 year ago (1 children)

With a quantized GGML version you can just run on it on CPU if you have 64GB RAM. It is fairly slow though, I get about 800ms/token on a 5900X. Basically you start it generating something and come back in 30minutes or so. Can't really carry on a conversation.

[–] planish@sh.itjust.works 3 points 1 year ago (1 children)

Is it smart enough that it can get the thread of what you are looking for without as much rerolling or handholding, so this comes out better?

[–] Kerfuffle@sh.itjust.works 2 points 1 year ago (1 children)

That's the impression I got from playing with it. I don't really use LLMs for anything practical, so I haven't done anything too serious with it. Here's are a couple examples of having it write fiction: https://gist.github.com/KerfuffleV2/4ead8be7204c4b0911c3f3183e8a320c

I also tried with plain old llama-65B: https://gist.github.com/KerfuffleV2/46689e097d8b8a6b3a5d6ffc39ce7acd

You can see it makes some weird mistakes (although the writing style itself is quite good).

If you want to give me a prompt, I can feed it to guanaco-65B and show you the result.

[–] planish@sh.itjust.works 1 points 1 year ago (1 children)

These are, indeed, pretty good, and quite coherent.

[–] Kerfuffle@sh.itjust.works 2 points 1 year ago

I was pretty impressed by guanaco-65B, especially how it was able to remain coherent even way past the context limit (with llama.cpp's context wrapping thing). You can see the second story is definitely longer than 2,048 tokens.

[–] dtlnx@beehaw.org 2 points 1 year ago (1 children)

I'd have to say I'm very impressed with WizardLM 30B (the newer one). I run it in GPT4ALL, and even though it is slow the results are quite impressive.

Looking forward to Orca 13b if it ever releases!

[–] micheal65536@lemmy.micheal65536.duckdns.org 1 points 1 year ago

Which one is the "newer" one? Looking at the quantised releases by TheBloke, I only see one version of 30B WizardLM (in multiple formats/quantisation sizes, plus the unofficial uncensored version).

[–] Yahma@kbin.social 1 points 1 year ago

Guanaco, WizardLM (uncensored) and Camel-13b have been the best models I've tried that are 13b+.

Surprisinly, the LaMini-LM (Flan 3b) and OpenLlama (3b) have performed very well for smaller models.

[–] Equality_for_apples@sh.itjust.works 1 points 1 year ago

I've been doing RP with Wizard-Vicuna 13b Uncensored. it's good, very fast (ggml v3 q5-k-s variant) But sometimes forgets it's roleplaying and spits out a story