this post was submitted on 14 Jul 2023
6 points (87.5% liked)

Self Hosted - Self-hosting your services.

11419 readers
1 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules

Important

Beginning of January 1st 2024 this rule WILL be enforced. Posts that are not tagged will be warned and if not fixed within 24h then removed!

Cross-posting

If you see a rule-breaker please DM the mods!

founded 3 years ago
MODERATORS
Zoe8338@lemmy.ml
dogmuffins@lemmy.ml
testman@lemmy.ml
6
Can I run local LLMs on Intel ARC/AMD with 8GB of RAM? (lemmy.world)
submitted 1 year ago by MigratingtoLemmy@lemmy.world to c/selfhost@lemmy.ml
8 comments fedilink hide all child comments
 

publication croisée depuis : https://lemmy.world/post/1474932

Hi there.

I wanted to run LLMs locally on my server (for better privacy), and was wondering if:

  1. I could use Intel ARC/AMD GPUs - these are often less expensive and AMD has open source drivers, which is something I like.
  2. If a PCIe x4 Gen 3 slot would be enough (it's an x16 slot with x4 speeds) - this is an important consideration.
  3. Would 8GB of RAM (in the GPU, I believe it's called VRAM?) be enough?

I'm looking at language models to train on my Reddit and Lemmy content, in an aim to make it write like me (and maybe even better than me? Who knows). I don't quite know which models I will train, or how I will do so (I certainly won't be writing anything from scratch), but I was wondering; with the explosion of FOSS AI models, maybe something like this would be possible with the hardware constraints I mentioned above?

Does the speed of the connection between the GPU and the CPU really matter in such applications?

Thanks!

you are viewing a single comment's thread
view the rest of the comments
[–] surrendertogravity@wayfarershaven.eu 1 points 1 year ago (1 children)

So I’m no expert at running local LLMs, but I did download one (the 7B vicuña model recommended by the LocalLLM subreddit wiki) and try my hand at training a LoRA on some structured data I have.

Based on my experience, the VRAM available to you is going to be way more of a bottleneck than PCIe speeds.

I could barely hold a 7B model in 10 GB of VRAM on my 3080, so 8 GB might be impossible or very tight. IMO to get good results with local models you really have large quantities of VRAM and be using 13B or above models.

Additionally, when you’re training a LoRA the model + training data gets loaded into VRAM. My training dataset wasn’t very large, and even so, I kept running into VRAM constraints with training.

In the end I concluded that in the current state, running a local LLM is an interesting exercise but only great on enthusiast level hardware with loads of VRAM (4090s etc).

[–] MigratingtoLemmy@lemmy.world 1 points 1 year ago (2 children)

I see. I do not want to spend $1000+ on a GPU, I suppose this will have to wait a few years

[–] surrendertogravity@wayfarershaven.eu 1 points 1 year ago (1 children)

Yup; hopefully there are some advances in the training space, but I’d guess that having large quantities of VRAM is always going to be necessary in some capacity for training specifically.

[–] fraichu@lemmy.ml 2 points 1 year ago (1 children)

I hope some GPU manufacturer starts allowing removable RAMs. 4 x 8 GB DDR5 might not be too bad given PCIe speeds aren't a bottleneck. If I could upgrade the RAM to 64 GB later, I'm ready to give $10k at 3080 level perf. Intel ARC people I hope you are already doing this!

[–] surrendertogravity@wayfarershaven.eu 2 points 1 year ago* (last edited 1 year ago)

I don’t know anything about GPU design but expandable VRAM is a really interesting idea. Feels too consumer friendly for Nvidia and maybe even AMD though.

[–] surrendertogravity@wayfarershaven.eu 0 points 1 year ago

Yup; hopefully there are some advances in the training space, but I’d guess that having large quantities of VRAM is always going to be necessary in some capacity for training specifically.