this post was submitted on 24 Oct 2023
261 points (92.0% liked)

Technology

34788 readers
344 users here now

This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.


Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.


Rules:

1: All Lemmy rules apply

2: Do not post low effort posts

3: NEVER post naziped*gore stuff

4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.

5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)

6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist

7: crypto related posts, unless essential, are disallowed

founded 5 years ago
MODERATORS
 

The tool, called Nightshade, messes up training data in ways that could cause serious damage to image-generating AI models. Is intended as a way to fight back against AI companies that use artists’ work to train their models without the creator’s permission.

ARTICLE - Technology Review

ARTICLE - Mashable

ARTICLE - Gizmodo

The researchers tested the attack on Stable Diffusion’s latest models and on an AI model they trained themselves from scratch. When they fed Stable Diffusion just 50 poisoned images of dogs and then prompted it to create images of dogs itself, the output started looking weird—creatures with too many limbs and cartoonish faces. With 300 poisoned samples, an attacker can manipulate Stable Diffusion to generate images of dogs to look like cats.

you are viewing a single comment's thread
view the rest of the comments
[–] JustEnoughDucks@feddit.nl 47 points 1 year ago (3 children)

I'm interested to know how they fool the AI while keeping it invisible to the human eye. Do they make additional layers? Do they change every nth pixel? Is every poisoning associated with another poisoned object? (Will a dog always be poisoned towards a cat?, etc...)

Interesting, but a bit hard to understand.

[–] bort@feddit.de 5 points 1 year ago (1 children)

how they fool the AI while keeping it invisible to the human eye

My guess is that AI companies will try to scrape as much as possible without a human ever looking at the data.

When poisoned data start to become enough of a problem, that humans have to look over very sample, then this would increase training cost to to a point where it's no longer worth to bother with it in the first place.

[–] JustEnoughDucks@feddit.nl 14 points 1 year ago

But that has absolutely nothing to do with how the mechanism works lol. Of course they are trying to eliminate data scraping, that is the whole controversy

[–] itsralC@lemm.ee 2 points 1 year ago (1 children)

Disappointingly, the article only says that it "changes pixels in ways imperceptible to the human eye"

[–] doctorcherry@lemmy.ml 2 points 1 year ago

I think that is a feature