this post was submitted on 25 Dec 2024
61 points (96.9% liked)

Sysadmin

7770 readers
25 users here now

A community dedicated to the profession of IT Systems Administration

No generic Lemmy issue posts please! Posts about Lemmy belong in one of these communities:
!lemmy@lemmy.ml
!lemmyworld@lemmy.world
!lemmy_support@lemmy.ml
!support@lemmy.world

founded 2 years ago
MODERATORS
DarraignTheSane@lemmy.world
00Lemming@lemmy.world
L3s@lemmy.world
61
How can I make my Lemmy server delete cached images? (gregtech.eu)
submitted 1 day ago by gregor@gregtech.eu to c/sysadmin@lemmy.world
3 comments fedilink hide all child comments
 

The storage usage is at 340GB currently, which is a lot and it's rapidly increasing. I use Backblaze B2 for my storage. Here is my docker compose file: x-logging: &default-logging driver: "json-file" options: max-size: "50m" max-file: "4"

services:
  proxy:
    image: docker.io/library/nginx
    volumes:
      - ./nginx_internal.conf:/etc/nginx/nginx.conf:ro,Z
      - ./proxy_params:/etc/nginx/proxy_params:ro,Z
    restart: always
    logging: *default-logging
    depends_on:
      - pictrs
      - lemmy-ui
    labels:
      - traefik.enable=true
      - traefik.http.routers.http-lemmy.entryPoints=http
      - traefik.http.routers.http-lemmy.rule=Host(`gregtech.eu`)
      - traefik.http.middlewares.https_redirect.redirectscheme.scheme=https
      - traefik.http.middlewares.https_redirect.redirectscheme.permanent=true
      - traefik.http.routers.http-lemmy.middlewares=https_redirect
      - traefik.http.routers.https-lemmy.entryPoints=https
      - traefik.http.routers.https-lemmy.rule=Host(`gregtech.eu`)
      - traefik.http.routers.https-lemmy.service=lemmy
      - traefik.http.routers.https-lemmy.tls=true
      - traefik.http.services.lemmy.loadbalancer.server.port=8536
      - traefik.http.routers.https-lemmy.tls.certResolver=le-ssl


  lemmy:
    image: dessalines/lemmy:0.19.8
    hostname: lemmy
    restart: always
    logging: *default-logging
    volumes:
      - ./lemmy.hjson:/config/config.hjson:Z
    depends_on:
      - postgres
      - pictrs
    networks:
      - default
      - database

  lemmy-ui:
    image: dessalines/lemmy-ui:0.19.8
    volumes:
      - ./volumes/lemmy-ui/extra_themes:/app/extra_themes:Z
    depends_on:
      - lemmy
    restart: always
    logging: *default-logging
    environment:
      - LEMMY_UI_LEMMY_INTERNAL_HOST=lemmy:8536
      - LEMMY_UI_LEMMY_EXTERNAL_HOST=gregtech.eu
      - LEMMY_UI_HTTPS=true

  pictrs:
    image: docker.io/asonix/pictrs:0.5
    # this needs to match the pictrs url in lemmy.hjson
    hostname: pictrs
    # we can set options to pictrs like this, here we set max. image size and forced format for conversion
    # entrypoint: /sbin/tini -- /usr/local/bin/pict-rs -p /mnt -m 4 --image-format webp
    #entrypoint: /sbin/tini -- /usr/local/bin/pict-rs run  --max-file-count 10  --media-max-file-size 500 --media-retention-proxy 10d --media-retention-variants 10d  filesystem sled -p /mnt
    user: 991:991
    environment:
      - PICTRS__STORE__TYPE=object_storage
      - PICTRS__STORE__ENDPOINT=https://s3.eu-central-003.backblazeb2.com/
      - PICTRS__STORE__BUCKET_NAME=gregtech-lemmy
      - PICTRS__STORE__REGION=eu-central
      - PICTRS__STORE__USE_PATH_STYLE=false
      - PICTRS__STORE__ACCESS_KEY=redacted
      - PICTRS__STORE__SECRET_KEY=redacted
      - MEDIA__RETENTION__VARIANTS=4d
      - MEDIA__RETENTION__PROXY=4d
      #- PICTRS__MEDIA__IMAGE__FORMAT=webp
      #- PICTRS__MEDIA__IMAGE__QUALITY__WEBP=50
      #- PICTRS__MEDIA__ANIMATION__QUALITY=50
    volumes:
      - ./volumes/pictrs:/mnt:Z
    restart: always
    logging: *default-logging

  postgres:
    image: docker.io/postgres:16-alpine
    hostname: postgres
    volumes:
      - ./volumes/postgres:/var/lib/postgresql/data:Z
      #- ./customPostgresql.conf:/etc/postgresql.conf:Z
    restart: always
    #command: postgres -c config_file=/etc/postgresql.conf
    shm_size: 256M
    logging: *default-logging
    environment:
      - POSTGRES_PASSWORD=password
      - POSTGRES_USER=lemmy
      - POSTGRES_DB=lemmy
    networks:
      - database
  postfix:
    image: docker.io/mwader/postfix-relay
    restart: "always"
    logging: *default-logging

  #pictrs-safety:
  #  image: ghcr.io/db0/pictrs-safety:v1.2.2
  #  hostname: pictrs-safety
  #  environment:
  #  ports:
  #    - "14051:14051"
  #  user: 991:991
  #  restart: always
  #  logging: *default-logging
  #  depends_on:
  #    - pictrs
networks:
  default:
    name: traefik_access
    external: true
  database:
all 4 comments
sorted by: hot top controversial new old
[–] ptz@dubvee.org 13 points 1 day ago* (last edited 1 day ago) (1 children)

Lemmy doesn't so much cache the images so much as "store them forever".

Regular post thumbnails are just "internal" images. I'm not sure if newer Lemmy versions log these internally like user uploads are, but assuming not, there's no direct way to deal with them.

Pict-rs keeps a true cache of image variants (alternate versions of images that were requested at different resolutions / formats). Those can be cleared programmatically:

curl -XDELETE -H "X-Api-token:YOUR_API_KEY" http://127.0.0.1:8080/internal/variants

Where your pict-rs API port is exposed to localhost on port 8080. Also note that those variants will be re-created on demand.

One easy-ish way to get rid of images is to select from the post table for any thumbnail_url that starts with your instance. e.g. https://your-instance/pictrs/image/%s. You can filter that by published date if you only want to wipe out thumbnails for posts older than a year, for example.

With that list, you can strip off the filename/alias (the uuid and extension) and pass each alias filename to a script that tells pict-rs to delete it (you need at least pict-rs 0.5 for this to work):

#!/bin/bash
ALIAS="$1"
API_KEY=FooBarAPIKeyIsFake

curl -XPOST -H "X-Api-token:$API_KEY" http://127.0.0.1:8080/internal/delete?alias=%24ALIAS

Below 0.5, the only way to delete images fro pict-rs without the delete token was to use the purge endpoint. But that deletes all aliases and not just the one you want to delete since it does de-duplication under the hood.

If you want to get fancy so that image posts still render correctly, when building your list of thumbnails to delete, you can check if the value of the url column is an image. If it is, grab the current thumbnail image to pass to the delete function and then update the thumbnail URL value to that of the post URL.

[–] possiblylinux127@lemmy.zip -5 points 1 day ago (1 children)

Wrong community

[–] qaz@lemmy.world 1 points 23 hours ago

Why? It seems on topic to me