this post was submitted on 09 Jun 2023
7 points (100.0% liked)

Python

3222 readers
1 users here now

News and discussions about the programming language Python

founded 5 years ago
MODERATORS
MinutePhrase@lemmy.ml
7
pickling large objects (lemmy.ml)
submitted 1 year ago by lntl@lemmy.ml to c/python@lemmy.ml
9 comments fedilink hide all child comments
 

I have a large object that I want to save to the disk because it takes a minute to generate. The OOM reaper kills the process while pickle.dump ing the object.

It's a tuple of dicts of tuple of array.array.

Can pickle dump in chunks? If not, is there another technique I can use?

you are viewing a single comment's thread
view the rest of the comments
[–] McWizard@feddit.de 1 points 1 year ago

A colleague of mine replaced the internal store format of pickle with Json iirc. It was like 10x faster. Not exactly sure how you do that, but I can check if you want to go that way.