this post was submitted on 02 Sep 2024
885 points (99.1% liked)
Programmer Humor
32444 readers
1592 users here now
Post funny things about programming here! (Or just rant about your favourite programming language.)
Rules:
- Posts must be relevant to programming, programmers, or computer science.
- No NSFW content.
- Jokes must be in good taste. No hate speech, bigotry, etc.
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Neat, thanks for sharing. Reminds me of old mainframe computers where students and researchers had to apply for processing time. Large data analysis definitely makes sense for C++, and it's pretty low risk. Presumably you'd be able to go back and reprocess stuff if something went wrong? Or is more of a live-feed that's not practical to store?
The data are stored, so it's not a live-feed problem. It is an inordinate amount of data that's stored though. I don't actually understand this well enough to explain it well, so I'm going to quote from a book [1]. Apologies for wall of text.
It's very cool, I wish I knew more about this. A figure I found for approximate data rate is 5GB/s per instrument. I think that's for the European XFELS.
Citation: [1]: Yoon, C.H., White, T.A. (2018). Climbing the Data Mountain: Processing of SFX Data. In: Boutet, S., Fromme, P., Hunter, M. (eds) X-ray Free Electron Lasers. Springer, Cham. https://doi.org/10.1007/978-3-030-00551-1_7
That's definitely a non-trivial amount of data. Storage fast enough to read/write that isn't cheap either, so it makes perfect sense you'd want to process it and narrow it down to a smaller subset of data ASAP. The physics of it is way over my head, but I at least understand the challenge of dealing with that much data.
Thanks for the read!