this post was submitted on 01 Mar 2024
41 points (100.0% liked)

No Stupid Questions

35726 readers
2739 users here now

No such thing. Ask away!

!nostupidquestions is a community dedicated to being helpful and answering each others' questions on various topics.

The rules for posting and commenting, besides the rules defined here for lemmy.world, are as follows:

Rules (interactive)


Rule 1- All posts must be legitimate questions. All post titles must include a question.

All posts must be legitimate questions, and all post titles must include a question. Questions that are joke or trolling questions, memes, song lyrics as title, etc. are not allowed here. See Rule 6 for all exceptions.



Rule 2- Your question subject cannot be illegal or NSFW material.

Your question subject cannot be illegal or NSFW material. You will be warned first, banned second.



Rule 3- Do not seek mental, medical and professional help here.

Do not seek mental, medical and professional help here. Breaking this rule will not get you or your post removed, but it will put you at risk, and possibly in danger.



Rule 4- No self promotion or upvote-farming of any kind.

That's it.



Rule 5- No baiting or sealioning or promoting an agenda.

Questions which, instead of being of an innocuous nature, are specifically intended (based on reports and in the opinion of our crack moderation team) to bait users into ideological wars on charged political topics will be removed and the authors warned - or banned - depending on severity.



Rule 6- Regarding META posts and joke questions.

Provided it is about the community itself, you may post non-question posts using the [META] tag on your post title.

On fridays, you are allowed to post meme and troll questions, on the condition that it's in text format only, and conforms with our other rules. These posts MUST include the [NSQ Friday] tag in their title.

If you post a serious question on friday and are looking only for legitimate answers, then please include the [Serious] tag on your post. Irrelevant replies will then be removed by moderators.



Rule 7- You can't intentionally annoy, mock, or harass other members.

If you intentionally annoy, mock, harass, or discriminate against any individual member, you will be removed.

Likewise, if you are a member, sympathiser or a resemblant of a movement that is known to largely hate, mock, discriminate against, and/or want to take lives of a group of people, and you were provably vocal about your hate, then you will be banned on sight.



Rule 8- All comments should try to stay relevant to their parent content.



Rule 9- Reposts from other platforms are not allowed.

Let everyone have their own content.



Rule 10- Majority of bots aren't allowed to participate here.



Credits

Our breathtaking icon was bestowed upon us by @Cevilia!

The greatest banner of all time: by @TheOneWithTheHair!

founded 1 year ago
MODERATORS
 

WebDAV has been around a lot longer and does many of the same things as object storage. It also has support for random access read/writes where object storage requires you to download, edit, and re-upload the whole file. Seems like a no-brainer if you wanted to offer cloud storage to customers.

I thought maybe supporting large uploads was the draw, but WebDAV can support chunking, so you don't need to allocate extra server resources to accommodate large files.

I use both daily, and WebDAV just seems like it does everything better: object storage feels like throwing files in a junk drawer and WebDAV more like an organized filing cabinet.

Aside from Nextcloud and a few FOSS applications, the only big thing I recall that adopted WebDAV was Frontpage back in the day.

So, what am I missing? What makes object storage so compelling that it became ubiquitous while WebDAV is practically a legacy spec?

top 14 comments
sorted by: hot top controversial new old
[–] hperrin@lemmy.world 30 points 8 months ago* (last edited 8 months ago) (3 children)

To give you a real answer, from someone who loves WebDAV and has written a WebDAV server with an S3 backend, object storage is easier/possible to run at scale and serves a different purpose.

Object storage is and always has been based on a key-value model. You put a key and value in, and later you can request that key to get that value. It technically has no concept of hierarchy. WebDAV supports so much more than that. WebDAV has collections (hierarchy), live and dead properties (S3 has something similar to these), methods like MOVE and PROPFIND, and a system of hierarchical locking (depth 1 locking on a collection and depth infinity locking on an entire namespace).

This means that in order to build a WebDAV server, you need to know a lot of information about what exists in the data storage. S3 is a lot “dumber” in that regard. The funny thing is S3 has added so much functionality that they’ve essentially rewritten WebDAV in a more convoluted form. Whereas on WebDAV you can just propfind a collection with depth 1, on S3 you need to list keys with a prefix and delimiter, then make additional requests for any other props you may need.

Unfortunately, the one thing WebDAV is missing that users of S3 often need is the concept of partial listing. In S3, when you list keys, you tell it how many keys you want back, then it will only give you that many keys max. If there are more keys that it didn’t give you, it will tell you the results are truncated and give you a continuation token. You can use this token in your next request to continue listing keys.

This is where the “at scale” thing comes in. If you have hundreds of millions of keys in a bucket, getting them all back at once would certainly break your system, and probably would tax the server unnecessarily. So basically the answer is S3 is designed for scale.

That being said, S3 is not really designed for humans to interact with. This is where the “different purpose” thing comes in. It doesn’t have a real concept of hierarchy, just common prefixes and delimiters. So something like renaming a directory would require copying every object with that prefix to a new key, then deleting the originals (which is what my S3 adapter does for my WebDAV server). S3 is more meant to be used with something like UUIDs or hashes for keys. Keys that don’t change. WebDAV is designed more like a file system.

I hope that explains it well.

PS: One minor correction, WebDAV itself does not support random writes. That’s a separate RFC that’s not part of WebDAV, but is perfectly compatible, and many WebDAV servers offer that functionality.

[–] hperrin@lemmy.world 3 points 8 months ago

An additional point is that CardDAV and CalDAV are both extensions of the WebDAV spec, and are widely used by a number of products, so WebDAV is definitely not a legacy spec. It’s the foundation to two very popular specs supported by billions of devices.

[–] bjoern_tantau@swg-empire.de 3 points 8 months ago (1 children)

Wait, so when I want a directory listing from WebDAV and the directory contained 1000 files, I would always have to wait for the whole thing? That explains so much.

[–] hperrin@lemmy.world 3 points 8 months ago

Yep. PROPFIND only has a Depth option.

[–] ptz@dubvee.org 3 points 8 months ago

Thanks for the detailed reply. That pretty much answers it.

I definitely agree on the different purposes, but sadly that doesn't help where object storage is used where it really doesn't make sense (my org replaced their fileserver with object storage and a client sync app - grr).

WebDAV itself does not support random writes. That’s a separate RFC that’s not part of WebDAV, but is perfectly compatible, and many WebDAV servers offer that functionality

Ah, true. I was looking at SabreDAV specifically which does support it and made a leap that it was part of the spec.

Also, I am definitely going to check out your Nephele Serve project. Thanks for mentioning that.

[–] maynarkh@feddit.nl 15 points 8 months ago (1 children)

I don't know much about the history, but I would guess that adoption was driven by the actual service that was provided, not how good the protocol was. AWS did their own thing instead of adopting WebDAV, who knows why. Then people started using S3 and building stuff on it since it was cheap. Now people build services that are S3 conformant so that the stuff built on S3 can be migrated to it.

This is all just an educated guess though.

[–] redcalcium@lemmy.institute 4 points 8 months ago* (last edited 8 months ago) (1 children)

When S3 was released, the huge draw was its pay-as-you-go model, not its new protocol. If amazon was using webdav instead of making their own protocol, I bet it'll still got popular.

[–] maynarkh@feddit.nl 2 points 8 months ago

Yeah, that was kinda my point. Economics drove adoption, not technological brilliance or even ease-of-use.

[–] key@lemmy.keychat.org 13 points 8 months ago (1 children)

S3 succeeded due to the scaling capabilities and the ability to abstract completely away from a server or disk. The straight forward Key/Value nature of the s3api was a big assistance in achieving the scaling and adoptability.

Comparing it to WebDav seems like comparing apples and... an orange smoothie.

[–] skullgiver@popplesburger.hilciferous.nl 3 points 8 months ago (1 children)

I don't get it, to be honest. WebDav is as key/value as S3. The key is the file path, the value is the file contents. Instead of a PATCH you send a POST request to a specific path with the key encoded in a different way. Same for a DELETE.

Of course, WebDAV was intended for file shares, not as a web API, but I don't see why WebDAV couldn't have been implemented.

[–] blakemiller@lemmy.world 1 points 8 months ago

Couldn’t say for sure but WebDAV probably would be clunky if fronted by a distributed database. The beauty of S3 is you add more servers, add more disks, and bam you’ve got more S3. That happens most easily when the metadata system sitting in the front can expand easily. I don’t know how easy that would be to plumb up with WebDAV. Whether or not one was better here, S3 ultimately won because it’s a primitive API that was essentially impossible to fuck up.

[–] litchralee@sh.itjust.works 5 points 8 months ago (1 children)

I'm only cursorily familiar with WebDAV, but I think the needs of cloud storage aligned much better to the object storage model than WebDAV's file/directory structure. For example, in a distributed cloud across continents, referencing a file in WebDAV might have a canonical path, but object storage would just need a key or hash. And by using a key/hash, automatic deduplication is achieved, since the same object should hash to the same key. File paths necessarily imply context, but the point of clouds is to be homogeneous. If paths need to be world-unique but locally-cached, then the path is just a unique identifier and we slowly end up with the database-like semantics of object storage anyway.

Phrased another way, a file/directory structure imparts an organization to the contents of those files. Cloud doesn't need that organization, so throwing stuff in the junk drawer is perfectly reasonable.

[–] theit8514@lemmy.world 4 points 8 months ago

This is funny because most object storages now use keys that represents a path. For example, you can host a website on S3 with folders for js/css/etc and it "just works".

[–] baggins@lemmy.ca 1 points 8 months ago

Dunno but I remember trying WebDAV back in the day when my webhost offered it as an alternative to FTP and I remember it not working very well for that.