this post was submitted on 02 Jul 2023
57 points (95.2% liked)

Selfhosted

40767 readers
362 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
 

I was told that I should post this here.

cross-posted from: https://lemmy.world/post/932750

Say you decide to self-host a Lemmy instance. When you create that instance, do you immediately need to download and store all the data that has ever been posted to all federated Lemmy instances? Or perhaps you only need to download and store everything that is posted to the federated Lemmy instances from that point forward? Or better yet, do you only store what the users on that instance do (i.e. their posts, and posts to the communities hosted on that instance)?

you are viewing a single comment's thread
view the rest of the comments
[–] Kalcifer@lemmy.world 3 points 2 years ago (6 children)

When your instance needs to fetch from another instance it will

Meaning it will only fetch what is being actively looked at?

Only communities that your users subscribe to will be updated by their “origin” instances.

So when an external community is subscribed to from an account located on your located instance, from the point of subscribing forward, your local instance will begin downloading every single post that will ever be made to that subscribed communty, regardless of who posted it?

Or better yet, do you only store what the users on that instance do (i.e. their posts, and posts to the communities hosted on that instance)?

This does happen, but it also stores what your users do on remote instances as well as “copies” of what they interact with. Images (currently the only media hosted by lemmy servers) are linked to thier “origin” as well. So you are storing text of posts and comments.

This is the main point of confusion to me. From my current understanding, it feels as if it contradicts what you had previously said:

Only communities that your users subscribe to will be updated by their “origin” instances.

If it's already pulling in all posts and comments on that community, what use is specifically storing anything that the users do on that community? Would it not be already stored?

[–] hawkwind@lemmy.management 4 points 2 years ago (5 children)

It works a lot like like email between instances. Let’s call your self hosted instance “A” and the popular remote instance “B.”

User on A searches for “poodles” and finds a community !poodles@B. When they click the search results: A sends B mail saying “send me the last 10 posts for poodles.” B sends A mail with the posts and the user sees the posts, but none have comments.

If nothing else happens then those 10 posts will just hang out doing nothing on A, but if the user clicks subscribe then A sends another mail to B saying “my user wants to follow poodles.” B replies saying “cool, I’ll send you everything from poodles now.” Now, anything a post or comment happens B checks lots list of subscribing instances and sends copies of them.

If user on A comments on !poodles@B or posts, it creates it on A but sends a mail to B saying “here is some new stuff for poodles!”

[–] Kalcifer@lemmy.world 2 points 2 years ago* (last edited 2 years ago) (4 children)

Thank you for the explanation!

Unfortunately, it seems, if I understand understand correcly, that this is not sustainable in the long term for small instances/servers. If Lemmy continues to grow in popularity, then the influx of content will continue to increase, thereby pushing small servers out of participation due to lack of resources. The data storage requirements, I fear, will become a very limiting issue.

I feel that if servers only tracked what their users directly participated in (i.e. only save comments, and posts directly made by the user), this issue would not be as problematic.

For example, I would like to host my own instance with only my account on it. I was initially hoping that my data storage requirements would only be directly proportional to how much I, as a user, use Lemmy; the server would only need to store my personally created data, and nothing else. Unfortunately, however, it appears that I would also have to have enough resources to sustain everyone elses posts which is a far steeper requirement.

[–] hawkwind@lemmy.management 3 points 2 years ago

Media takes up space. The text from posts and comments is trivial. The database for lemmy.world is only 25 GB. Wikipedia text is only 21 GB.

load more comments (3 replies)
load more comments (3 replies)
load more comments (3 replies)