We are aware of an issue with some clients not being able to sync (receiving 502 errors, etc..). It does not appear to affect all clients and the web view is still working. We will update this post as we know more and when we have resolved it.

Updates:

09-Sept 11:19 PM UTC We attempted to roll back from Lemmy v0.18.4-beta.8 to v0.18.4 (as well as even earlier beta versions) but the service failed to start. We've reverted back to v0.18.4-beta.8 for now and will continue investigating the cause of the sync issues.

10-Sept 3:30 AM UTC And... we are back, where we started. If you hadn't noticed, we were down for almost 2.5 hours while we tried to rollback the database migrations introduced in the beta releases so we could go back to the stable v0.18.4. When that failed, we tried to restore from the backup we took just prior to the rollback attempt. When that failed, we used Digital Ocean's restore to point in time and created a new DB cluster and switched to it. So... we are back on v0.18.4-beta.8 again, with the expectation that 3rd party clients are still broken. Unfortunately, we are not sure what we can do about this as we believe it is something in Lemmy that is breaking it. The mitigation we have put in place to avoid this going forward is that we no longer use the "latest" tag when loading a docker image and now hard lock to a specific version. We will update that version in a more intentional manner going forward.

10-Sept 4:15 AM UTC After some additional digging through the Lemmy diff for beta.5 to beta.8 (and then looking through the PRs that were merged to create this diff), the issue is indeed with the Lemmy code and is even called out in one of the PRs,. What this means is that the 3rd party clients that are not able to access the site will need to update their code to fix things on their side (or else they will be broken when v0.19 is released and all other instances update to it).

Did you know that Discuss.Online has a Matrix server?

If you're not familiar with Matrix you can read more here: https://matrix.org/about/

Matrix is a federated real-time chat server.

Join the community space! Go to: https://matrix.to/#/#community-channels:discuss.online.

Need a username? Contact @jgrim@discuss.online or comment on this post. A registration code will be sent to you for [yourhandle]@discus.online

Matrix Clients Discuss.Online provides two web based client options. You can find more at: https://matrix.org/ecosystem/clients/

Here are the Discuss.Online hosted ones:

• https://element.discuss.online
• https://cinny.discuss.online

See you there!

Please welcome @lazyguru@discuss.online to the discuss.online admin team.

I’ve known lazyguru for over ten years. He’s a close friend. His skills are like mine. He will be able to help contribute to all projects of the discuss.online ecosystem.

lazyguru will help with moderation, system administration, monitoring, and development of new features. Joe has already improved some monitoring and is working on some changes to the infrastructure to make it more scalable. He will soon be helping with SocialCare.cloud once the first beta is released.

Please, take some time to welcome him!

Thanks, Jason

tl;dr;

We use SendGrid to send our emails out. By default, SendGrid enables link tracking. Link tracking was causing issues with allowing users to verify their accounts. In short, I disabled link tracking, and links should work for all new emails in the future.

Contact support if you need your account activated. hello@discuss.online from the email you used to sign up.

Issue

When new users sign up or when existing users reset their password, an email is sent to verify ownership of the account. These emails are sent from us through a service called SendGrid.

SendGrid, by default, will mask all the URLs to enable what they call link tracking. It's a marketing thing to track engagement with email. I have no use for it, but I didn't disable it before.

The issue is that sometimes this link doesn't resolve. An HTTPS error is shown to the user.

Resolution

The quickest and most logical resolution was to disable link tracking. I never use it. I did just that; however, any previous emails sent will still be broken.

I will continue to investigate the issue in hopes of resolving previous links.

If you are unable to verify your account, please email hello@discuss.online from the email you used to sign up. We'll manually verify each account that hasn't been able to be activated.

The small outage from this morning was from a planned update to Lemmy 18.3. Let me know if you notice any issues!

Hello,

There is a unique problem of the fediverse. When an instance goes offline, its communities will never sync again.

Recently, vlemmy.com shutdown. Quite a few communities synced with discuss.online and other instances. Because vlemmy.com is not longer brokering communication, these communities will never be in sync again.

We have a several options:

1. Leave them there. Do nothing.
2. Leave them there but make a post that it's dead and hope people see it.
3. Purge the communities. Act like they never existed.
4. Build some elaborate system to work around vlemmy being gone. This would take a lot of work and collaboration with other instances.

Let me know what makes the most sense to you as users. Are any of you still using vlemmy communities? What about long-term planning? Maybe this isn't an issue now but what if lemmy.world vanished?

Please, let me know what you think. I'm torn on this one.

Thanks, Jason

I'm looking to get member feedback on the design of the site.

Lemmy comes with many default core themes. Some parts of these themes are broken with dark mode. For example, upload boxes might show white if your system is set to light mode.

I want to set a decent default theme. If unaware, you can change your theme in your account settings. There are 10 different themes currently.

There are 12 different theme options:

1. Browser Default
   • When this is set, it uses Darkly or Litely depending on browser settings
2. Browser Default Compact
   • When this is set, it uses Darkly Compact or Litely Compact, depending on browser settings
3. Darkly
4. Darly Red
5. Darkly Compact
6. Darkly Pure Black
7. Litely
8. Litely Red
9. Litely Compact
10. Vaporwave Dark
11. Vaporwave Light
12. i386

I currently have the default set to Browser Default Compact, which should set dark or light mode depending on whether your browser is dark or light.

I use Vaporwave Dark, personally.

What do you use? Which do you prefer?

I could make something new. I tried some new ones that broke with an update. I didn't fix them after the update, so I removed them. I could create new ones again.

Let me know what you think. This is a discussion/poll on what I should do as a default.

Thanks, Jason

There will be a short 5 to 10-minute outage as I scale down the environment. I was ambitious about the growth of this community during the Rexidus / Rexxit. The server sizes for discuss.online way outpace the daily load. Usage is typically well below 5% on all servers. Lemmy has become much more memory efficient, and I've made other adjustments to improve performance. Memory usage hangs around 10% for the backend and 30% for front-end servers.

The target is to reduce to better align with current donation payments. Server costs are over 10x the monthly donations right now.

Let me know if you have any issues. Please, remember to check https://status.discuss.online for updates during the outage.

Thanks, Jason

If you noticed some strange site behavior, it could have been caused by some load testing I performed.

A misconfiguration in rate limiting caused all users to share the same bucket. This has caused strange behaviors for some users.

I've stopped load testing and adjusted rate limits to prevent further disruptions.

Let me know if you still experience any issues.

You can always email: hello@discuss.online if you have issues logging in.

Thanks, Jason

We're currently running the latest release candidate of Lemmy 18.2. It has many fixes; however, there are still some remaining issues.

One example is that some buttons do not appear correctly on the dark themes. I will keep my custom dark theme as the default theme for now. I think it just looks better, even with broken upload buttons. I will attempt to apply a fix to it later today.

If you see any other issues, please, let me know so I can forward them to the core development team-- if you don't wish to do so yourself.

Thanks a lot! -jgrim

We've updated Lemmy 18.1. There may still be some issues; however, I believe it resolves much more than it creates. Let me know if you see anything funky.

Need moderators

Hello all!

I've created several communities here on discuss.online to help seed the content for growth.

I'm looking for mods to help water them! Gardening analogies are starting to grow on me.

Please DM me if you are interested.

Lemmy isn't known to have the best moderation tools. I have a somewhat useful way to moderate through Grafana. I'll provide dashboards and the ability to create alerts for you.

Community list

Here is the list of communities I'm looking for moderators for:

• ~~Neurodivergent~~
• ~~BattleBit Remastered~~
• FlipperZero (Unofficial)
• ~~Apple~~
• ~~Linus Tech Tips (Unofficial)~~

Please reach out even if you are slightly interested.

For communities already with mods more can be added. Please, reach out to the listed mod to apply.

Create your own!

I have community creation open. Feel free to create and moderate a different community.

Did you know:

It's possible to create blogs on Lemmy.

1. Create a new community.
2. Check "Only moderators can post to this community."

This will allow you to create your blog posts while allowing others to comment.

Thanks, Jason

This is happening from 8:45 pm ET until 9:30 pm ET.

For status updates go to https://status.discuss.online.

The update will take from 5 minutes to 1 hour.

You may have intermittent access to the site and API.

There has been a release of Lemmy version 0.18.

There is a very long list of changes for the back end and front end. You can see them here: https://github.com/LemmyNet/lemmy/releases

I do plan to upgrade at some point. However, I'm considering waiting until Lemmy 0.18.1. In 0.18 they removed captcha support. This has been a very useful tool to keep bots from taking over the instance. If I upgraded to 0.18 now I'd have to turn on registration applications. I fear that the registration application would be too much friction to gain users.

Please, let me know what you think of this change. An announcement will be made once an upgrade is to take place.

You can view instance status and maintenance schedules here: https://status.discuss.online.

On June 19, 2023, I setup scaling for the environment. My experience doing this is with AWS; however, I host this service with DigitalOcean. The good news is I now understand how DigitalOcean Load Balancers work. The bad news is I had intermittent outages throughout the day.

Sorry, I'll do better.

• jgrim

I noticed some random slowness when using the web client. It looks like the web client's backend is significantly CPU hungry. I decided to segregate the front-end and back-end clients. I can already tell a significant difference.

Here is an update from the core developers of Limmy. It addresses a lot of open questions and concerns by the community. It's a great read.

Discuss.online is ready to scale. I have separated services into dedicated nodes.

I had set up object storage for image uploads. Dedicated database cluster. This all costs more but helps prevent a significant outage if there is sudden growth.

Other communities had to react to a large influx. I thought I'd get ahead of it to keep you all happy!

Be sure to share with your friends! We're ready for them here at discuss.online!

A matrix instance has been setup for the domain discuss.online. This allows the user of usernames like @jgrim:discuss.online.

The consideration is to offer a matrix username and server access for each member of discuss.online.

This service is still in beta and heavily manual. If you wish to access the discuss.online matrix instance, please ping me directly. The automation of signing up will come soon; however, no release date has been set yet.

Hello,

Currently, community creation is open. Until there is an unreasonable amount of local communities, it will remain open to help grow the instance.

Please, share, create, & discuss!

Hello and welcome,

This is the first post of discussion.online. I'm hoping it's one of many.

I discovered the power & freedom of federated content while using Matrix & Mastodon. I have been sparingly working on a federated social site for a few years. Starting this community would be a great way to start something and learn.

Reddit has lost its appeal for me. I no longer want to doom scroll nonsense and repeated content. I hope the communities within The fediverse fill the gap and return the internet to the people.

There is a shift. I'm part of it. Please join me.

-Jason