So I've been running self-hosted email using Mailu for a couple of months (after migrating out of Google Workspace). Today it turned that although my server seems to be capable of sending and receiving emails, it also seems to be used by spammers. I've stumbled upon this accidentally by looking through logs. This seems to have been going on for all this time (first "unknown" access happened just a couple of hours after I've set everything up).
While browsing the logs there were just so many crazy things happening - the incoming connections were coming through some kind of proxy built-in to Mailu, so I couldn't even figure out what was their source IP. I have no idea why they could send emails without authorization - the server was not a relay. Every spammy email also got maximum spam score - which is great - but not very useful since SMTP agent ignored it and proceeded to send it out. Debugging was difficult because every service was running in a different container and they were all hooked up in a way that involved (in addition to the already mentioned proxy) bridges, virtual ethernet interfaces and a jungle of iptables-based NAT that was actually nft under the hood. Nothing in this architecture was actually documented anywhere, no network diagrams or anything - everything has to be inferred from netfilter rulesets. For some reason "docker compose" left some configuration mess during the "down" step and I couldn't "docker compose up" afterwards. This means that every change in configuration required a full OS reboot to be applied. Finally, the server kept retrying to send the spammy emails for hours so even after (hypothetically) fixing all the configuration issues, it would still be impossible to tell whether they really were fixed because the spammy emails that were submitted before the fix already got into the retry loop.
I have worked on obfuscation technologies and I'm honestly impressed by the state of email servers. I have temporarily moved back to Google Workspace but I'm still on the lookout for alternatives.
Do you know of any email server that could be described as simple? Ideally a single binary with sane defaults, similarly to what dnsmasq is for DNS+DHCP?
Oh no! I'm sad to see that you've run into troubles :(.
There are other "fully put together" solutions like mailinabox and mailcow, that could be worth looking into for you. I haven't used them personally, but you might find them worth looking into. I'd never heard of mailu before, actually.
Totally understand the desire to just move to a hosted solution after running into these problems, but even if you do that I think you should keep running a mail server in the back of your mind for the future
you've already learned a lot about it I'm sure, and maybe with a bit more experience you'll be ready to tackle it again :).
I don't actually use any of the fully assembled solutions like mailinabox, and I wonder if in the future it might be a good idea to try configuring everything manually. You already have some familiarity with how mail works at this point, and having more control over the setup and how everything fits together might actually work out for you. Personally I'm running an OpenSMTPD + Dovecot mailserver and having a great time. I'd recommend it.
https://poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/
Either way, I think you should keep using a custom domain for e-mail because then you have options in the future :).