The hard part is getting used to it. How do I share my public keys? How do I use GPG (the program)? How do I access them easily? What do I do when I want to encrypt my mails on desktop (maybe Windows+Linux), laptop, and phone? It's just relatively much work to gather the knowledge.
+ the fact there're not many people using it
https://www.practicalnetworking.net/series/cryptography/rsa-example/