Probably yes, it depends on your threat model.
If you are using E2EE on a matrix.org account then your message content, attachments (images) and most other traffic isn't accessible to anyone but the people in the chat. However Matrix isn't the most private option, it has a number of leaks such as reactions and chat topics (these are being worked on but aren't close to happening).
For most people Matrix is a very private and secure option and the fact that it is federated is a huge plus. If you want something more secure you are probably looking at Signal (which you don't want to use and isn't federated) or Simplex Chat (which doesn't have multi-device support).