I run a full blown home server rather than a raspi server so take my opinion with a grain of salt. But Ive heard of alot of people say yes thats possible. But i can imagine it overloading very quickly though. Not the same, but the VPS im running my lemmy instance from couldnt handle the matrix server i was testing with bots and other integrations (2 vcores, 2gb ram). You could get away pretty well an i3 or i5 if power consumption isnt a massive concern, doesnt even need to be new hardware either. You might even find a second hand machine for as much as a raspi as well.
Being a desktop CPU, you will have more available power should you outgrow your current needs from an SBC/raspi.
An additional advantage is if you are hosting your own videos, you can use quicksync on 8th gen intels (and onwards) without needing a gpu, and you can transcode multiple streams at once.
Infinity for lemmy has been released on izzyondroid/f-droid. Pretty much exactly like its reddit counterpart in features and was my preference over sync. Si im happy it works for lemmy :)