But then you've created dozens (or hundreds) of opportunities a day for someone to get your passcode by shoulder-surfing, which you probably wouldn't even notice in many situations. I'd argue that unless someone forcibly borrowing your face or fingers to unlock your phone is a strong possibility, entering a passcode each time is less secure than using biometrics.
Especially since the passcode also protects various security settings.