I don't really see him as being intentionally cruel to her, especially once he came to know her at all. I think he is often clueless at her and doesn't always explain himself well. Sometimes he has been harsh when he thinks that protects her.
She is also sometimes the same right back.
They share a love for books and research and knowledge and enjoy simply being in each other's company.
They have protected each other and know inner secrets about each other that no one else knows. This has created a (non-sexual) intimacy in their relationship. In a world where adult men and women aren't allowed to be close friends, it is a challenge to find a path where they keep this where they aren't officially paired off. Thank goodness we don't live in a world like that ourselves, any longer anyway.
Normally I would squick about the age difference and worse the mentor/student relationship but it helps that (a) Rozemyne is actually older than him if you count both lives and (b) Ferdinand has never shown any romantic overtures at any time to anyone during the story. Also, at this time she actually has power over him in multiple dimensions. She literally has his life in her hands and she was urged by a god and even Ferdinand himself to kill him so she would receive the full book of Meistionora. And she now outranks him politically and socially, and has plenty of alternatives for partners if she will so choose.
I also think Ferdinand is pretty confused about what Rozemyne is thinking. There's a hint that he suspects she is choosing him above all others, but when he asks in the indirect noble fashion she refuses to answer. She's got the royal family amulet. He's having trouble making sense of this and when he can't figure that out his trust level plummets.
This is complicated by the fact that Rozemyne does not herself really seem to want to answer these questions, not even to herself.
It's only when he decides to put himself back in the familiar role of her guardian that he can tease out some of the answers about what she doesn't want, and from there I suspect he's putting some pieces together about what she might want and how to get her there. Like that he's talked her out of going through with any integration into the Royal Family.
If they do ship in the end, I suspect it will be a relationship of friends and equals.