I'm not sure you're overthinking. It sounds like you're realistically considering the good and the bad things that could happen.
One thing that is essential is that you are both crystal clear on expectations. You understand that what you're wanting as casual sex. It's just possible that he has been harboring feelings for you and he would get really hurt. Don't assume he's seeing the situation the way you are.
What you're describing is often called referred to as "Friendship, with benefits." In reality, once you've crossed the boundaries of friendship, it's hard to go back.
Are you truly sure it's "just sex?" Would you feel OK if he told you about having sex with someone else? If you're "friends," that wouldn't bother you.
Of course, it's possible that he could be wanting the same thing as you. That's a conversation no one can have except you and him.