Ouch!!! if he's getting more attached to you, he'll keep getting more attached - the more you wait, the worse it will be.
If this makes it easier, think of "not telling him" as "leading him on", in a way it is.
It's not about giving him a logical explanation... if you just know you don't want a "relationship" then be straight up about that. Sometimes we need a reminder what the deal was, IF the understanding was of course really clear to begin with.
Even in a friends with benefits situation, we are pack animals and tend to get attached, it's only human and natural. And as long as you're with the not-really-a-relationship guy, you're making yourself unavailable to the possibility for a real relationship.
But the longer you stay with him, the hard it's going to be to separate later.