Samantha: I think you have two separate but interacting issues going on here.
It concerns me that he won't talk to you about this. I don't know what's going on there, but that is the bigger problem. As far as the looking, that is just something that we do. Men are aroused visually, the same way that women are aroused aurally. That's why we don't get swept away by listening to someone sing. I'm sure he does find these other women attractive. It's often an unconscious response.
Committing to one woman isn't something that men do naturally. It is an act of the will, to be faithful to one person. The fact that he is looking at other women doesn't mean that he is or is not doing more. When a woman starts looking at other guys, it's usually because there's a problem in her primary relationship. I'm afraid you're interpreting male behavior from a female point of view. When a guy looks, it just means that he's looking. I don't think that says a lot about the quality of your relationship, one way or the other.
If you think there's more than looking going on with the neighbor, then that's another matter entirely. As far as the flirting, that's just how some people relate. Of course, it could mean he's trying to get somewhere with them. You just can't tell by someone being flirty what their intentions are.
Right now, I don't think there is strong evidence that anything is going on. I kind of suspect you may be putting two and two together and getting six. I think his being unwilling to talk to you is by far the bigger problem here. That could indicate that there is something to hide, or that he simply thinks you're being unreasonable and doesn't want to get into a discussion about it.
I wish I could be clearer, but I just don't think there's enough information for a firm conclusion right now. I will tell you that jealousy is a relationship killer. If he's just looking, I'd let that one go. You can even tease him about it. If you get mad, you just come off as insecure.
He has either committed to be faithful to you, or he hasn't. Looking isn't a sign one way or the other.