I see red flags. If you even think you might have feelings for someone beyond friendship or enjoying their sexuality then you shouldn't go for it. Swinging and threesomes should not have any emotional component, that is poly amorous. I'm not saying poly amorous-ism is bad, I just don't think that is what anyone wants in your situation.
As far as the friend, well, you just have to trust your gut instinct because he's the only one who really knows. And always remember to go with your gut, because your heart will lie to you and your brain will try to make logic out of something in which there is none. The gut is the only thing that you can trust.
If you are to do anything, make sure it is as a couple, never alone. Even a simple phone call can lead to mistrust and troubles if it's kept in the dark. Tell him straight out, everything that is said or done when your husband is not present will be brought to your husband's attention. That way there is never any doubt that things are on the up and up.
If I were you, I would leave it at a flirtatious friendship just so there is no chance that anything else could happen. Sometimes people who are divorcing will do things they ordinarily wouldn't, so just be careful.