I'm no psychologist, but I do love the study of the human mind, and I'm a perpetual student of life...
Well, to me it sounds like you definitely have thought about this a lot, probably ruminating over it for hours, causing anxiety... then the anxiety builds and you get upset about being anxious, and it's just a self-feeding fire of worry and anxiety... I know how it goes.
You have explained that you enjoy playing with the other girl parts, which you cannot do in a monogamous relationship with your guy besides playing with yourself (fun, but can get boring)... I can understand this, and I also understand when you say you like your man's boy parts and don't really want someone else's boy parts... but it is a bit unfair in my opinion, to expect him to let you play (doesn't matter if it's another girl or another guy) and not want him to play too. If it's something that you can both agree on, and I mean him actually agree, not be coaxed into saying he agrees, then it's understandable.... I know some couples where one partner just isn't interested in swinging but allows their mate to play (or so they say..) but for the most part it's a win win situation for all involved parties.. you know you're going home with your man, and you both get to have a little different flavor while you're out... not that you don't both love each other and only want to be with one another, but it's nice to add some zest and spice to the sauce once in a while....
But I feel it is unfair to expect that you can have some spice but he can't.
Generally these issues result from, as you say, a lack of self-confidence. Sometimes people are afraid their mate will be more attracted to the other person and therefore leave them.... this is why it takes a good, strong, open and honest relationship to make an open relationship work.
I think now may not be the time for you to explore your girly love attractions, as you have some emotional issues you need to get settled for yourself. I went through a very rough depression after we opened up our relationship, and during that time all outside play was off.. it wasn't until I had been to some therapy and got things balanced again before we played with anyone but each other. You know what they say, a chain is only as strong as it's weakest link.. and in an open relationship situation, it works the same way. If you're not mentally and psychologically able to commit to it, then the relationship isn't either.
Talk with your man, maybe he can help ease your mind a bit that he does not want anyone else... it works when you know your heart belongs to your partner, but your fun parts can be enjoyed with others... :)
You both have to agree on what is and is not acceptable before you can play. And you have to respect each other's limits and not go beyond them.
This is something that can be very fun and stimulate your intimacy with one another... or it could be a relationship bomb. It's up to you to make it what you want it.