What you have would be classified as moderate ED (Erectile dysfunction). Clinical studies have found that about 60% of men show considerable improvement with Korean Red Ginseng. Unless a person is on blood thinning medication, there's really no reason not to try it. If you try it, the research was done only on Korean Red Ginseng, which has been found to increase testosterone.
FYI, some prescription medications can cause the difficulty you're describing. SSRI's, a class of common antidepressants, are notorious for this. Also, some medications used to treat high blood pressure can cause this problem.
If you're not on any of these medications AND the ginseng doesn't work then you'll need a medical workup. Things that need to be ruled out include low testosterone, diabetes, and thyroid problems.
We used to think that erection difficulties were mainly psychological. Now we know that it usually has an underlying biological basis. I would try the natural route first, and if that doesn't work get a medical workup to make sure there's not an underlying problem.
Guys are reluctant to talk to doctors about these things. In reality, treatment exists and there's absolutely no reason to be frustrated by a problem like this. I would 1. make sure it's not a medication side effect. 2. Try the ginseng. 3. See a doctor.