I was curious on this one, so I did a little research...
According to the Mayo Clinic - Frequent masturbation does not reduce your sperm count nor does it affect your ability to achieve an erection.
According to malefertility.md - Theoretically, this can decrease the sperm reserve in an individual with a low or low-normal count and thus reduce semen quality at the time that the highest quality is needed - when the wife is ovulating.
Here's my opinion...
It only takes one, so a few thousand here or there really shouldn’t make a difference. Other life factors have a bigger impact on fertility such as smoking, drug use, alcohol consumption, and exercise.