Actually sounds like a bad solenoid - I've had the same problem on cars and boats (although never on a bike)...sometimes a poor connection (open/short) can reduce the volts/amps that reach the solenoid. Check the simple stuff first! A loose connection would make for an easy fix. Good Luck.
+1 with Juggs. Split the control box on the handlebar (2 screws) and spray it with WD-40 inside. Then try it. If the starter kicks in first try it's your button. If not, the problem is elsewhere. Simple easy test.
This thread is very helpful as I am having the same issues. However it is difficult to tell from the pictures the exact location of this electrical terminal. Any clarification as to its location would be greatly appreciated!!