Well, Doesn't sound like the engine or the genny has a good ground to the frame, but you did a work around with the separate ground wire to resolve the issue, so it probably isn't the cause of the present problem. A good ground from the headlight bucket to the engine or good chassis ground is always a good idea. Grounding through the head bearings tends to shorten headlight bulb life. It acts as an intermittent connection, causing stress to the filament. Again, not your current worry.
Well, if the indian experts say a negative reading is o.k., I'll believe them, But, it says to me that the genny doesn't make enough to meet the headlight's draw. But, Hey, what do I know, I'm 1500 miles away
Just as an aside, the relay isn't going to open 'till things get really bad, so not opening isn't a much more than a "go-no go" indicator.
Try this; disconnect the headlight, power it up directly from your battery with the ammeter in the line, check current draw. Write that down. ( well, if you're over 40 write it down). Multiply by 6 to get the watts, or power draw of the bulb. See how that compares to stock bulb. Of course, if you have some Indian friends, they may have already told you that the headlight bulb you're using is something they've tried and works o.k. for them. If so, I'm running out of quick ideas.
I keep coming back to the genny not making enough to do the job. The Voltage readings are somewhat confusing, but the basic fact seems to be that the coil is starving for electricity when the headlight's on. That's the problem. Either we're not making it, or it isn't getting to the coil when the light's on, and you're running down the road.
How about this:
You've made the road run, the bike starts cutting out, and you stop the machine. Still have a problem? Does motion down the road enter into the equation?
Make a run, create the problem, head for home, leave it running, check your voltages, still good?
Can you sit there with the meters hooked up and turn the light on and off and create the problem and make it go away?
Another question, is the ammeter you're using part of the bike's equipment? Or a digital one you're hooking up to take readings?
That's it for now, answer me these, said the Riddler...
DD