The dropping of protection occurs at attack send, when you select to do so and issue the attack.
Attacking a ghost with anything other than a CS does not drop protection, meaning that because you didn’t spark the drop upon attack issue, you didn’t drop. The travel time of the unit and hitting after the ghost flipped is irrelevant.