Just thinking out loud here:
what if we extrapolate the scores to a scale of ten? for example, player A wins it with 20 votes, players B,C and D have 18, 10 and 6. The winning player always get's ten, wich means we have to divide his score by 2. we do the same with the rest so it's: 10, 9 5 and 3 points for the rest.
That way, if a player really stands out, he'll be rewarded for it quite a bit. But players scoring some points consistantly won't get 50 points behind just for one game.