I agree, I think if you invite someone who is an ex-jounin, the Inf cost is doubled. This will fix people gaining INF from jounin, and if you think of the actual meaning of influence then this makes sense. If you are forced to add some really strong guy to your Village, then you admit that your village may be struggling a bit offensively, which could result in you loosing even more influence than usual.