When you do anything that helps move air easier its gonna drop boost a little since there is less pressure. But you make more power in the end anyway.

Id say 4" is probably big enough for your setup now. Maybe do 3-3.5 into 4".