Volume change on master when removing SW ?

So Im new to working with SW, i have the plugin on my master with nothing else, when i take the plugin off the volume then increases the master and it clips. If i turn safe headroom off and turn the output to full its then louder with SW on than it is when its off. I am finding it hard to get a balance so when i take SW off the master there is no difference. Does anyone know any tips to achieve this ? :)