Make sure the Allowed-IPs
is as small a subnet as possible. Your device will only route traffic over your VPN that has a destination IP in that subnet.
That way you're only tunneling the traffic that needs to go over it. Everything else will go out the normal route.
Having your device package up and encrypt every packet takes some overhead and will inherently lower your bandwidth throughput, so it's worth minimizing the number of packets that have to go through that process.