OpenVPN not changing my IP address

When I installed the OpenVPN client and connected to a VPN it changed my IP address correctly. However, subsequent times it didn't change it. It turned out I had to run the application as an administrator, and that was it.