Changing the default agent for a library only affects newly added media. Any existing media retains the agent with which it was originally matched; so, refreshing metadata continues to use that agent.
To change the agent used for an existing item, you have a couple of options. If you have relatively few items to which you need to do this, you can use the Fix Match option:
In the search options, choose the new agent which you’d like to use.
If have many items for which you’d like to change the agent, set the library’s default agent to whichever you’d like, and then perform the Plex Dance on them. This will cause the items to be removed from your library completely and re-scanned and matched by the new agent.