Answer without guarantee:
In principle, you can pass through any addressable device to a VM, especially standalone PCIe devices.
As far as I know, your motherboard must support SR-IOV in order to pass a port to a different VM.

But why would you want to do that? Network virtualisation or "software defined networking" is one of the great advantages, and not just of VMs... And I don't think you'll suffer a major loss of performance through virtualisation either...

But perhaps you have simply expressed yourself incorrectly: If you do NOT use passthrough, you can assign any available network port to any VM...