It seems there might be a little confusion here. You have to specify an IP address for the interface itself when you create it:
networkInterfaceSetting.Address = new IPAddress(config.network.IPAddress);
In this step, you must assign an address or get one from DHCP.
IPAddress.Any is no good here. After that, you have to create and bind a socket to the interface to listen for connections:
serverSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
//bind socket to all available endpoints (should only be the ethernet on the board)
ep = new IPEndPoint(IPAddress.Any, config.network.Port);
As you note, in this step, you are telling your interface (to which you have assigned an IP, or gotten one for DHCP) what IP address(es) to listen to for a connection. Specifying IPAddress.Any() here means your client can have any IP address.