The number in the Listen method is not the maximum number of active connections but rather the number of pending connections allowed. A pending connection is one that is just starting up. If a second connection tries to connect, and the number is 1, then it will be rejected. If the Listen queue size is greater than one, then additional connections will queue until the older ones are completed.
This just involves starting a connection, not the maximum number of simultaneous connections.
I usually set the listening number to 5. Completing an Accept() is milliseconds.