Hi Hwa - I see your points. I think there is 'symmetric lag hashing' option that maybe be able to do what you're thinking, but it only work on IP packet and not Layer 2 packets. ICX6k don't support it, but ICX7K do. I don't know if VMware or Linux even support it. So it's not really a good option for load-balancing either.
Since lag hashing result is locally significant, I think even the hashing algorithm is the same on both LACP peer, it doesn't guarantee that the same pair of traffic will always get mapped to the same port on both end. What can be expected is that the ICX would always map SrcA/DstB to the same port as long as that port is up, this could be Port1 depend on the hash result of ICX. And similarly ESX would probably always map SrcB/DstA to the same port as long as that port is up, this could be Port 2 depend on the hash result of ESX.
At the high level on the ICX6K, when the traffic is Layer3 (being routed between VLAN/VE), it uses the IP headers and protocol fields in hashing. When the traffic is bridged/Layer2 (being switched in the same VLAN), it depends on the traffic type, IP and/or Ethernet headers along with Layer 4 ports info are used in hashing. I believe this is a standard method among vendors, and should be very similar between them.
Thanks,
Vu