Load Balancing issues...

Let say you design a distributed software application, and you decided to add a load balancing feature on it. Probably, you'd like to consider network latencies, rules, etc. This load balancer will forwards connections to appropriate endpoint. Then client starts to use that endpoint. But what about binary transfers? How you can handle, let say, file transfers? I mean, if you dynamically forward client to different endpoints each time, how those r/w cycles could be handled for file/data sync. Isn't it creates additional network costs and decreases the benefits of loadbalancer? Do you have any idea? Please mail me your thoughts.