Why does Genji want the Akashi lady to move closer to him?

The Akashi lady has given birth to Genji's daughter. Genji wishes to spend more time with his daughter and decides that the proper course of action would be to move the Akashi lady and his daughter closer to himself. The Akashi lady does not want to move, therefore she must give up her daughter to live with Genji.