Why may it be problematic for Genji to take care of Tamakazura?

Tamakazura becomes unsure who she should trust. To no Chujo wishes for Tamakazura to stay in Genji's care; however this is problematic because Tamakazura is now left in the care of a man who is also trying to seduce her.