When you get to a different way in the warehouse where you see Malix, Harold Anderson will tell you that the father didn't get along with him because the father found out he was studying demon magic years earlier. The father never went to his house because of Harold studying demon magic. Harold tried giving you many toys, but the father threw most of them away. He was happy to find you when you got lost so he could finally see you. He got the mother to get the father to let you go over to Harold's house. He also brought the incubi to his estate. When you find him, he told that he didn't expect the father to let you move in so soon. He also didn't tell the incubi about it being his house. In Seduce Me the Otome, if you go for Damien, it will be soon revealed that the most powerful warlock in the human world, aka your dear Harold, helped the incubi brothers out of the Abyssal Plains.