You're not going to them. They're coming to you.
You're a normal guy/girl with an abnormal problem. You've unknowingly summoned them from their worlds and now you're stuck with them. But who says that's a bad thing?
When issei was slaughtered by his girlfriend... Instead of Rias appearing, Lucifer appears.
You see, the magic circle that Rias gave was mistaken for the one that Lucifer was given.
"Interesting...