When you take the Elevator to get down to Hollowtown Island you do not go towards the Ghostlady but the other way and you need a Key that you get very late into the main Quest and I don't know if it missable but either way you can get back down to where it is via a Mirror on the Binding Light.

how do you get the key for this door? Is it someplace after I get the airship?