I would consider accepting the request, and there is one very important reason for that: LinkedIn is not just about your first level connections, it's about the power of your network. And in my opinion, it's your second level connections that actually drive this network.
It's not just about who you know, it's also about who your contacts know. To make this clearer, ask yourself the following questions:
- If the recruiter for your dream-job or the perfect candidate for the job that you are hiring for is a second level connection through this "ghost", would you consider reaching out to him?
- You had a common social circle. If a good old friend from that circle discovers you through this person, would you consider connecting with the "ghost" worth it?
- Thinking from the reverse side: If some time later this person is looking for a job and you are in a position to help (either directly or by making a connection), would you consider helping him out?
If your answer to these questions is "Yes", then do consider connecting. If things are really bad between two of you, then, by all means, ignore the invite. But the fact that you posted this question here instead of directly ignoring the invite, indicates that things may not be that bad.