I didn't invite everyone we knew to our destination wedding, the main reason being that the chapel only held 50 ppl, plus we wanted to stay on budget as best we could. We ended up with 60 ppl there with a few standing during the ceremony.
One person we didn't invite was my sis' sil. After she realized she didn't get an invite she was like, "I want an invite! I want to go!!!" So I sent her an invite as I didn't want to hurt her feelings. Well, she didn't GO! lol! She just wanted to be invited so she could decline.
I'm not sure what you should do, but if you don't want to invite someone, I say don't. You are the one who has to remember your wedding for the rest of your life. I invited that girl when I really didn't want to and I am the one who got jabbed, lol.
There was also one person I didn't want to invite, but felt obligated to. He did go and brought his gf and they proceeded to get married on the same day as ours - by the JoP right before our wedding, and then proceeded to have their reception at our reception since all his family happened to be at ours, too! They showed up in jeans and t-shirts, lol!!! OMG, I hate weddings, I really do. Elope! Elope while you still can!!! Forget everybody else!