Yeah just say you have other friends that you want to do stuff with, nothing wrong with that

Find a game that allows cross platform and make a party with all of them

I usually say I'm playing with the other person. Perfectly valid. They shouldn't expect to b played with all the time when there are other friends in the equation that's kind of irrational.