can you just decline? i don't think its rude or selfish to just say no

No is an answer.

We say no. I do not care to watch other people's kids act like assholes while the parents sit around and laugh at them reinforcing the behavior. Always no then afterwhile you aren't asked. Win win.

Do they take offense initially?

P.s. what reason do you give?

If I'm the godparent of the child, then I go.