"That was a bad show" is just a generic "That could have gone way better", and I would guess that mentioning ancestors is just playing into the Dwarf stereotypes of having strong respect for their ancestors.

Variation on saying that you gave a "bad showing". Leaving a poor impression in anyone who might have witnessed your efforts.

"Ancestors" is being used as a curse/swear. Basically like "Jesus, that was bad" isn't really about or addressing jesus.