I'm guessing you used hot glue to glue the xover components to the xover board; right? If yes, you can use your hot glue to also glue the board to the speaker cabinet.
One thing I would do if I were you is zip tie your inductors to your xover board, those are usually what can potentially break loose when you move the speaker around just because they're a tad heavy for just using hot glue.
I generally screw the xover to the cabinet using 3/4" long woodscrews. But I have used hot glue successfully too, it's just harder to remove if you ever needed to for whatever reason.