So you have two goals, right?
1. Keep the kids in their seats.
2. You want to focus on speaking and listening, rather than reading and writing.
How about having the kids work in groups, is that okay? Within those groups, they can play all kinds of speaking games. There are board games;
THIS game is old but still pretty popular at elementary school. There are worksheets that only have pictures on them, maybe in a calander format, where you choose a date and the students say the picture or form a grammar-specific sentence using that picture.
There's all kinds of cool speaking things you can do that keep the students in their desks.