Preheat oven to 350°. Grease an 8-in. cake pan. Make cake: In a large bowl with a mixer on medium speed, beat butter and sugar until fluffy. Beat in eggs, buttermilk, and vanilla. Scrape down sides of bowl and slowly beat in remaining cake ingredients until smooth. Spread batter in prepared pan.
Bake until a toothpick inserted in the center comes out with only a few crumbs clinging to it, 25 to 30 minutes. Cool pan on a rack 10 minutes, then invert cake onto rack and let cool completely.
Meanwhile, toss pitted cherries with sugar and brandy in a medium bowl and set aside.
Make cream: Whisk cream and sugar in a large bowl until soft peaks form. Whisk in brandy until stiff peaks form.
Cut cake in half horizontally. Set half the cake, cut side up, on a cake stand and top with cherry mixture. Spread three-quarters of cream over cake and cherries, then top with remaining cake layer, cut side up. Swirl remaining cream on top. Garnish cake with whole cherries.