A nice old photograph for sale on eBay today - an 1864 view of the Harbour (with a couple of others). Not sure whether to bid yet, but have a look for yourself:
Notable things (notable to me anyway): the wall on the left surrounds the railway turntable, which I think is where the current roundabout is outside the Pavilion. The Pavilion itself is another 40 years away of course, so we see a ship being built and beyond that either a timber yard or, more likely, a chandlers with masts and spars ready for the fleets of ships visiting for refitting and stocking. No leisure use of the sands here - the shore is just a workplace near the sea. The industrial age is creeping in though - note the mix of sails and steam funnels among the larger ships in the harbour.
If you fancy bidding (and I'm not the seller, I promise), or just want to see what it made, it's item 230170405763, ending 6.30 this evening (Thursday 20th).
Sold for £27.00, so clearly to a railway enthusiast!