Don't worry about it. Highly unlikely to be damaged. You likely have bigger problems to solve.
The voltage entering your home varies from macro issues likely when, say, everyone puts the kettle on during a sports commercial break That voltage is further varied by the big electrical devices in your home,say the compressor in your refrigerator coming on/off.
The mains voltage to USB converter - e.g. Anker Powerport - will age faster if it is left plugged in, and indeed a sufficiently big spike will cause it to permanently fail. More common spikes will cause the Anker Powerport to turn itself off to protect it and your USB device and you then have to unplug and replug the Powerport.
Some of the spikes, the rarer greater spikes, the Powerport cannot shield from its USB device and it will cause your USB device (e.g BT speaker) to fail. These rarer spikes you can significantly reduce by buying a 3-pin mains surge protector.
In an opposite problem you have the fact the lithium battery inside your BT speaker is slowly oxidising and losing battery capacity over time. This battery aging is faster if: kept hot or cold; deeply discharged and recharged; kept fully charged. So if you are keeping it plugged in it is kept fully charged so it is contributing the battery aging. However, as you are keeping it plugged in, then it is not being expected to power itself from its battery so the battery aging is mute.
- any expensive or important items, use a 3-pin mains surge protector, ideally from a manufacturer which offers insurance as then they implies it is better designed.
- Anker Powerport is a good choice to protect your USB devices, it has protection built in (to the limit of what a small device can offer).
- in general you should not worry about your BT speaker left plugged in.