Have you tried the call on a lower volume?
I don't know how portable speakers work but I believe it detects when the person is talking and slightly disables the microphone to prevent a feedback loop...once they stop talking it re-enables it fully to pick up your voice and silences the speaker. Since its a speaker...you wouldn't be with your mouth right on it so it'll have to be sensitive enough to register your voice at a distance...so between that change in TX and RX there may be a slight second where they're both working at the same time. I am not aware on where this type of problem is worse or better in other devices. On my fathers Galaxy S5 when he has it on speaker...he has to yell sometimes to hear him well, as if the phone "dims" the microphone so the speaker output doesn't go back into the mic. (hopefully I'm making some sense)