For what it’s worth I experienced a similar problem.
I found by trial and error that i only had the problem when the controllers were not synced to the first four colours on the base station.
Base station buttons colours are : 1. Red 2. Green 3. Blue 4. Yellow 5. Orange 6. White
i.e. Originally synced the controllers to Red / Green / Blue / Orange (colours 1, 2, 3, 5) then I had problems ( I used those colours to match the wrist straps on the controllers)
When I synced Red / Green / Blue / Yellow ( colours 1, 2, 3, 4) then no problem.
This May be a complete red herring but maybe they need to be sequential without gap if only using 4 cars - worth checking.