uncommon questions - inflationary perturbation and the ground state

Inflationary perturbations, as we observe them today in the CMB, originate from quantum fluctuations in the early universe. Those were ground-state fluctuations (the Bunch Davies-vacuum), which is Lorentz-invariant due to the Lorentz-covariance of all (known) quantum theories. Why is it the case that the CMB appears different to observers with different relative velocities, if vacuum fluctuations would appear identical to all observers?