Using 3D radiation-hydrodynamic simulations and analytic theory, we study the orbital evolution of asymptotic giant branch (AGB) binary systems for various initial orbital separations and mass ratios, and thus different initial accretion modes. The time evolution of binary separations and orbital periods are calculated directly from the averaged mass-loss rate, accretion rate and angular momentum loss rate. We separately consider spin–orbit synchronized and zero-spin AGB cases. We find that the angular momentum carried away by the mass loss together with the mass transfer can effectively shrink the orbit when accretion occurs via wind-Roche lobe overflow. In contrast, the larger fraction of mass lost in Bondi–Hoyle–Lyttleton accreting systems acts to enlarge the orbit. Synchronized binaries tend to experience stronger orbital period decay in close binaries. We also find that orbital period decay is faster when we account for the non-linear evolution of the accretion mode as the binary starts to tighten. This can increase the fraction of binaries that result in common envelope, luminous red novae, Type Ia supernovae and planetary nebulae with tight central binaries. The results also imply that planets in the habitable zone around white dwarfs are unlikely to be found.