The center-of-mass reference frame is the frame that moves with the system's center of mass.

If the center of mass has velocity

$$\mathbf V_{\rm CM}=\frac{\mathbf P}{M}$$

in some inertial frame, then each particle's velocity in the center-of-mass frame is

$$\boxed{\mathbf v_i^*=\mathbf v_i-\mathbf V_{\rm CM}},$$

where the asterisk denotes quantities measured in that moving frame.

The total momentum in the center-of-mass frame is

$$\mathbf P^=\sum_i m_i\mathbf v_i^.$$

Substituting the relative velocities,

$$\mathbf P^* =\sum_i m_i\mathbf v_i -\mathbf V_{\rm CM}\sum_i m_i =\mathbf P-M\mathbf V_{\rm CM}.$$

Since $\mathbf V_{\rm CM}=\mathbf P/M$,

$$\boxed{\mathbf P^*=\mathbf0}.$$

Thus the defining dynamical feature of the center-of-mass frame is that the system's total momentum is zero.

Example

Consider two objects moving along one line:

$$m_1=2,\mathrm{kg},\qquad v_1=5,\mathrm{m/s},$$

$$m_2=3,\mathrm{kg},\qquad v_2=-1,\mathrm{m/s}.$$

Their total momentum is

$$P=(2)(5)+(3)(-1)=7,\mathrm{kg,m/s},$$

and their total mass is

$$M=5,\mathrm{kg}.$$

Therefore

$$V_{\rm CM}=\frac75=1.4,\mathrm{m/s}.$$

In the center-of-mass frame,

$$v_1^*=5-1.4=3.6,\mathrm{m/s},$$

$$v_2^*=-1-1.4=-2.4,\mathrm{m/s}.$$

The momenta are

$$p_1^*=(2)(3.6)=7.2,\mathrm{kg,m/s},$$

$$p_2^*=(3)(-2.4)=-7.2,\mathrm{kg,m/s},$$

so

$$P^*=0.$$

The objects are not individually at rest; their momenta cancel.

Why this frame is useful

The center-of-mass frame removes the translation of the system as a whole and leaves only motion relative to the center of mass. In collisions, this can expose symmetries that are hidden in the laboratory frame. In two-body dynamics, it separates overall translation from relative motion.

For an isolated system, the center of mass moves with constant velocity, so its reference frame is inertial. If an external force accelerates the center of mass, the corresponding accelerating frame is not inertial and Newton's laws require additional care.