That's one way to narrow it done.
Let me share I was on a team behind a motion detection system used for doors. I see it all over the US today even 17 years later. It works but we didn't use OpenCV. It's a custom single board design without an OS. Many call them embedded computing.
Anyhow, the entire system works with 2 cameras (one for each side of the door) and in short we capture an image then run it through a Sobel filter to get edges. No pixel method here. Just count edges. When the edge count changes in a zone (group of pixels) then we flag up as movement in that zone.
Got it?