You need to delete every thing that has been allocated with new. It's as simple as that.
In practice, however, it can be complicate to ensure that nothing is left behind (undeleted). There are many techniques and guidelines to follow in order to help you with that. I wrote two lenghty tutorials on two aspects (or set of guidelines) that can help you, the first is on RAII and the second is on ownership relationships. They may be a bit advanced, but if you can ignore the more complicated things, and stick to the essentials, that is, make sure to use RAII classes as much as possible and use smart-pointers in a smart way. Also, by and large, you should prefer local variables (on the stack) as opposed to variables allocated with new.
I see. Thank you.
If I fail to delete everything, what would it look like? When I close my program (which works perfectly until I close it), the window disappears, but the process (in Windows Task Manager) stays active until I end it.
If the program finishes and there is still memory that wasn't deleted, it won't matter. The memory gets allocated by a thing called the "heap", which is a part of the program (something the compiler generates for you). This heap is like a manager of memory, it allows you to allocate and deallocate memory (via new and delete). This heap gets its memory by asking the OS for it. When your program ends, the heap is destroyed along with all the memory that it is managing, so it doesn't matter if there is still some memory that wasn't deallocated (by deleting it in your program). As a double safety, if the heap is also corrupted (which can happen, depending on the system, if you ask it to delete memory twice or if you ask it to delete memory that it doesn't own or that wasn't allocated before), then destroying the heap at the end of the program might not be able to release all the memory that it requested from the OS, if that is the case, it won't matter either because the OS also keeps track of what memory was given to which program, and if your program ends (end of the process), the OS will reclaim all the memory that was allocated to it.
What you do have to worry about, however, is having memory leaks (memory that you forgot to delete) within a running loop. The effect of this is that the memory consumption will grow incrementally at each iteration of the loop, and eventually you will run out of RAM capacity, in which case the OS will start to use all the RAM plus memory from the hard-drive (virtual memory (in Windows), or the swap drive (on Unix/Linux)). When that happens, your program, along with the entire system, will slow down tremendously. At this point, you must terminate your program to release all that memory, and your system should be able to recover quickly from that (unless you have a really bad OS, like some (or all) versions of Windows). If you don't terminate the program, your program will keep on eating up all the available memory, at which point your system will likely crash, or, at the very least, your program will crash, and at worse, you'll get a blue screen of death.
So, in any case, you should always make sure you clean everything up, it is just good practice to do so. Developing this habit (and using the guidelines and techniques like the ones in my tutorials) will make sure you write robust code that can easily run for days without any problems. For instance, I sometimes program simulations that can run for several days, if any memory was leaking from this code, I would be in big trouble (it isn't fun to start a simulation that should last for about 2-3 days, then leave it to itself, and find out 2 days after that it crashed after the first hour!).
If you mean the problem about your process remaining active after your program ended, then I have no idea where that comes from. You should maybe post some code (please reduce it to the essential, in which the error still appears).