If you want an app to allow complete page creation and site design from scratch, there are lots of great apps available. One of my favorites is WebEdit Pro from www.interspire.com. It's relatively pricey, but very robust and professional.
If instead, you have templates or simple sites that you create for your clients, and then you want to give them the ability to edit pre-defined areas of the pages including images or even page content that you store in a databse, I developed an app named SnippetEdit. You can check it out at www.snippetedit.com.
The previous poster mentioned FCKeditor. I plan to incorporate FCKeditor into SnippetEdit instead of the old HTMLarea I use now. Main thing is that FCKeditor is compatible with Firefox as well as IE. FCKeditor is not really a standalone product. Rather it is an excellent component you can use to build a page/site editor.
Yeah, if you are familiar with PHP scripting, and you want something like the rich text editors you see at eBay and at many forums, including daniweb, FCKeditor is an excellent choice. Download it (it is free), and check out the PHP samples. You can customize the toolbar icons to remove things you don't want. The way it comes is WAY too many features for common purposes (in my opinion). You should be able to script together a simple page editor in no time.
Maybe you can post your code to the Code snippets section of Daniweb when you work out your solution. :)
My first guess is that your browser is simply blocking the popup windows. In TinyMCE, when you click to add a link or image, see if you notice a yellowish bar at the top of your browser window indicating popups are blocked. This happened to me. I thought, "this thing doesn't work", then allowed popups and walla, it works.