Member Avatar for Tom Tolleson

Hi! I just inherited a rather large legacy site here at work that has no database behind it. It's a large volume of HTML pages with the content written right into the HTML page. I need to extract the content and bring it into a database, or XML files.

Each section of the HTML pages has header tag and a standard title, so I'm thinking I should write a perl script to parse the pages based on header tags and insert them into MYSQL.

Before I begin, I thought I'd check with you guys to see if you have had any similar experience and recommendations.

Thanks!

Tom Tolleson

  • 3 Contributors
  • 2 Replies
  • 98 Views
  • 4 Days Discussion Span
  • Latest Post Latest Post by MidiMagic

Dani AI

Generated

This kind of migration benefits from a repeatable pipeline rather than ad-hoc edits. The quick fixes mentioned by and can work on a tiny, uniform set, but regex or manual find/replace is brittle when tags are nested, templates vary, or the HTML is malformed.

Begin with an inventory and a small sample set, then run a controlled pipeline:

  • Identify distinct templates and the header/tag patterns that reliably mark content.
  • Normalize malformed HTML (run through a Tidy pass) so parsers behave predictably.
  • Parse with a tolerant HTML parser or convert to well-formed XHTML and use XPath for extraction.
  • Map extracted pieces to a schema or an XML structure, keeping a field for raw HTML as a backup.
  • Insert via prepared statements inside transactions; batch operations for speed.
  • Validate results on a representative subset, iterate, then run full import with logging and checkpoints.

Use robust Perl tools rather than brittle text tricks. For cleaning use HTML Tidy. For parsing consider HTML::TreeBuilder or a streaming parser like . For XPath-based extraction after normalization, XML::LibXML works well.

For MySQL writes use DBI with placeholders and transactions (DBI / DBD::mysql). Pay attention to character encodings (meta tags and HTTP headers), relative URLs and image paths, duplicate detection, and preserving originals so the import is repeatable and reversible.

Run small, logged trials; refine extraction rules per template; keep the process idempotent so fixes can be re-applied without duplicating content.

Recommended Answers

All 2 Replies

Member Avatar for pritaeas

I've done the same, only with PHP. Using a regex I stripped out the actual content and put it into the DB. You may need to escape the content, but that depends on your insertion method and column type.

Member Avatar for MidiMagic

I did it with notepad.

I used find and replace to replace each tag with either nothing or the separators needed to import the data into the database.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.