Member Avatar for youlichika

I have ordered yahoo news rss, I use simplepie to separate the news items like title, content, url, date... and then insert them into database. I catch the source every 2 hours with cron. How to judge RSS source, if there has new then UPDATE into database?

I mean if the rss source has 10 items, 5 are old, 5 are new, I want just update the 5 items new and ignore the 5 old items. I do not want insert the repeart items.
And I want to get `echo` a list of the 5 new items.

This is my insert code.

mysql_select_db("rss",$db1);
mysql_query("INSERT INTO yahoonews (link, title, date, content, image, imagelink) VALUES ('".$link."', '".$title."', '".$date."', '".$content."', '".$image."', '".$imagelink."')");
  • 3 Contributors
  • 2 Replies
  • 106 Views
  • 22 Hours Discussion Span
  • Latest Post Latest Post by mschroeder

Dani AI

Generated

Short practical plan for that builds on the replies from and and gives a reliable, low‑work way to only insert new items and list them.

Start by treating each feed item with a stable identifier (GUID or permalink). Store that identifier in the table with a UNIQUE index and keep a small fingerprint of the content (normalized text -> hash). This lets the database reject exact duplicates while still letting you detect content changes.

Example schema pattern (illustrative):

CREATE TABLE yahoonews (
  id INT AUTO_INCREMENT PRIMARY KEY,
  guid VARCHAR(255) NOT NULL,
  link VARCHAR(255),
  title TEXT,
  content TEXT,
  pubdate DATETIME,
  content_hash CHAR(40),
  last_seen TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  UNIQUE KEY (guid)
);

Use a single upsert for each item:

INSERT INTO yahoonews (guid, link, title, content, pubdate, content_hash)
VALUES (?, ?, ?, ?, ?, ?)
ON DUPLICATE KEY UPDATE
  title=VALUES(title), content=VALUES(content), content_hash=VALUES(content_hash), last_seen=CURRENT_TIMESTAMP;

To produce the list of new items for display, either (a) before inserting, check which GUIDs in the incoming feed are missing (one SELECT WHERE guid IN (...)) and only insert those, collecting them to echo; or (b) after each upsert, detect newly inserted rows by checking the insert response (or by setting a created_at column and selecting rows created during this run).

Notes and cautions: normalize and strip insignificant HTML/whitespace before hashing to avoid false positives; do not rely only on pubDate (feeds can have bad dates); protect operations with prepared statements and a short DB or file lock so two cron runs do not race; keep a fallback dedupe on link/title if GUIDs are missing or unreliable. This pattern gives durable deduplication and an easy way to echo just the new items.

Recommended Answers

All 2 Replies

Member Avatar for pritaeas

Compare the link or title with the date from the rss against your database.

Member Avatar for mschroeder

Create hashes for the common fields that change. I'm thinking the title and content/body of each article. Store these two hashes with each item in your database.

When you process the feed again, generate the same hashes and look for any records where both hashes match. If you find a result disregard the item from the rss feed as it is identical to one in the database already.

When you find a mismatched title or body hash, update whichever of the fields does not match with the changes.

When you find no match on the title or body hash, insert the new item into your database.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.