DaniWeb Forum Index > Web Development > PHP

::Making Daniweb FAQ Bot::

Please support our PHP advertiser: PostgreSQL or MySQL? Compare and contrast the two most popular open source databases

Last »

•

Join Date: Sep 2007

Posts: 1,456

Reputation: cwarn23 has a spectacular aura about

Solved Threads: 136

cwarn23

Offline

Nearly a Posting Virtuoso

::Making Daniweb FAQ Bot::

Jul 1st, 2009

I was thinking of a new project for making a bot that scans daniweb for information and enters useful information into a database then people can ask the website questions and php+mysql will provide answers in paragraph form for the user to read. Is this ok on the legal side of things as I have written bots for my personal use before and have made a tutorial. Also if this is ok by the mods then any help with the algorithm would be appreciated.

Try not to bump 10 year old threads as it can be really annoying.
Like php then read my website at http://syntax.cwarn23.net/
Star-Trek-Atlantis - now that's what I call a movie ^_^
My favourite PC. - MacGyver Fan
Bad english note: dis-iz-2b4u

•

Join Date: Oct 2008

Posts: 2,612

Reputation: adatapost has much to be proud of

Solved Threads: 463

adatapost

Offline

Posting Maven

Re: ::Making Daniweb FAQ Bot::

Jul 1st, 2009

cwarn23,

Congratulation for the art-work - A sharp, smooth, smart, and intellegent code work.

Failure is not fatal, but failure to change might be. - John Wooden

•

Join Date: Sep 2005

Posts: 1,075

Reputation: digital-ether is just really nice

Solved Threads: 66

digital-ether digital-ether is offline

Offline

Veteran Poster

Re: ::Making Daniweb FAQ Bot::

Jul 2nd, 2009

I've posted the question to the moderator lounge:

http://www.daniweb.com/forums/post90...tml#post905956

www.fijiwebdesign.com - web design and development and fun
Cpanel Email - Let users Register email accounts on your website upon registration
Ajax Chat - Fully browser based chat!

•

Join Date: Dec 2004

Posts: 4,190

Reputation: peter_budo has much to be proud of

Solved Threads: 483

peter_budo peter_budo is offline

Offline

Code tags enforcer

Re: ::Making Daniweb FAQ Bot::

-1

Jul 2nd, 2009

Nobody beside mod or admin can follow that link

Learn to see in another's calamity the ills which you should avoid.
Publilius Syrus
(~100 BC)
LJC - London Java Community, Graduate & Undergraduate Software Development Community, JAVAWUG (Java Web User Group), The London Android Group

•

Join Date: Sep 2007

Posts: 1,456

Reputation: cwarn23 has a spectacular aura about

Solved Threads: 136

cwarn23

Offline

Nearly a Posting Virtuoso

Re: ::Making Daniweb FAQ Bot::

Jul 3rd, 2009

Is the topic on the moderators lounge looking hopeful or haven't there been that many replies yet?

•

Join Date: Apr 2005

Posts: 16,153

Reputation: jbennet is a name known to all

Solved Threads: 530

jbennet

Offline

Moderator

Re: ::Making Daniweb FAQ Bot::

-7

Jul 3rd, 2009

Not many replies yet, dont think dani has seen it yet. Happygeek (administrator) has and its currently being discussed.

Last edited by jbennet; Jul 3rd, 2009 at 10:58 am.

If i am helpful, please give me reputation points.

•

Join Date: Sep 2005

Posts: 1,075

Reputation: digital-ether is just really nice

Solved Threads: 66

digital-ether digital-ether is offline

Offline

Veteran Poster

Re: ::Making Daniweb FAQ Bot::

Jul 3rd, 2009

•

Originally Posted by cwarn23

Is the topic on the moderators lounge looking hopeful or haven't there been that many replies yet?

still little discussion on that yet.

Is the RSS feeds available to every forum not a better alternative then a spider? It would involve less parsing...

www.fijiwebdesign.com - web design and development and fun
Cpanel Email - Let users Register email accounts on your website upon registration
Ajax Chat - Fully browser based chat!

•

Join Date: Sep 2007

Posts: 1,456

Reputation: cwarn23 has a spectacular aura about

Solved Threads: 136

cwarn23

Offline

Nearly a Posting Virtuoso

Re: ::Making Daniweb FAQ Bot::

Jul 3rd, 2009

I just checked the rss feeds on forum 17 and the rss feeds seems to be full of questions instead of answers. I was thinking more along the lines where the user could type a question like I did in post #1 or perhaps even send a link to their post and php would be able to string together answers from the database. And the database would be populated from bot scans. For now I'll find another site to scan as a demo while a decision is being made.

•

Join Date: Apr 2005

Posts: 1,402

Reputation: ShawnCplus is a glorious beacon of light

Solved Threads: 225

ShawnCplus ShawnCplus is offline

Offline

Code Monkey

Re: ::Making Daniweb FAQ Bot::

Jul 3rd, 2009

Some tips for your code cwarn:

Don't use preg_* functions unless absolutely necessary, if you're comparing a static string use strpos or stripos.

Define reused regular expressions in one place (you use /(.*)[.]([^.\?]+)(\?(.*))?/ three times in the same script, define it once in a variable and use it that way, one point of failure is always better.

If you're going to go for speed at the cost of memory usage get rid of in_array. Build an 1-level index and use isset() so you're performing O(1) operations instead of O(n) the in_array and array_key_exists functions are expensive. Example:

 Help with Code Tags 
 php Syntax (Toggle Plain Text) 
$some_big_array = array(1 => somestring, ..., 10000 => anotherstring);
 
$strings_index = array('somestring' => 1, ...., 'anotherstring' => 1);
 
$search = 'somestring';
if (isset($strings_index[$search])) // O(1) operation
 
if (in_array($search, $some_big_array)) // O(n) operation
$some_big_array = array(1 => somestring, ..., 10000 => anotherstring);

$strings_index = array('somestring' => 1, ...., 'anotherstring' => 1);

$search = 'somestring';
if (isset($strings_index[$search])) // O(1) operation

if (in_array($search, $some_big_array)) // O(n) operation

STOP USING global
Don't use unset() right before an assignment of the same variable, ie.,

 Help with Code Tags 
 php Syntax (Toggle Plain Text) 
$datac = "somestring";unset($datac);$datac = "anotherstring";
$datac = "somestring";unset($datac);$datac = "anotherstring";

The unset becomes wasted time because reassignment is an implicit flush of previous memory

Last edited by ShawnCplus; Jul 3rd, 2009 at 10:25 pm.

GCS d- s+ a-->? C++(++++) UL+++ P+>+++ L+++ E--- W+++
N+ o K w++(---) O? !M- V PS+>++ PE+ Y+ PGP !t- 5? X- R tv+
b+>++ DI+ D G++>+++ e+ h+>++ r y+
PMs asking for help will not be answered, post on the forums. That's what they're there for.

•

Join Date: Sep 2007

Posts: 1,456

Reputation: cwarn23 has a spectacular aura about

Solved Threads: 136

cwarn23

Offline

Nearly a Posting Virtuoso

Re: ::Making Daniweb FAQ Bot::

#10

Jul 4th, 2009

Although that's good advice Shawn and I will take note of that.
The only thing I would disagree with is the unset(). I have noticed in the past when reassigning string to variable in a loop a few million times the server runs out of memory where as if unset is used I have noticed the memory problem does not occur.