Ad:

PHP Discussion Thread
Unsolved
Views: 2117

Jul 22nd, 2006

data-grabbing & mining - need script-help

Expand Post »

this is probably one of the best places to ask such questions. so i do it now.

first of - i have to explain something; I have to grab some data out of a phpBB in order to do some field reseach. I need the data out of a forum that is runned by a user community. I need the data to analyze the discussions.

to give an example - let us take this forum here. How can i grab all the data out of this forum - and get it local and then after wards put it in a local database - of a phpBB-forum - is this possible"?!"?

Nothing harmeful - nothing bad - nothing serious and dangerous. But the issue is.
i have to get the data - so what?

I need the data in a allmost full and complete formate. So i need all the data like

username .-
forum
thread
topic
text of the posting and so on and so on.

how to do that?

i need some kind of a grabbing tool - can i do it with that kind of tool. How do i sove the storing-issue into the local mysql-database.

Well you see that is a tricky work - and i am pretty sure taht i am getting help here. So for any and all help i am very very thankful

#many many thanks in advance

metabo_man http://images.devshed.com/fds/smilies/smile.gif
- a Ethno-reseracher

Last edited by metabo_man; Jul 22nd, 2006 at 6:18 pm.

Similar Threads

transfer data from an asp .net page into a javasscript script in ASP.NET
data grabbing from html sites in Python

metabo_man

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

6 posts
since Jul 2006

Permalink

Jul 23rd, 2006

Re: data-grabbing & mining - need script-help

Ok, first you get a pencil and a piece of paper... Oh you want it done automatically.

If it's your forum, you can simply download the database and do your data mining in that. If it's not your forum, you may find a friendly admin that will help you.

Puckdropper

Reputation Points: 23

Solved Threads: 23

Posting Pro in Training

Offline

494 posts
since Jul 2004

Permalink

Jul 23rd, 2006

Re: data-grabbing & mining - need script-help

hi

thanks for the reply.

Quote originally posted by Puckdropper ...

Ok, first you get a pencil and a piece of paper... Oh you want it done automatically.

yes - but agreed

web automation can be a difficult task;at least it depends on countermeasures implemented. Some will ban accesses based on the user-agent you send, some will use HTTP_REFERER values to restrict 'bots, some will use session-based authorization schemes to keep 'bots from interacting with the site.

but for the automation task we can look at WWW::Mechanize as it encapsulates many of the low-level web automation tools provided by perl.

Quote ...

If it's your forum, you can simply download the database and do your data mining in that. If it's not your forum, you may find a friendly admin that will help you.

yes right - but i need to do it on the fly. I have no time to ask. And to do for some research i want to have some kind of data that is gained without long and controversial debates....

metabo

metabo_man

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

6 posts
since Jul 2006

Permalink

Jul 23rd, 2006

Re: data-grabbing & mining - need script-help

Quote originally posted by metabo_man ...

to give an example - let us take this forum here. How can i grab all the data out of this forum

Hmm, I'm not sure about how you would grab all the information from this forum, but I can see a way how to do it for individual threads.

All I would do is click the view source code button on your internet browser, then copy that info to a text file.

Then you can just write your own parser (in any language for that matter not just php) to extract the relevant info.

Quote ...

username .-
forum
thread
topic
text of the posting and so on and so on.

Last edited by iamthwee; Jul 23rd, 2006 at 6:26 am.

iamthwee

Featured Poster

Reputation Points: 1536

Solved Threads: 431

Posting Expert

Offline

5,865 posts
since Aug 2005

Permalink

Jul 23rd, 2006

Re: data-grabbing & mining - need script-help

hello

many thanks that is of big help. I do now want to grab all the informations but only that of certain individual threads.

Quote originally posted by iamthwee ...

Hmm, I'm not sure about how you would grab all the information from this forum, but I can see a way how to do it for individual threads.

All I would do is click the view source code button on your internet browser, then copy that info to a text file.

Then you can just write your own parser (in any language for that matter not just php) to extract the relevant info.

that sounds interesting - i will look how to solve it. I will dive into the thing and return here - in case of having more questions or ideas or just think that i need more help.

best regards
metabo

metabo_man

Reputation Points: 10

Solved Threads: 0

Newbie Poster

Offline

6 posts
since Jul 2006

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.

Previous Thread in PHP Forum Timeline: Time seems to be off...

Next Thread in PHP Forum Timeline: new PHP user - help

DaniWeb Message

Cancel Changes

User Name
Password