Okay, so what I am trying to do is pull user data from a phpBB forum site, basically just what a member's ID is, their "rank", join date, and the other information fields that are generally displayed on their account to the public.

The problem is, the site is setup so that only registered users can see the member section of the board. So regardless of what my custom URL is to find a user's profile page, the HTML source code will not display the fields since the URL does not have my "logins" so to speak. The forum sees my url as an unregistered and gives it limited access.

So, is there a way to somehow have the url run using my own logins for the board, like from my stored passwords? That way when it goes to pull the data the board will think the program is really me just surfing the site rather than some unknown source.

I know that it could potentially leave me vulnerable since I would have my own data within the program, but I just want to know if it is possible, and how so? I think is, because I think I have seen it before where you mask the url somehow but I can't remember where I saw it at.

Also, phpBB doesn't have like a dev tool setup where you can make objects with access codes, so I just gotta brute it.

Thanks much!

Recommended Answers

All 11 Replies

This is basically parsing the HTML source code of the member page, but I just need the website to think that the program is coming from my own browser (thus a registered user), rather than some unknown source which wouldn't be a registered user.


Try WebDriver/Selenium. Usually used for testing web site gui from Java code, but it should enable you to log in us an user and then through examination of web site html structure you would be able to search for web elements and retrieve relevant info from the element

Is this webdriver a package in java or a pluggin I need to find?

It is library that is available to many languages, including Java. Also it has simple browser plugin that enables you to create/record you steps in browser and then export the test in various formats(very popular JUnit)

So I tried using a tutorial I found, the first one worked, the second one went beserk lol, could you try it real fast and see if you can figure out what went wrong?

In the meantime, i shall keep looking for more examples.

Ok, I am making some headway with this, so far here is what I got, before I start my URL check and such I basically run this method

private void AuthenticateUser() {
        WebDriver driver = new HtmlUnitDriver();
        System.out.println("auth ok1");
        System.out.println("auth ok2");
        System.out.println("auth ok3");
        System.out.println("auth ok4");
        System.out.println("auth ok5");

my problem is the last part, "I think".


I can't figure out what the Login button is named, or how exactly to have this script choose that button after my credentials are entered. In fact, I am not sure if any of my details are truly lining up or not which may be another cause.
Could you possible explain it to me? I shall keep attempting random stuff, but I am sure you probably understand it better

Here is the page that I am trying to login to with this program, so you can just view it's source code and find the class details.


EDIT, I am assuming these are the values I am going after as well.

<input class="post" type="text" name="username" size="25" value="" tabindex="1" />

<input class="post" type="password" name="password" size="25" tabindex="2" />

<input type="submit" name="login" class="btnmain" value="Login" tabindex="5" />

And I know it isn't working because when the URL part goes (after it authenticates) the HTML that is returned is the page that says

The board requires you to be registered and logged in to search users.

Been wondering...is the Webdriver only login only good for webdriver? or for the whole program.

Like, i run the authentication, and I don't know if it works or not, based on the post above that I mentioned.

But what if it IS working? could it be that when I run the next part of the code which is....

try {
            // Create a URL for the desired page
            URL url1 = new URL(completeUrl);
            // Read all the text returned by the server
            BufferedReader in1 = new BufferedReader(new InputStreamReader(url1.openStream()));
            String str1;

            while ((str1 = in1.readLine()) != null) {

                // str is one line of text; readLine() strips the newline character(s)

                dataParser = dataParser + " " + str1;


        } catch (MalformedURLException e) {
        } catch (IOException e) {

will that method NOT work with the login credentials that the webdriver used? If not, how can I have webdriver parse the page for me? Or have the StreamReader use the credentials of webdriver?


Alrighty, new update. IT WORKS! I got it going and working just fine, however there is one annoying issue.
When I run the webdriver through Firefox everything works great since Javascript is enabled. :D
Now, when I use HtmlUnitDriver, I have to activate javascript, which I found out how to do, however, when the program run's it display like...a billion "warning messages" in the console to me, and it makes the program run like 4-5 seconds slower than if I use the FirefoxDriver.

In the end though, it all works and everything runs fine, but is there some way to not have the program show the billion error messages to me?

The error is as follows:

Oct 27, 2011 3:43:21 PM com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter runtimeError
SEVERE: runtimeError: message=[The data necessary to complete this operation is not yet available.] sourceName=[http://www.igcgamers.com/forum/mchat/jquery-1.5.0.min.js] line=[16] lineSource=[null] lineOffset=[0]


WARNING: Automation server can't create object for 'ShockwaveFlash.ShockwaveFlash.7'.
Oct 27, 2011 3:43:15 PM com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter runtimeError

And there are a few more similar messages that all revolve around the same concept as the one stated above.
So, everything works, but all the messages just slow the program down a tid bit.
How can I fix it?


Sounds like one of this issues we run into when we been testing GWT website. There can be delays as you have to wait for server or there is heavy rendering context. There was something in WebDriver which said wait for element or similar...

Yea, I am just not sure how to actually have the site wait and check...you would think they could write some method that make sure if the page was "truly" loaded or not.

Yea, I am just not sure how to actually have the site wait and check...you would think they could write some method that make sure if the page was "truly" loaded or not.

Your question is not really clear here. The method which I previously mentioned does wait for and certain amount of time looking for element that is specified in "By" which can be xpath, element ID, class name and more. Therefore you have to know what page will be displaying and based on that provide most appropriate "By" to look for. This was also a reason why it was pain in the back side to do any UI testing on GWT (too much time spent on waiting for various things to happen)

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, learning, and sharing knowledge.