Member Avatar for gsingh2011

I need to log in to a website using python. The two fields are username and password. This is what I have tried:

import urllib
import urllib2
import httplib

username = "username"
password = "password"

url = "https://webconnect.bloomfield.org/Zangle/StudentConnect/logincheck.aspx"

password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, "https://webconnect.bloomfield.org/Zangle/StudentConnect/", username, password)
handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(handler)
req = urllib2.Request(url)
f = opener.open(req)
print f.read()

This only returns the html code of the page with the login form, meaning the login wasn't successful. Of course when I tried, I used a valid username and password. While you can't test it out on this site, is there anything you see that is obviously the problem. I'm not very good with programming with websites, I figured out that much from tutorials...

  • 5 Contributors
  • 21 Replies
  • 243 Views
  • 1 Month Discussion Span
  • Latest Post Latest Post by Beat_Slayer

Recommended Answers

All 21 Replies

Member Avatar for gsingh2011

Thanks! That is very useful. I've spent so long trying to get other random code to work, but that was by far the easiest solution. I have one more question though. Now that I'm logged in, I need to navigate to a different page that I can only access once I'm logged in. How do I go to that page? br.open('http://websitename.com/page.aspx')? When I tried that it said my session had expired and that I had to log back in. How can I get to that page without it kicking me out? Typing that page name in the browser after I'm logged in works fine, btw. Here's the code I'm using:

import mechanize
import cookielib

# Browser
br = mechanize.Browser()

# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

# Follows refresh 0 but not hangs on refresh > 0
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

# User-Agent (this is cheating, ok?)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

# Open some site, let's pick a random one, the first that pops in mind:
r = br.open('https://webconnect.bloomfield.org/Zangle/StudentConnect/')
#r = br.open('http://google.com')

# Show the available forms
#for f in br.forms():
#    print f

br.select_form(nr=0)
br.form['stuident']='username'
br.form['stupassword']='password'
br.submit()
br.open('https://webconnect.bloomfield.org/Zangle/StudentConnect/stuassignments.aspx') #This is where I try to open that page... It doesn't work.
print br.response().read()
Member Avatar for gsingh2011

Here's an update. I tried the code on facebook, and it logged in successfuly, but it wasn't able to go to another page when I used br.open(). When I switched it to br.follow_link() it worked fine. The problem is I'm not always going to have a link on the page I want to go to, like in my original example in the code posted above. I need to be able to directly go to that url...

Member Avatar for Tech B

I think the problem in the original code is you didn't handle any cookies, or change your useragent. Take a look at this.

Facebook really doesn't like datamining on there servers.
I'm not saying that's what your doing; just giving a heads up.

commented: good point +10
Member Avatar for gsingh2011

It does seem like a cookie problem, but according to http://wwwsearch.sourceforge.net/mechanize/ cookies are automatically handled. And the code you posted doesn't work for me, I tried it a while ago. The mechanize code at least logs me in, its just going to another page that's the problem.

Member Avatar for gsingh2011

Anyone?

Member Avatar for gsingh2011

Could you post the exact code you used to log in to those websites? And after you log in, could you switch to a different page on that website and print the source? I hate asking for code, but this just isn't working for me...

Edited by gsingh2011 because: n/a
Member Avatar for Tech B

I don't use mechinize, this is done with urllib, urllib2, cookielib.
This is a very basic example and should get you started. I used this exact code
to log into www.hellboundhackers.org and complete the programming challenges, so I
know for a fact this code works.
You'll have to change to url's to the site you want to log into, and find the login_data
in the source of the page your trying to log into. More detailed example in the code.

If your still having trouble, post the source of the page so I can look at it.

import urllib, urllib2, cookielib

cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
opener.addheaders.append(('User-agent', 'Mozilla/4.0'))
opener.addheaders.append( ('Referer', 'http://www.urltosite.com/index.html') )


#You will need to look at the source of the page your
#trying to log into and find the name="" for the username and password
#example
#<input type='text' name='user_name' class='textbox' style='width:100px' /><br />
#<input type='password' name='user_pass' class='textbox' style='width:100px' /><br />
#The login button looks like this:
#<input type='submit' name='login' value='Login' class='button' /><br />
login_data = urllib.urlencode({'user_name' : 'Name',
'user_pass' : 'Pass',
'login' : 'Login'
})

#This is where you are logged in
resp = opener.open('http://www.urltosite.com/index.html', login_data)
resp.close()

#Now you can go to member only pages
resp = opener.open('http://www.urltosite.com/Member_only_index.html')
d = resp.read() #The source to the member only page

#Create a html page to save the source to
#so we can see if it worked
f = open("test.html","w")
f.write(d)

#close file and page handles
f.close()
resp.close()
Member Avatar for gsingh2011

I tried this with facebook and demonoid, and it when I open up test.html it tells me to login Here's the facebook login source:
<input type="text" class="inputtext" name="email" id="email" value="" tabindex="1" /></td>
<td><input type="password" class="inputtext" name="pass" id="pass" tabindex="2" /></td>
<td><label class="uiButton uiButtonConfirm uiButtonMedium"><input value="Login" tabindex="4" type="submit" />

I'm hoping you have a facebook account so you can try it yourself... Thanks.

Member Avatar for Tech B

I do have a face book account and have been trying to log in with python with no success yet. I did discover that the url we need to be trying to log into is http://www.facebook.com/login.php

Member Avatar for Beat_Slayer

You need a diferent aproach because the login page it's on php with a javascript login.

Member Avatar for Tech B

Why do they have to make everything so difficult.....

Member Avatar for SoulMazer

Why do they have to make everything so difficult.....

Well, I have some marginally good news. Some users with the same problem as you created a library to hopefully make the connection process easy. Pyfacebook's home: link. This is the current home: do not use the Google code page, as it is supposedly outdated. You can find examples in the "examples" directory on github (link) or from the facebook developer's area (link).

Hopefully that will make things easier.

Member Avatar for gsingh2011

I will take a look at Pyfacebook and see if I can get it too work. Also, while facebook was my original intention for my program, I want to extend it to other sites. The other site I was trying was in my first few posts. It's a school site, so you wont be able to test anything with it, but just like facebook I wasn't able to log in. What are some reasons this could be happening? Tech B, you mentioned javascript earlier....

Member Avatar for gsingh2011

I wasn't able to figure it out yet. I'd rather figure out how to do a simple connection without external libraries than waste time learning how to use this library anyway... thanks for the suggestion though.

Tech B, did you read my last question?

Member Avatar for Tech B

Sorry, my internet has been having some issues.

You will need to figure out what; and how many arguments to send in the url to submit the data. Go over the source of the log in page and see what else can be passed.

Member Avatar for gsingh2011

I was just about to post here. I just found some code online that I modified a little and it works now, but only for facebook mobile. That will do for now, but it still bugs me not knowing why I can't login to the normal facebook. And I still have to figure out how to log in to those other sites, but at least this worked. Here's the code if you want to see it. Thanks for everything.

import urllib2                                                   

email = "email here"                                                            
password = "password here"     

opener = urllib2.build_opener (urllib2.HTTPCookieProcessor ())                 
urllib2.install_opener (opener)                                                
loginRequest = urllib2.Request ('https://login.facebook.com/login.php?m&amp;nex\t=http%3A%2F%2Fm.facebook.com%2Fhome.php','email=%s&pass=%s&login=Login' % (email, password))                        

urllib2.urlopen (loginRequest)                                                 
connection = urllib2.urlopen ('http://m.facebook.com/')                        

f = open("test.html","w")
f.write(connection.read())
f.close()

BTW, does anyone know how this person figured out that url? https://login.facebook.com/login.php?m&amp;nex\t=http%3A%2F%2Fm.facebook.com%2Fhome.php? Knowing how they got it will help me find the correct url for other sites.

Edited by gsingh2011 because: n/a
Member Avatar for Beat_Slayer

He is usinf the mobile facebook address because it asks less information to login, like authentication token and api key.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.