Submitting data to a form

[gsingh2011]

I need to log in to a website using python. The two fields are username and password. This is what I have tried:

import urllib
import urllib2
import httplib

username = "username"
password = "password"

url = "https://webconnect.bloomfield.org/Zangle/StudentConnect/logincheck.aspx"

password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, "https://webconnect.bloomfield.org/Zangle/StudentConnect/", username, password)
handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(handler)
req = urllib2.Request(url)
f = opener.open(req)
print f.read()

This only returns the html code of the page with the login form, meaning the login wasn't successful. Of course when I tried, I used a valid username and password. While you can't test it out on this site, is there anything you see that is obviously the problem. I'm not very good with programming with websites, I figured out that much from tutorials...

[woooee]

Mechanize is an easier way IMHO to do this. This link explains how to log in with a user name and password to a remote site
http://stockrt.github.com/p/emulating-a-browser-in-python-with-mechanize/

[gsingh2011]

Thanks! That is very useful. I've spent so long trying to get other random code to work, but that was by far the easiest solution. I have one more question though. Now that I'm logged in, I need to navigate to a different page that I can only access once I'm logged in. How do I go to that page? br.open('http://websitename.com/page.aspx')? When I tried that it said my session had expired and that I had to log back in. How can I get to that page without it kicking me out? Typing that page name in the browser after I'm logged in works fine, btw. Here's the code I'm using:

import mechanize
import cookielib

# Browser
br = mechanize.Browser()

# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

# Follows refresh 0 but not hangs on refresh > 0
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

# User-Agent (this is cheating, ok?)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

# Open some site, let's pick a random one, the first that pops in mind:
r = br.open('https://webconnect.bloomfield.org/Zangle/StudentConnect/')
#r = br.open('http://google.com')

# Show the available forms
#for f in br.forms():
#    print f

br.select_form(nr=0)
br.form['stuident']='username'
br.form['stupassword']='password'
br.submit()
br.open('https://webconnect.bloomfield.org/Zangle/StudentConnect/stuassignments.aspx') #This is where I try to open that page... It doesn't work.
print br.response().read()

[gsingh2011]

Here's an update. I tried the code on facebook, and it logged in successfuly, but it wasn't able to go to another page when I used br.open(). When I switched it to br.follow_link() it worked fine. The problem is I'm not always going to have a link on the page I want to go to, like in my original example in the code posted above. I need to be able to directly go to that url...

[Tech B]

I think the problem in the original code is you didn't handle any cookies, or change your useragent. Take a look at this.

Facebook really doesn't like datamining on there servers.
I'm not saying that's what your doing; just giving a heads up.

[gsingh2011]

It does seem like a cookie problem, but according to http://wwwsearch.sourceforge.net/mechanize/ cookies are automatically handled. And the code you posted doesn't work for me, I tried it a while ago. The mechanize code at least logs me in, its just going to another page that's the problem.

[gsingh2011]

Anyone?

[Tech B]

You must be doing something wrong then. I can log into all of the sites I'm a member at via python; including here at daniweb.

Post the source of the page your trying to log in to.

I tried to visit https://webconnect.bloomfield.org/Zangle/StudentConnect/logincheck.aspx
and
https://webconnect.bloomfield.org/Zangle/StudentConnect/
they both aren't coming up.

[gsingh2011]

Could you post the exact code you used to log in to those websites? And after you log in, could you switch to a different page on that website and print the source? I hate asking for code, but this just isn't working for me...

[Tech B]

I don't use mechinize, this is done with urllib, urllib2, cookielib.
This is a very basic example and should get you started. I used this exact code
to log into www.hellboundhackers.org and complete the programming challenges, so I
know for a fact this code works.
You'll have to change to url's to the site you want to log into, and find the login_data
in the source of the page your trying to log into. More detailed example in the code.

If your still having trouble, post the source of the page so I can look at it.

import urllib, urllib2, cookielib

cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
opener.addheaders.append(('User-agent', 'Mozilla/4.0'))
opener.addheaders.append( ('Referer', 'http://www.urltosite.com/index.html') )


#You will need to look at the source of the page your
#trying to log into and find the name="" for the username and password
#example
#<input type='text' name='user_name' class='textbox' style='width:100px' />
#<input type='password' name='user_pass' class='textbox' style='width:100px' />
#The login button looks like this:
#<input type='submit' name='login' value='Login' class='button' />
login_data = urllib.urlencode({'user_name' : 'Name',
'user_pass' : 'Pass',
'login' : 'Login'
})

#This is where you are logged in
resp = opener.open('http://www.urltosite.com/index.html', login_data)
resp.close()

#Now you can go to member only pages
resp = opener.open('http://www.urltosite.com/Member_only_index.html')
d = resp.read() #The source to the member only page

#Create a html page to save the source to
#so we can see if it worked
f = open("test.html","w")
f.write(d)

#close file and page handles
f.close()
resp.close()

[gsingh2011]

I tried this with facebook and demonoid, and it when I open up test.html it tells me to login Here's the facebook login source:

I'm hoping you have a facebook account so you can try it yourself... Thanks.

[Tech B]

I do have a face book account and have been trying to log in with python with no success yet. I did discover that the url we need to be trying to log into is http://www.facebook.com/login.php

[Beat_Slayer]

You need a diferent aproach because the login page it's on php with a javascript login.

[Tech B]

It being a php page has nothing to do with it, the javascript is whats holding it back.

I got the login name working by using url arguments i.e. facebook.com/login.php?name=my_login_name&pass=my_pass

The pass didn't work though.

[SoulMazer]

It's probably not working because you are not supplying all the required arguments. Sadly, it looks like Facebook wants quite a few arguments to log in such as authentication token and api key, but the Facebook devs have provided some information on all of it: http://wiki.developers.facebook.com/index.php/Authorization_and_Authentication_for_Desktop_Applications .

Good luck.

[Tech B]

Why do they have to make everything so difficult.....

[SoulMazer]

Why do they have to make everything so difficult.....

Well, I have some marginally good news. Some users with the same problem as you created a library to hopefully make the connection process easy. Pyfacebook's home: link . This is the current home: do not use the Google code page, as it is supposedly outdated. You can find examples in the "examples" directory on github (link) or from the facebook developer's area (link) .

Hopefully that will make things easier.

[gsingh2011]

I will take a look at Pyfacebook and see if I can get it too work. Also, while facebook was my original intention for my program, I want to extend it to other sites. The other site I was trying was in my first few posts. It's a school site, so you wont be able to test anything with it, but just like facebook I wasn't able to log in. What are some reasons this could be happening? Tech B, you mentioned javascript earlier....

[gsingh2011]

I wasn't able to figure it out yet. I'd rather figure out how to do a simple connection without external libraries than waste time learning how to use this library anyway... thanks for the suggestion though.

Tech B, did you read my last question?

[Tech B]

Sorry, my internet has been having some issues.

You will need to figure out what; and how many arguments to send in the url to submit the data. Go over the source of the log in page and see what else can be passed.