Member Avatar for james.lu.75491856 
def GenContent(address,what):
    return urllib.urlopen("http://www.yelp.com/search?find_desc="+what+"&#find_loc="+address).read()

I am using python to do this.
It works prefectly in my browser.

  • 3 Contributors
  • 5 Replies
  • 328 Views
  • 2 Weeks Discussion Span
  • Latest Post Latest Post by chriswelborn

Dani AI

Generated

The 400 you saw almost always means the request URL or headers were malformed, not that the page “works in a browser” and not for your script. The first post shows the string &#find_loc= in the built URL — that is a red flag. A literal # starts a fragment (never sent to the server) and HTML-escaped entities (&) can end up inside the URL string if you build it from HTML text. Also unencoded spaces or non‑ASCII characters in what or address will break the request.

Practical, reliable fixes:

  • Stop concatenating raw strings. Build the query properly so parameters are percent-encoded and separated with & (not #).
  • Use a modern HTTP client (requests) and pass parameters in a dict so encoding is automatic. For example:
import requests

r = requests.get("https://www.yelp.com/search",
                 params={'find_desc': what, 'find_loc': address},
                 headers={'User-Agent': 'Mozilla/5.0'},
                 timeout=10)
html = r.text
  • Print or inspect the final URL (r.url with requests) and try that exact URL in your browser to reproduce the error. That shows whether the server actually sees the malformed string.
  • Set a browser-like User-Agent and follow redirects; some servers reject requests that look like bots. If you still get 400, check for embedded HTML entities in your inputs or extremely long queries.

Notes and cautions:

  • The GUI automation approach posted by will work but is fragile, OS/browser dependent, and slow. is right that this is easily solved in Python without simulating clicks.
  • For sustained, legal access to Yelp data use their official API (Yelp Fusion) and respect robots.txt and terms of service to avoid being blocked.

Quick checklist: remove any # fragments, percent‑encode parameters, verify the exact URL your code sends, and send a normal browser User-Agent.

Recommended Answers

All 5 Replies

Member Avatar for james.lu.75491856

I posted this in two fourms, python and webdev

Member Avatar for Gribouillis

What is your question ?

Member Avatar for james.lu.75491856

Is there a way to make it not be like this?
It works perfectly in my browser.

Member Avatar for james.lu.75491856

I did it the hard way, I used this:

import win32api
import win32con
import win32com
import ctypes
import webbrowser
import time
import os
import shutil
SendInput = ctypes.windll.user32.SendInput

# C struct redefinitions 
PUL = ctypes.POINTER(ctypes.c_ulong)
class KeyBdInput(ctypes.Structure):
    _fields_ = [("wVk", ctypes.c_ushort),
                ("wScan", ctypes.c_ushort),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", PUL)]

class HardwareInput(ctypes.Structure):
    _fields_ = [("uMsg", ctypes.c_ulong),
                ("wParamL", ctypes.c_short),
                ("wParamH", ctypes.c_ushort)]

class MouseInput(ctypes.Structure):
    _fields_ = [("dx", ctypes.c_long),
                ("dy", ctypes.c_long),
                ("mouseData", ctypes.c_ulong),
                ("dwFlags", ctypes.c_ulong),
                ("time",ctypes.c_ulong),
                ("dwExtraInfo", PUL)]

class Input_I(ctypes.Union):
    _fields_ = [("ki", KeyBdInput),
                 ("mi", MouseInput),
                 ("hi", HardwareInput)]

class Input(ctypes.Structure):
    _fields_ = [("type", ctypes.c_ulong),
                ("ii", Input_I)]

# Actuals Functions

def PressKey(hexKeyCode):

    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput( hexKeyCode, 0x48, 0, 0, ctypes.pointer(extra) )
    x = Input( ctypes.c_ulong(1), ii_ )
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))

def ReleaseKey(hexKeyCode):

    extra = ctypes.c_ulong(0)
    ii_ = Input_I()
    ii_.ki = KeyBdInput( hexKeyCode, 0x48, 0x0002, 0, ctypes.pointer(extra) )
    x = Input( ctypes.c_ulong(1), ii_ )
    ctypes.windll.user32.SendInput(1, ctypes.pointer(x), ctypes.sizeof(x))
def PressEnter():
    PressKey(0xD)
    ReleaseKey(0x0D)
def GetContent(l,googlechrome=True):
    webbrowser.open(l)
    time.sleep(1.5)
    def click(x,y):
        win32api.SetCursorPos((x,y))
        win32api.mouse_event(win32con.MOUSEEVENTF_LEFTDOWN,x,y,0,0)
        win32api.mouse_event(win32con.MOUSEEVENTF_LEFTUP,x,y,0,0)
    def rightclick(x,y):
        win32api.SetCursorPos((x,y))
        win32api.mouse_event(win32con.MOUSEEVENTF_RIGHTDOWN,x,y,0,0)
        win32api.mouse_event(win32con.MOUSEEVENTF_RIGHTUP,x,y,0,0)
    rightclick(100,100)
    time.sleep(.5)
    click(220, 191)
    time.sleep(2.5)
    PressEnter()
    time.sleep(3.5)
    for i in os.listdir("C:/Users/James/Desktop/"):
        if i.endswith(".htm") or i.endswith(".html"):
            url = i
            break
    r = open(url).read()
    os.remove(url)
    if googlechrome:
        try:
            shutil.rmtree(url+"_files")
        except:
                pass
    return r

you need pywin32

  1. It uses webbrowser to open a webbrowser
  2. wait
  3. teleports to a position
  4. wait
  5. right clicks
  6. wait
  7. clicks "Save as"
  8. wait
  9. presses enter
  10. wait
  11. reads file
  12. deletes html file
  13. deleted folder with all the files if using googlechrome
Edited by james.lu.75491856
Member Avatar for chriswelborn

What was the original problem? Your method reminds me of Visual Basic's "SendKey()". If downloading and writing an html page to disk was the problem, it can be easily solved in python.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.