All Questions
35 questions
2
votes
3
answers
2k
views
Can we reload a page/url in python using urllib or urllib2 or requests or mechanize?
I am trying to open a page/link and catch the content in it.
It gives me the required content sometimes and throws error sometimes.
I see that if I refresh the page a few times - I get the content.
...
- 45
1
vote
2
answers
1k
views
Unable to get the redirected URLs in Python. Tried using requests, urllib, urllib2, and mechanize
I have a huge list of URLs which redirect to different URLs.
I am supplying them in for loop from a list, and trying to print the redirected URLs
The first redirected URL prints fine.
But from the ...
- 45
1
vote
1
answer
147
views
Requests or Urllib - Login in a website, send download request to url, and save as xlsx
I am going crazy with the following problem. What I want to do is login into a website, download a file, while saving the download request as an xlsx.
I am pretty sure I need to use the requests ...
- 309
0
votes
1
answer
99
views
I cannot fill out field using urllib, urllib2, mechanize libraries, I want to submit a form
Here is a sample of my code with urllib and urllib2:
import urllib
import urllib2
url = 'http://example.com/schedule-appointment.php'
name = "Name:"
phone = "Phone:"
email = "E-mail:"
office = "...
- 1
0
votes
1
answer
426
views
Error trying to download file using Python
I have been trying to download a link to automate the stock market and my code runs but the zip file does not get downloaded.
import urllib
import urllib2
import requests
url ='https://www.nseindia....
- 31
3
votes
2
answers
909
views
BeautifulSoup not extracting all html
We are trying to get product urls from this page of Forever 21's site (http://www.forever21.com/Product/Category.aspx?br=f21&category=dress&pagesize=100&page=1). For some reason, ...
- 41
0
votes
1
answer
124
views
python urllib, urllib2 how to get SHARP links
okey my dear helpers, here is the question, I can not get the
' http://example.com/#sharplink ', by the way in the site making infinite loop so I used redirect handler and it need to enable the ...
- 795
0
votes
2
answers
2k
views
Selecting from dropdown list using urllib
I want to select an item from a drop down list from a .aspx page then click a button. Then a table is updated based on the value in the dropdown. When the item in the dropdown is selected, it will ...
- 119
0
votes
0
answers
53
views
Beautifulsoup & mechanize crawler won't work
I'm trying to go to a website and scrape all the links that include the host websites url. when I run the code with the print statement in the if statement uncommented, I get a list of scraped urls ...
- 431
3
votes
2
answers
236
views
Trying to download page in python with urllib2 and requests but keep getting redirected
I am trying to simply download a page with python.
http://webapps.rrc.state.tx.us/CMPL/viewPdfReportFormAction.do?method=cmplP4FormPdf&packetSummaryId=97770
If i get the response code from the ...
- 33
-1
votes
1
answer
204
views
Downloading files with multiple extensions on a webpage using mechanize library in Python
My first question on Stack Overflow!
I'm trying to download resumes of a job posting website. I've found the link that leads to the download, but those downloads have a '.php' ending, and hence I ...
- 11
0
votes
1
answer
129
views
How would I find these grades and these class names using mechanize and BeautifulSoup?
So, I've figured out how to login in to the website that allows me to check my grades, progress book, with python, mechanize, and BeautifulSoup. But I've run into a problem with getting the name of ...
- 59
2
votes
1
answer
580
views
Accessing web table using Python - NIST website
I am trying to access a table from the NIST website here:
http://physics.nist.gov/PhysRefData/XrayMassCoef/tab3.html
Assume that I click the element zinc. I would like to retrieve the information for ...
- 12.4k
2
votes
2
answers
1k
views
Downloading a pdf from link but server redirects to homepage
I am trying to download a pdf from a webpage using urllib. I used the source link that downloads the file in the browser but that same link fails to download the file in Python. Instead what downloads ...
- 27
1
vote
1
answer
78
views
Python - Fill log in form and then fill another form accessible only after logging in
I use mechanize to log in, but then after I submit the logging in details and I sign in successfully I'm not sure how to keep the session active and fill in the next form. Could anyone give me some ...
- 2,612
1
vote
2
answers
331
views
urllib.urlopen does not work for this url though mechanize works
My code below doesn't work for the URLs in nytimes which are articles. Please try changing the URL variable to something else and you'll see that it works. Why is that?
#url = "http://www.nytimes.com"...
- 11
3
votes
1
answer
6k
views
What's the fastest way to check a webpage's status?
I'm trying to check a webpage's status with Python. I've done timers to test, but nothing yields anything much better than the other. The worst and best only differ by 20%. I really just need the ...
- 24.7k
0
votes
2
answers
6k
views
How do I log in to my online bank account and print the transaction history?
I want to log in to my online bank account and print the transaction history.
I'm using an alternative to mechanize called Splinter because it's much easier to use and more clearly documented.
The ...
- 2,289
1
vote
1
answer
257
views
capture http calls and headers in python
We are testing videos in our website, and in order to play it should authenticate the user, get the authorization for the device he is playing and so on, check his entitlements etc.,
we have many ...
- 477
0
votes
1
answer
1k
views
Submit form in Python 3
I am trying to submit a form using Python 3. In Python 2 I did the following:
br=mechanize.Browser()
br.set_handler_robots(False)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; ...
- 759
2
votes
2
answers
6k
views
Python authenticate and launch private page using webbrowser, urllib and CookieJar
I want to login with cookiejar and and launch not the login page but a page that can only be seen after authenticated. I know mechanize does that but besides not working for me now, I rather do this ...
user1552586
3
votes
1
answer
610
views
What is a urllib code equivalent for a mechanize submit() method?
Im not sure the Title is clear enough so this is whats up:
I've started learning Python about 3 weeks ago, and recently I gave myself a task - to login to a website and then read specific page, and ...
- 125
5
votes
1
answer
1k
views
Python unable to retrieve form with urllib or mechanize
I'm trying to fill out and submit a form using Python, but I'm not able to retrieve the resulting page. I've tried both mechanize and urllib/urllib2 methods to post the form, but both run into ...
- 498
1
vote
0
answers
713
views
Check if a web page has change which requires login / cookie
I'm using this code that I found online to check a webpage to see if it has been modified since the last time it was checked. However I now have a need for it to work behind a site that require ...
- 167
6
votes
1
answer
3k
views
Executing JavaScript in href of links with Python
I am trying to download some PDF's automatically on a site (http://bibliotecadigitalhispanica.bne.es) using Python.
I've tried using the urllib/urllib2/mechanize modules (which I have been using for ...
- 97
0
votes
2
answers
3k
views
How do you open a URL with Python without using a browser?
I want to open a URL with Python code but I don't want to use the "webbrowser" module. I tried that already and it worked (It opened the URL in my actual default browser, which is what I DON'T want). ...
- 375
8
votes
4
answers
31k
views
What should I do if socket.setdefaulttimeout() is not working?
I'm writing a script(multi-threaded) to retrieve contents from a website, and the site's not very stable so every now and then there's hanging http request which cannot even be time-outed by socket....
- 4,983
0
votes
3
answers
3k
views
What should I do with hanging http POST request in python?
Sample code:
socket.setdefaulttimeout(150)
MechBrowser = mechanize.Browser()
Header = {'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 GTB7.1 (...
- 4,983
10
votes
3
answers
41k
views
Python: Clicking a button with urllib or urllib2
I want to click a button with python, the info for the form is automatically filled by the webpage. the HTML code for sending a request to the button is:
INPUT type="submit" value="Place a Bid">
...
- 1,156
0
votes
1
answer
6k
views
urllib2 download captcha image
I am trying download captcha image from VBB board by using "mechanize" aka urllib2.
This is where captcha locate (login with any username and pass you will be asked captcha):
<img id="imagereg" ...
- 27
2
votes
2
answers
5k
views
Downloading pdf files using mechanize and urllib
I am new to Python, and my current task is to write a web crawler that looks for PDF files in certain webpages and downloads them. Here's my current approach (just for 1 sample url):
import mechanize
...
- 1,257
1
vote
2
answers
1k
views
Python urllib2 parse html problem
I am using mechanize to parse html of website, but with this website i got strange result.
from mechanize import Browser
br = Browser()
r = br.open("http://www.heavenplaza.com")
result = r.read()
...
- 27
0
votes
2
answers
628
views
which function of mechanize is equal with urllib.urlencode
urllib.urlencode could encode url's params. It seems no likely function in Mechanize.
So, I have to use urllib and Mechanize, because I only need urlencode.
Any function could implement the same ...
- 51
0
votes
1
answer
2k
views
python urllib post question
im making some simple python post script but it not working well.
there is 2 part to have to login.
first login is using 'http://mybuddy.buddybuddy.co.kr/userinfo/UserInfo.asp' this one.
and second ...
- 325
11
votes
4
answers
6k
views
Force python mechanize/urllib2 to only use A requests?
Here is a related question but I could not figure out how to apply the answer to mechanize/urllib2: how to force python httplib library to use only A requests
Basically, given this simple code:
#!/...
- 90.2k