2,496 questions
3
votes
3
answers
95
views
How to pass JSON using WWW::Mechanize in perl
We are using WWW::Mechanize to send json data to our api.
My code is below.
#!/usr/local/bin/perl
use WWW::Mechanize;
sub get_csrf_token {
my $sso = shift;
my $referer = shift;
my @...
- 21.1k
0
votes
0
answers
15
views
Modeling a Reaction Wheel in SolidWorks (without Force/Torque Manager)
I am trying to simulate the effect of transferring the torque of a reaction wheel to a pendulum.
The design is simple: support, lever and wheel.
Mechanical connections: coincidence and concentricity.
...
- 137
0
votes
0
answers
122
views
Is automatic scraping forbidden on EURLEX?
Is it possible that the EU has removed the possibility to scrape the EURLEX Website?
For a university course I am required to scrape the following website for all the listed documents in an html ...
- 72
0
votes
1
answer
57
views
finding username field with mechanize
I am attempting to automate navigation of https://support.paloaltonetworks.com/, which requires me to first log in on https://sso.paloaltonetworks.com/
I've managed to open the site with the following ...
- 13
0
votes
0
answers
82
views
Perl Mechanize Error GETing (Bad file descriptor)
Consider this short piece of code:
use strict;
use warnings;
use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
$mech->get("https://www.flightaware.com/live/airport/KCLE");
my $...
0
votes
1
answer
59
views
Using Nokogiri to scrape element from a web page with ruby
Description
Have a web page which loading into Mechanize ruby gem. Problem is that I can see the HTML fragment in developer tools but not having any luck extracting the element and the associated ...
- 2,320
0
votes
1
answer
223
views
How resolve a 403 Forbidden message Mechanize Ruby
403 => Net::HTTPForbidden for https://www.state.gov/countries-areas-archive/tunisia/page/2/ -- unhandled response (Mechanize::ResponseCodeError)
This is what i read in console , i look to scrape ...
1
vote
0
answers
45
views
Mechanize authentication error 401 when saving file to a shared drive
I am connected to a shared SMB volume. Using Mechanize in a Ruby script, I am trying to download several files from a proprietary asset manager, and save them directly to the shared drive, i.e.:
agent ...
- 213
-1
votes
1
answer
51
views
Python doesn't find standard default path for SSL keys on Windows
to tell you the truth, I don't have much idea why this error is occurring, I've been using the same development environment for about 3 months, for study, today when I sat down to study I came across ...
1
vote
1
answer
178
views
Attempting to select a <textarea> on a website and submit it? (Using Python Mechanize)
I am trying to submit the textarea on https://keywordsheeter.com/ (name="findersearchQueryInput") then I want to be able to print the results of the submitted textarea to my console.
I tried ...
- 15
0
votes
0
answers
91
views
Python, cant use mechanize - No module named 'mechanize'
hope u can help me.
i have the problem that i cant use mechanize in Python. i really dont know why.
My python version is: Python 3.10.9
i just wrote this in a python file:
import mechanize
browser = ...
- 1
0
votes
1
answer
94
views
Web-scraping with Ruby Mechanize
I have tried scraping a web page with ruby mechanize but is not working. Basically that website have some products and i need the links of the products.
HTML
I was test this code below, and I ...
-2
votes
1
answer
112
views
Download file in python on clicking on href
I am trying to click on below href link.
href="javascript:;"
<div class="xlsdownload">
<a id="downloadOCTable"
download="data-download.csv"
...
- 11
1
vote
0
answers
178
views
python mechanize get element by id and data-row for auto-posting
I'm trying to auto-post a form using mechanize library with python.
Mechanize library uses "name" attribute. But for my project, I need to work with "id" and "data-row" ...
- 225
0
votes
1
answer
336
views
Automating Web form Filling using Selenium or Mechanize Python with Pandas Dataframe
Automating Web form Filling using Selenium or Mechanize Python with Pandas Dataframe.
I am looping through a three columns of pandas dataframe to auto-complete the form for me and submit. Below is the ...
user19671534
1
vote
3
answers
189
views
Why is mechanize not installing properly via pip on RPi? (python 3.9)
I can't get to successfully install the package 'mechanize' on a Raspberry Pi (so, ARM chip) with Debian Bullseye, python 3.9 in a virtualenv.
When I look in the virtualenv's sitepackages folder, ...
- 24.8k
0
votes
1
answer
119
views
Unable to click download button with web scraper
for whatever reason mechanize is unable to click this export button to automate downloading a public government csv file, i'm trying to automate downloading a fishing report, does anyone have any ...
- 2,562
1
vote
0
answers
36
views
How can I open text file with mechanize browser in Python - Password List
The issue I am having with my code is, only the last password in the text file is been tried by the script, but if I put the correct password in the last line it login successful
again I want if the ...
- 69
0
votes
0
answers
181
views
How can I get a contain part of an HTML element by ID or NAME - Python - browser or BeautifulSoup
from gettext import find
import mechanize
browser = mechanize.Browser()
browser.set_handle_robots(False)
cookies = mechanize.CookieJar()
browser.set_cookiejar(cookies)
browser.addheaders = [('...
- 69
0
votes
1
answer
317
views
Structure of HTML Form Data
I know this is stupid question but how does payload of this form data be formatted. I can't make request more than one time so i can't see real request send.
I have tried same on local html file and ...
- 405
1
vote
1
answer
224
views
python mechanize submit response.read() doesn't return correct page?
all! I am trying to do some data processing on a publicly accessible database (property tax records, I know, not very glamorous!). The code runs perfectly, but it doesn't return the right page. When I ...
- 11
1
vote
0
answers
619
views
How to bypass Google recaptcha in a form using Mechanize
I need to fill a form on a webpage and bypass its captcha.
using this library I'm able to solve the captcha and get g-recaptcha-response stored in a variable. now the question is, how to pass the ...
- 11
0
votes
0
answers
79
views
Get Dynamiclly loaded Source Code from python using mechanize and bs4
I want to get source code of page that loads from javascript actually that page is linkedin profile page and i want to get job and education details.
I'm not using selenium i don't want browser window ...
- 405
0
votes
1
answer
64
views
How to edit page code with python mechanize?
I have some code on page like this:
<div class="tabs1_cnt" id="pupil_tabs_content_2189609">
<div class="pupil_tabs_content_item tabs1_cb" id="...
0
votes
1
answer
305
views
Ruby/Mechanize workaround for Let’s Encrypt Root Certificate Expiration and OpenSSL 1.0.2
With the expiration of Let's Encrypt's root certificate DST Root CA X3, my ruby scripts using OpenSSL to connect to servers that use this certificate are returning the error:
/usr/local/Cellar/ruby/2....
1
vote
1
answer
432
views
Mechanize::ResponseCodeError (404 => Net::HTTPNotFound unhandled response):
Trying to scrape images from https://en.wikipedia.org/ website using mechanize gem. I am getting Mechanize::ResponseCodeError (404 => Net::HTTPNotFound for https://upload.wikimedia.org/wikipedia/...
- 1,189
2
votes
2
answers
179
views
How to scrape icon link of image using mechanize gem
I have a url where I have to scrape all images using mechanize gem, but some image url's are in rel=icon.
I have to get the image from this url:
<link rel="icon" href="https://...
- 1,189
0
votes
1
answer
108
views
Using ruby mechanize on forms not in <form> tag?
So I'm trying to web scrape a site which has what I would consider a form - several inputs to specialize search such as <select>s and text inputs - that is not within <form> tags.
Rather, ...
- 27
1
vote
0
answers
466
views
Python: fastest way to write/download online images to disk?
How I can speed up the speed at which my program writes an online image to disk? Right now ~150KB takes ~8 seconds to download, 3 seconds for opening the image online and 5 seconds to write it.
...
- 317
2
votes
1
answer
170
views
save directly to file and get filename using WWW::Mechanize
I would normally save to a file using this:
$mech->save_content($mech->response->filename);
But due to some big files which cause "Out of memory" I have to use this instead:
$mech-&...
- 23.3k
3
votes
1
answer
78
views
Why is youtube not giving me video links and only giving me their resource pages?
For some reason youtube only gives me their resource pages instead of giving me their video links
which is what I am looking for.
code:
import mechanize
import urllib
import urllib.parse as urlparse
...
- 135
0
votes
0
answers
96
views
Mechanize instance can't .open a specific website
I'm trying to use the mechanize .open method on a URL so I can eventually login and scrape this website. I've tried using this code on other websites and they work, but for some reason I'm getting the ...
1
vote
0
answers
131
views
How to fill a form input field using its ID rather than the form name with mechanize?
I am trying to automate geocoding with python. There is a simple website that does this, but I don't want my user to manually visit the website. I want to automatically take his adress and convert it ...
- 103
0
votes
0
answers
112
views
Getting 403 error when running Python mechanize within docker container
I've built a Python script with the module mechanize which scrapes data from a website. When I run the code on my python environment on my localhost, everything works fine. Now i've started to ...
0
votes
1
answer
197
views
python web scraping issues with mechanize
I am trying to scrape web results from the website: https://promedmail.org/promed-posts/
I have followed beutifulsoup. mechanical soup and mechanize so far unable to scrape the search results.
import ...
0
votes
1
answer
238
views
Can't login using mechanize from python. What am I doing wrong?
I'm trying to use mechanize in python to login to this site: https://login.haaretz.co.il/
On the surface, it looks like a 2-phase login process, same as google, but following receipts for google login ...
- 3
1
vote
1
answer
200
views
Sending custom cookies (and viewing the ones set naturally) with Perl's WWW::Mechanize
I'm trying to scrape a particularly troublesome website. Though all the parameters match and the referrer matches, I see different results when perl runs it than when I watch dev tools.
When I do a ...
- 5,393
1
vote
0
answers
289
views
Node has gone away in WWW::Mechanize::Chrome
WWW::Mechanize::Chrome loses somehow the node, when calling a different method like ->find_all_links()
No node with given id found
-32000 at /usr/local/share/perl/5.20.2/Chrome/DevToolsProtocol/...
- 1,162
1
vote
1
answer
122
views
Is it possible to get a checkbox with Mechanize other than name attribute?
I have been trying scraping with Ruby and Mechanize.
The websites I saw say that the way to get a checkbox and check it is below.
form.checkbox_with(:name => 'name').check
However, the checkbox ...
- 91
1
vote
1
answer
854
views
How to edit a facebook post in Python?
I have a post on my fb page which I need to update several times a day with data elaborated in a python script. I tried using Selenium, but it gets often stuck when saving the post hence the script ...
- 3,539
1
vote
1
answer
52
views
How to use mechanize package inside a function to log into website?
I am using mechanize to log into a website and scrape it with beautifulsoap. While I got it working without using functions, I don't know how to put the login-functionality into a function and then ...
- 938
5
votes
1
answer
2k
views
How to fill and submit a form using python
I am filling a form of a web page with the help of mechanize module but I am getting error when I run my code.
I just want to fill the form and submit it successfully.
My attempt :
code snippet from ...
- 106
-2
votes
1
answer
139
views
How to upload file using mechanize python?
I need to upload a file(txt) in a form like this:
the structure of the add button is as follows:
after upload I have to send the file with the send button:
the structure of the send button is as ...
1
vote
3
answers
341
views
Perl: Getting UTF-8 file with Mechanize content_file
Creating a perl script (Strawberry Perl v5.32.0 on W10) to download my Google Calendar. Google provides a 'private' (no login required) url to a 'basic.ics' file.
When opening this url in my browser (...
- 21
1
vote
2
answers
13k
views
Python click button on web page without selenium
So there is a button on a webpage that downloads a csv file after clicking it. In the past I have used selenium to do this, but given the current application of this script being ran on databricks I'd ...
- 123
-1
votes
1
answer
860
views
How to fill JavaScript form using Python?
I want to use Python to fill this form.
I tried using Mechanize but this is a Microsoft Form which uses JavaScript and has no form tag and no GET/POST URL. Maybe BeautifulSoup/Selenium can do this, ...
- 74
1
vote
2
answers
119
views
Which regex tag to use in a Mechanize function?
I retrieved all the links from the web page containing /title/tt inside the url in a list.
my @url_links= $mech->find_all_links( url_regex => qr/title\/tt/i );
but the list is too long so I ...
- 29
0
votes
1
answer
310
views
Python + Mechanize - Emulate Javascript button click using POST?
I'm trying to automate filling a car insurance quote form on a site:
(following the same format as the site URL lets call it: "https://secure.examplesite.com/css/car/step1#noBack")
I'm ...
user14039114
0
votes
1
answer
532
views
How to download data from Jotform using python?
I am collecting some survey data from jotform, my data include audio recording and the URL for audio in form is
'https://www.jotform.com/widget-uploads/voiceRecorder/201374133/981221_121.wav'
If I ...
- 12.6k
0
votes
1
answer
273
views
Does the Mechanize gem support TLS 1.2 and how to set it?
I want to use the Mechanize gem to do a TLS 1.2 POST request, but I have poor knowledge into SSL and TLS, so I can't seem to find any useful way to determine if I set it correctly. Mechanize.new....
- 366