3

My web application requires downloading content from the user URL specified. Currently this request go through my server, which is inefficient and could get my server IP blocked.

Is there a way to let the user download the URL content directly? The same-origin policy seems to prevent using AJAX or an iframe to download and reuse this content.

Any ideas? For example is there a way via flash to download and reuse URL content?

Improve this question
2
  • Why do you need to be so shady?
    – Gareth
    Commented Oct 24, 2009 at 11:31
  • when they do that, send them an email asking your IP to be unblocked.
    – Martin v. Löwis
    Commented Oct 24, 2009 at 13:54

2 Answers 2

Reset to default
1

You could use Tor to mask your requests, but if you're having to go such lengths to crawl a website perhaps you shouldn't be doing it?

Also, with your approach the iframe request will include your page URL as the referrer, which makes identifying these requests at the server end pretty straightforward...

Improve this answer
1
  • that's interesting - I didn't think about referrer
    – hoju
    Commented Oct 24, 2009 at 13:25
0

If it's a specific web side, I recommend to talk to the website operators rather than trying to crawl anonymously.

Improve this answer
1
  • Ok: why do you need the search to be anonymous if they won't bother with you?
    – Martin v. Löwis
    Commented Oct 24, 2009 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.