All Questions
4 questions
1
vote
3
answers
2k
views
Boto error in Scrapy: "The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256."
I'm trying to crawl the following spider:
import scrapy
from tutorial.items import QuoteItem
class QuotesSpider(scrapy.Spider):
name = "quotes"
custom_settings = {
'...
- 56.9k
0
votes
1
answer
381
views
Retry Unhandled error in Deferred: in scrapy
I got BotoServerError in Scrapy when uploading images to AWS S3 because it happens to be a server down. How can I handle those errors so I can set the item pipeline to sleep for a while and retry ...
- 19.1k
1
vote
0
answers
562
views
Scrapyd S3 feed export "Connection Reset by Peer"
I'm running Scrapyd with a FEED_URI set to export to S3, but I received the following error at the very end of my scrape. Note that it successfully uploaded a few hundred kb of data to the bucket as ...
- 1,839
3
votes
2
answers
428
views
Scrapy crawl appends locally, replaces on S3?
I implemented a Scrapy project that is now working fine locally. Using the crawl command, each spider appended it's jsonlines to the same file if the file existed. When I changed the feed exporter to ...
- 975