All Questions
15 questions
-3
votes
1
answer
525
views
What is efficient and fastest way to get number of records( no. of lines) in zip file using aws s3 command?
I want to get number of records in zip file which is present in s3 bucket. Could you please tell me what is the fastest way to get the result?
I am running below command but that is not working. ...
- 233
4
votes
3
answers
14k
views
Copy file from the s3 to the local?
I have a lot of files in my s3 bucket, so is there any aws cli command which I can use to find a most recent file with a prefix name in s3? and how can I copy that file from s3 to my local folder? Can ...
user13542716
5
votes
3
answers
7k
views
Faster way to Copy S3 files
I am trying to copy around 50 million files and 15TB in total size from one s3 bucket to another bucket.
There are AWS CLI option to copy fast. But in my case, I want to put a filter and date range. ...
- 1,548
0
votes
2
answers
380
views
Move a file in S3 to a new folder with the same name
I have an AWS S3 bucket with hundreds of files in it. I want to create a new folder for each file, and I want the folders to have the same name as the files.
Basically, from this:
bucket_name/
├── ...
- 161
-1
votes
1
answer
28
views
Few errors when attempting to write a script that uploads any data in EC2 Instance to S3 Bucket using BOTO
I have been trying to back up my EC2 Instance to an S3 Bucket but have contentiously come across few errors when I run the file. Most notable error being S3ResponceError: 403 Forbidden
FYI, I am ...
6
votes
5
answers
33k
views
s3 - how to get fast line count of file? wc -l is too slow
Does anyone have a quick way of getting the line count of a file hosted in S3? Preferably using the CLI, s3api but I am open to python/boto as well.
Note: solution must run non-interactively, ie in an ...
- 320
0
votes
1
answer
573
views
Difference between python boto get_contents_to_filename and s3 cp?
I copy files from AWS S3 to my local file system. From an I/O perspective, what's the difference between using get_contents_to_filenamehttp://docs.pythonboto.org/en/latest/ref/s3.html from the boto ...
- 990
2
votes
2
answers
1k
views
How Do I download latest files from S3 Bucket that contains A LOT of files
There are a lot of files in my S3 bucket, I am looking to download most 1000 recent ones (uploaded by date).
How do I go on about doing that with AWS cli or s3 boto
- 41
3
votes
3
answers
10k
views
HTTP Code 400 getting objectfrom AWS S3 bucket with Boto3 in python
I'm working on a script that is supposed to download a file from my s3 bucket and then make some changes on it.
The problem is that my script runs well when executed as root but does not when ...
- 455
4
votes
1
answer
2k
views
Cannot read a key from S3 with boto, but can with aws cli
Using this aws cli command (with access keys configured), I'm able to copy a key from S3 locally:
aws s3 cp s3://<bucketname>/test.txt test.txt
Using the following code in boto, I get ...
- 2,484
17
votes
7
answers
26k
views
How to check when is the last time S3 bucket has been updated?
Does S3 bucket has information in regard to when is the last time it has been updated? How can I find the last time any of the objects in the bucket were updated?
- 15.6k
1
vote
1
answer
1k
views
get aws cli 'multipart_chunksize' value using boto
I am developing an application in python and I need to calculate the number of S3 Puts, for this I require the total size of file to be divided with multipart_chunksize to find total number of chunks.
...
- 5,029
1
vote
2
answers
561
views
Grant Amazon ML permission to read from Amazon S3 programmatically
I am developing an Amazon Machine Learning application and am reading my data from amazon s3. I ran into a read permission error and found a solution using amazon explorer. I was wondering if there ...
- 65
1
vote
1
answer
879
views
How can I download all files from Amazon S3 that are in a folder that was created last month?
How can I download a folder from Amazon S3 that was created last month until the present?
I have this code using boto:
for key in bucket.list():
if last_month < dateutil.parser(key....
- 77
56
votes
1
answer
38k
views
Difference between s3cmd, boto and AWS CLI
I am thinking about redeploying my static website to Amazon S3. I need to automate the deployment so I was looking for an API for such tasks. I'm a bit confused over the different options.
Question: ...
user1781186