All Questions
843 questions
0
votes
0
answers
40
views
Audio Recorded on Chrome and Uploaded to S3 Using Presigned URL Plays in All Browsers Except Safari
I'm developing a web application that allows users to record audio and upload it to AWS S3 using presigned POST URLs. The recorded audio is played back using an element with the S3 link as the source....
0
votes
3
answers
100
views
Duplicity SSLEOFError: EOF occurred in violation of protocol
I have duplicity running for years, stable and without problems. Now I encountered the following problem when I backup my files to Amazon S3.
All files in the format "duplicity-full....
- 431
0
votes
0
answers
23
views
Different shaped responses for generate_presigned_post
My application's deployed across 2 regions, and calling generate_presigned_post from the two regions returns different response shapes. The first returns:
{
"url": "https://app-1-...
- 8,157
1
vote
1
answer
158
views
botocore >= 1.28.0 slower in multithread application
The official Boto3 docs recommends creating a new resource per thread: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#multithreading-or-multiprocessing-with-resources
...
- 989
3
votes
2
answers
2k
views
S3:HeadObject returns 403 on MinIO instance proxied by Cloudflare
When sending HeadObject request to MinIO instance proxied by Cloudflare, 403 is returned instead of 404 (for non-existent files). GET, PUT and DELETE operations can be performed with no issue.
For ...
- 806
-1
votes
1
answer
116
views
Grant s3 user access in bucket level policy
I have minio with bucket named "bucket" and IAM user named "user1"
I'll try to grant access to this bucket with Bucket Level Policy
client = boto3.client('s3', endpoint_url='...
- 71
1
vote
0
answers
232
views
Error when using pandas .to_csv function to upload to S3. "Anonymous access is forbidden for this operation"
I'm getting and error "Anonymous access is forbidden for this operation" when trying to use the df.to_csv function.
df.to_csv("s3://testdatateam1/test6.csv",
...
- 23
1
vote
0
answers
490
views
Multithreading in AWS boto pagination Python not working
I tried multithreading with paginator but instead of running on multiple thread, it is running on a single thread.
response: <class 'botocore.paginate.PageIterator'>
page.get('Contents') return ...
- 110
0
votes
1
answer
996
views
S3 tagging and access control policies not working for limiting the tags keyset on an object
Trying to restrict tags to only a given set of keys that can be attached to the objects. Using bucket level policies to define this condition. However, the logic is not working. Bucket policy (https://...
- 1,920
0
votes
1
answer
1k
views
trying to use boto copy to s3 unless file exists
in my code below,
fn2 is the local file and "my_bucket_object.key" is a list of files in my s3 bucket.
I am looking at my local files, taking the latest one by creation date and then looking ...
- 99
1
vote
0
answers
127
views
Return filename of specific pattern which was last modified from s3 bucket
I am writing a python script where I have to fetch just the filename (Eg. 'abc.csv') from a directory in a S3 Bucket which was last modified. I have managed to fetch the filename from the bucket but ...
- 11
0
votes
1
answer
737
views
Cannot upload the files into AWS S3 bucket with ACL pubic-read using python
I am trying to upload the file into s3 bucket but I am getting access denied error when I am trying to upload the files with ACL public-read
It is working fine with ACL private
class S3:
def ...
- 287
2
votes
0
answers
351
views
S3 boto - list objects in parallel
I want to quickly (in parallel?) list directory/prefix with 100K+ objects.
The pagination works (see below) but I'm unable to get NextKeyMarker until I download the last page - it breaks the idea of ...
- 1,547
0
votes
1
answer
893
views
Python : XML file downloaded from S3 full of string escaping characters
I have a number of XML files that I have added to S3 (localstack sever). I can view these files through Cyberduck and they are valid xml files. However, when I download the objects, the XML data is ...
- 7,043
-1
votes
1
answer
4k
views
How I can upload file to folder in bucket in S3 using Python?
I have local S3 and want to upload some files into folders in mu bucket.
For example:
my_bucket
- folder1
- folder2
I have to upload file file1.json to folder1 and file1.json to folder1
I try do ...
- 41
1
vote
0
answers
98
views
Can I read a trained linear model from s3, without reconstructing a local copy?
In order to run a dask pipeline on a coiled cluster that uses a previously trained linear model in each task, I believe I need to read the model directly from S3. Reading the model within a task did ...
- 11
0
votes
1
answer
602
views
Differentiate between website endpoint from REST API endpoint for AWS S3
I have an input provided by a user, that would be used as the endpoint url for bucket operations for an S3 bucket.
Is there a way to differentiate if the url is a REST API endpoint or a website ...
- 399
0
votes
1
answer
763
views
How do you effectively move and partition files in s3 using boto3?
There are around 10k files in an s3 location which got exported from dynamodb PITR export to s3 option. These files aren't partitioned in any way and it is within a single folder which is a problem ...
- 49
0
votes
1
answer
764
views
import S3 from s3 library
I installed s3 library on my python 3.8 environment using
pip install s3
which installed s3 version 3.0.0
I then try to run
from s3 import S3
But I would get an error of
Traceback (most recent call ...
- 2,883
0
votes
0
answers
440
views
Load a .mov video in python from S3 via Lambda
I am trying to load a .mov video file into a Lambda python script for processing.
import json
import urllib.parse
import boto3
import os
print('Loading function')
s3 = boto3.client('s3')
def ...
- 65
0
votes
0
answers
108
views
Boto s3 filter to get prefix smaller than a given prefix
I've an s3 storage which stores the files as such
2021/05/06/folder1/file.ext
2022/05/06/folder2/file.ext
2022/06/06/folder3/file.ext
is there a way to get all object paths which have prefix lesser ...
- 1
3
votes
2
answers
4k
views
list S3 objects till only first level
I am trying to list s3 obejcts like this:
for key in s3_client.list_objects(Bucket='bucketname')['Contents']:
logger.debug(key['Key'])
I just want to print the folder names or file names that are ...
- 3,390
0
votes
1
answer
803
views
ssl error with copying file in s3 server?
I tried to collect static files on the S3 Server for my Django project with the command :
python manage.py collectstatic
But It failed because of SSLError :
During handling of the above exception, ...
- 565
0
votes
1
answer
562
views
Can I use s3fs to perform "free data transfer" between AWS EC2 and S3?
I am looking to deploy a Python Flask app on an AWS EC2 (Ubuntu 20.04) instance. The app fetches data from an S3 bucket (in the same region as the EC2 instance) and performs some data processing.
I ...
- 1,461
2
votes
1
answer
2k
views
Using python Download the latest file from s3 bucket inside folder not from inside folder --folder
I want to download the latest file from s3-bucket inside folder only. Actually inside folder there are multiple folders along with files. But i need to download only file of latest date and upload it ...
- 177
0
votes
1
answer
1k
views
Copy file from s3 subfolder in another subfolder in same bucket
I'd like to copy file from subfolder into another subfolder in same s3 bucket. I've read lots of questions in SO and I came finally with this code. It has an issue, when I run it it works, but it ...
- 1,069
0
votes
1
answer
229
views
Unable to upload multiple python dataframes to s3
I am trying to upload google play console reports to s3 using boto3. The code below works well when i try to print dataframes in loop, which means i am successfully getting the files i need.
from io ...
3
votes
3
answers
3k
views
How to upload the folder to digital ocean spaces?
From the link, https://www.digitalocean.com/community/questions/how-to-upload-an-object-to-digital-ocean-spaces-using-python-boto3-library. It only states to upload files to the spaces.
I want to ...
- 996
0
votes
2
answers
5k
views
catch errors if s3 file is not downloaded
I download a file from S3 like this:
s3 = boto3.client('s3')
s3.download_file('testunzipping','DataPump_10000838.zip','/tmp/DataPump_10000838.zip')
For now it always works. However, I wanted to add ...
- 3,390
4
votes
1
answer
6k
views
find last modified date of a particular file in S3
According to this answer: https://stackoverflow.com/a/9688496/13067694
>>> import boto
>>> s3 = boto.connect_s3()
>>> bucket = s3.lookup('mybucket')
>>> for key in ...
- 3,390
0
votes
2
answers
1k
views
GzipFile not supported by S3?
I am trying to iterate through some file paths so that I gzip each file individually. Each item in the testList contains strings (paths) like this: /tmp/File.
After gzipping them, I want to upload ...
- 3,390
3
votes
1
answer
2k
views
how to specify ContentType for S3 files?
I am trying to convert files to gzip files and then upload them to S3. When I check in S3, the files are there but they don't have a type specified. How can I specify the content type?
for i in ...
user13067694
1
vote
1
answer
6k
views
S3 object has no attribute Bucket
I am trying to run a code snippet like this:
s3_file_path = "testunzipping/sample.csv.gz"
s3 = boto3.client('s3')
lst = s3.list_objects(Bucket='testunzipping')['Contents']
firstbucket = s3....
- 3,390
0
votes
0
answers
494
views
Filter Boto3 Object Collection by depth
I am trying to list all folders at given depth level in an s3 bucket.
I can easily filter by Prefix using objects.filter.
Is there a way to limit to a fixed level of depth?
For now I am counting the ...
- 5,287
1
vote
1
answer
99
views
boto3 print rules from bucket encryption
Im writing a python script to retrieve information of AWS and im trying to get only the SSEAlgorith but i get TypeError: list indices must be integers or slices, not str
Is there any way to do this? I ...
- 13
1
vote
1
answer
508
views
Save image data from a iterator object to AWS S3 in python
I am calling an API that returns a iterator object containing image data. I'd like to iterate over them and upload to s3. I could either open them into .png or .jpeg before or after dumping / ...
- 2,014
0
votes
1
answer
383
views
Programatically generating web console URL to s3 object
Is there a way to generate urls to the s3 web console (https://s3.console.aws.amazon.com/s3/object/...) within python using botocore or boto3? I know boto can be used to generate presigned urls, but ...
- 31
0
votes
0
answers
198
views
Migrating from Boto2 to Boto3 while using python 2.7
I am currently using boto2 to upload some files into an s3 bucket where the objects are sorted based on keys(which are generated) with the objects I am uploading. I am now required to migrate this ...
- 1
-3
votes
1
answer
525
views
What is efficient and fastest way to get number of records( no. of lines) in zip file using aws s3 command?
I want to get number of records in zip file which is present in s3 bucket. Could you please tell me what is the fastest way to get the result?
I am running below command but that is not working. ...
- 233
1
vote
1
answer
833
views
python boto - AWS S3 access without a bucket name
I have credentials ('aws access key', 'aws secret key', and a path) for a dataset stored on AWS S3. I can access the data by using CyberDuck or FileZilla Pro.
I would like to automate the data fetch ...
- 1,223
2
votes
3
answers
2k
views
Why is the wrong region being sent to Amazon S3 bucket query?
I'm trying to stream music stored in an Amazon S3 bucket.
import boto3
SERVICE_NAME = 's3'
REGION_NAME = 'eu-west-2'
def generate_presigned_url(bucket_name, object_key, expiry=3600):
client = ...
- 90
0
votes
1
answer
3k
views
How to use Python boto3 to get count of files/object in s3 bucket older than 60 days?
I'm trying to get the count of all object which are older than 60 days? Is there any way to perform a query or any python boto3 method to get this required output?
0
votes
1
answer
442
views
How to move an object within the same S3 bucket using boto client?
I cant seem to find a function that can move a S3 object within the same bucket. I know there's other ways other than s3 client but I need to use S3 client in this case.
- 255
3
votes
2
answers
3k
views
Searching s3 for a bucket using boto3
I'm trying to create a python script that uploads a file to an s3 bucket. The catch is that I want this script to go to s3 and search through all the buckets and find a bucket that contains a certain ...
- 65
2
votes
2
answers
3k
views
How to use S3 Select with tab separated csv files
I'm using this script to query data from a CSV file that's saved on an AWS S3 Bucket. It works well with CSV files that were originally saved in Comma Separated format but I have a lot of data saved ...
- 135
0
votes
1
answer
1k
views
S3 unit tests boto client
Having issues writing a unit test for S3 client, it seems the test is trying to use a real s3 client rather than the one i have created for the test here is my example
@pytest.fixture(autouse=True)...
- 1,379
0
votes
1
answer
592
views
Boto 3 is much slower than boto 2 for reading many small objects from an S3 bucket
I've noticed that boto3 takes about 3 times as long as boto2 to read the same objects from an S3 bucket. The Python script below illustrates the problem. My environment is Ubuntu 18.04, Python 3.7.9, ...
- 4,851
3
votes
1
answer
2k
views
S3 bucket policy using Boto3
I am working on a boto script to collect logs and store it into an S3 bucket, I am getting an error "botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ...
user14226172
1
vote
1
answer
322
views
boto support for Amazon Trust Services migration
tl;dr:
Will old boto versions that don't include amazon trust services certs become obsolete in a few months when amazon migrate from DigiCert to Amazon Trust Services?
Amazon's CA for S3 is changing ...
- 103
1
vote
2
answers
2k
views
Boto3 Moto bucket not found after mocking it
I am trying to used Boto3 moto's mocks3 utility to test my code that connects to s3, the function basically list down all the folders with partition date and returns the latest one. I see no exception ...
- 1,647