4

I created a mock s3 bucket with moto. I put an object into it. I am trying to read with pandas read_csv.

S3_BUCKET_NAME = 'mock_bucket'
def s3_bucket():
    mocks3 = mock_s3()
    mocks3.start()
    os.environ["AWS_DEFAULT_REGION"] = "us-east-1"
    s3 = boto3.resource('s3')
    s3.create_bucket(Bucket=S3_BUCKET_NAME)
    return s3
s3 = s3_bucket()
bucket = s3.Bucket(S3_BUCKET_NAME)

bucket.put_object(Key='/2022/12/14/6/25/t_ext_gop/t.csv', Body='test')
a = pd.read_csv('s3://amx-mock-bigdata-stemtime/2022/12/14/6/25/t_ext_gop/t.csv')

I get the following error:

AttributeError: 'MockRawResponse' object has no attribute 'raw_headers'

How do I resolve this?

Improve this question
3
  • 1
    Do you use aio-botocore? That is a known issue with that lib, see github.com/aio-libs/aiobotocore/issues/755
    – Bert Blommers
    Commented Dec 23, 2022 at 20:00
  • Did you find a solution how to unittest using read_csv and moto?
    – datahack
    Commented Apr 20, 2023 at 8:46
  • patch the csv reader to read local file instead of mocked s3 file 🫠
    – Simon30
    Commented Aug 27 at 14:25
Related questions
2
Unit Test with Pandas Dataframe to read *.csv files
36
How to read a csv file from an s3 bucket using Pandas in Python
4
Mock CSV DictReader
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.