386 questions
0
votes
1
answer
52
views
Scraping audit logs from storage accounts
I tried reading azure diagnostic logs from storage accounts via Grafana Loki and Promtail, and it works with old logs that are already archived. As soon as I try to instantaneously scrape diagnostic ...
- 43
0
votes
1
answer
15
views
Error 422: Unprocessable Entity errors with Openstack python-swiftclient
I am writing a Python script using python-swiftclient and zipfile to zip and upload files to the Swift API endpoint of an Openstack Object Store. I store the zipped data in memory as an io.BytesIO ...
- 420
2
votes
0
answers
48
views
Generating Signed URL for Replit Object Storage Bucket using GCS client libraries
I'm using replit object storage, which is a wrapper of GCS(google cloud storage). Replit does provide a library to upload and download objects, but that library isn't very flexible. Replit handles ...
- 21
0
votes
1
answer
71
views
curl: How to handle errors when uploading to object storage
I'm uploading a file with curl in bash to object storage like this:
curl "https://mybucket.fsn1.your-objectstorage.com//myfile.bin" -T "myfile.bin" --user "${S3KEY}" --...
0
votes
0
answers
29
views
Data migration from Google BigQuery to Yandex Cloud | FileNotFoundError: [WinError 2] | gsutil_path | gsutil
I'm working on the data migrattion from Google BigQuery to Yandex Cloud.
I found the next page with a guide:
https://yandex.cloud/ru/docs/managed-clickhouse/tutorials/bigquery-to-clickhouse#migrate-...
- 17
0
votes
0
answers
96
views
ibm_botocore.exceptions.CredentialRetrievalError
I am receiving the following error with the ibm botocore s3 client.
ibm_botocore.exceptions.CredentialRetrievalError: Error when retrieving credentials from https://iam.cloud.ibm.com/identity/token: ...
- 1
0
votes
0
answers
53
views
How do I connect to IBM Cloud Object Storage API from IBM Code Engine Function?
Could someone please help me figure out how to connect to IBM Cloud Object Storage API from IBM Code Engine Functions?
I'm just starting with GET requests for now, but I eventually want to be able to ...
-1
votes
1
answer
149
views
SeaweedFS S3 Gateway Stuck Connecting to Incorrect gRPC Port [closed]
I've been setting up SeaweedFS on a cluster of three nodes and encountered issues when configuring the S3 gateway. The S3 gateway tries to connect to the incorrect gRPC port 28888 instead of the ...
- 35.2k
3
votes
0
answers
488
views
Is anyone using Nats Object Store in production?
We are using Nats in production since a while and recently there has been a case where the messageSize has exceeded 8MiB.
I tried extending the default message size in server-config to 10MiB, but it ...
1
vote
2
answers
307
views
Contabo Object Storage (S3) with Strapi CMS
I'm using Contabo Object Storage (compatible with S3) with Strapi CMS.
For this purpose I'm using: @strapi/provider-upload-aws-s3
I've configured it like this (upload and delete works):
upload: {
...
- 75
2
votes
0
answers
377
views
How to mount an S3 compatible object storage to use it with jupyter notebook
I have 2 TB of data on a S3 compatible object storage. I used to keep that data in aws s3 and mount the bucket to my ec2 instance with s3fs. Now I need to use the S3 compatible storage provided my ...
- 120
1
vote
0
answers
74
views
Static Website Hosting on Object Storage in Open Telekom Cloud
I have my frontend application hosted on Object Storage in Open Telekom Cloud as a Static website. By default, after setting up the static website I got endpoint in the format https://bucket-name.obs-...
- 11
0
votes
0
answers
167
views
Create external table using Parquet files (partitioned) in Oracle ADW
I am trying to create an external table using partitioned parquet files stored in Oracle Object Storage.
These parquet files are written using Pyspark.
The folder structure is below:
Getting error ...
-1
votes
1
answer
91
views
Object storage bucket umount takes too long
I mounted a 1TB object storage bucket (not aws s3) on my M2 Mac using s3fs. After the mount, I tried to cd into the mounted directory, and it was hanging for too long. Then I tried to umount the ...
- 120
0
votes
0
answers
164
views
AWSCLI command to get object - object id/x-amz-id-2 AND List/Search object having given object id / x-amz-id-2
AWSCLI command to get object x-amz-id-2 AND List object having given object id / x-amz-id-2
Please advice which AWSCLI we can use to
Get object id / x-amz-id-2 of single/multiple objects under a ...
- 99
0
votes
0
answers
48
views
How is data stored in object storage?
what really object storage is from the storage perspective?
I understand that any unstructured or static data is stored like a raw object in the object storage, But my doubt is how is it stored on ...
- 11
0
votes
0
answers
204
views
Copy S3 bucket to E2E Object Storage using MinIO Client
I want to Copy S3 bucket Object to E2E Object Storage using MinIO Client.
I can't figure out how to configure and copy, some things i tried using official doc of minio client.
Referral Links
https://...
- 307
0
votes
0
answers
65
views
Connection to ovh cloud in node.js
I want to connect to ovh cloud with axios.
const username = 'user-123456';
const accessKey = '1242121414421';
const secretKey = '2414124141';
const containerName = 'storage';
const ...
0
votes
0
answers
79
views
How can I cache images (express, node, object storage)?
I have an object storage and an express API that accesses the images via the aws-idk framework to resize them, but I have a problem, the process takes a long time. I tested this with Postman and found ...
- 11
0
votes
0
answers
89
views
Use my login system to control access to Digital Ocean Spaces objects
I have my own login system where users can login and perform actions on the web app and one of the things that they can do is upload files.
I decided to use object-based storage such as Digital Ocean ...
- 1,857
0
votes
0
answers
73
views
Copy list of objects across buckets while keeping object prefixes intact
ℹ︎ This question is about the MinIO Client, not the MinIO Server object storage!
I have two S3-compatible buckets A and B, and I need to copy a list of objects from A to B. The catch is that
I do ...
- 544k
2
votes
0
answers
244
views
Multipart upload performance issues to Digital Ocean Spaces from browsers on Windows
I've run into a really strange issue with uploads to Digital Ocean Spaces.
We'd been getting more and more complaints from users (starting beginning of November) that uploads were really slow. We ...
- 546
0
votes
1
answer
61
views
multiple destination for statsd in object-server.conf of swift open stack
I want to send my statsd metrics from object server module of swift open stack to multiple hosts.
How should I change my object-server.conf in /etc/swift path?
The present version of object-server....
- 1
-2
votes
1
answer
2k
views
How do I install/emulate object storage locally? [closed]
There are 3 storage types I know:
File Storage (e.g. regular NTFS file system)
Block Storage (e.g. SQL server installed locally)
Object Storage
As far as I know, Object Storage is only provided by ...
- 72
-1
votes
1
answer
105
views
IBM Cloud Storage - Test Docker
Is there any way to test IBM Cloud Storage offline/on my local system?
For AWS S3 there are plenty of libraries like the following:
https://github.com/findify/s3mock
https://github.com/localstack/...
- 377
0
votes
1
answer
460
views
own domain in s3 vultr bucket
I have an S3 Vultr bucket, I'm using it to stream, but I want to use my own domain, like s3proxy.mydomain.com, instead of sjc1.vultrobjects.com/mybucket.
I want my domain (s3proxy.mydomain.com) to be ...
0
votes
1
answer
829
views
About the MinIO's backup
I used the mc mirror command to perform master-slave replication, and found that it only synchronized the data of the Buckets, but not others, such as policies, groups, users, etc. If I want to ...
- 1
0
votes
0
answers
300
views
HTTP Injection Attack
Similar to SQL injection attack, we believe that our application is prone to HTTP injection attack. We have a HTTP GET API that takes a query parameter and sends the response back making a HTTP call ...
- 2,065
0
votes
0
answers
71
views
Unable to create instance of Cloud Object Storage
I am taking the devops course on Coursera, and one of the course assignment is to create an instance of Cloud Storage Object.
So I already had a little bit of trouble verifying, after a really long ...
- 58
2
votes
1
answer
1k
views
Best practice arround using S3 as file storage for an application with database
I think that using S3-compatible to do file storage for application assets or "attachments" is one of the most common use cases, but I see a couple of issues, which I am unclear on how to ...
- 842
1
vote
1
answer
495
views
Oracle Object Storage Client NullPointerException on request
I am setting up my object storage client as described by the docs:
I'm using sdk version 3.17.0
Tested in java 17 & 19
I've used the config file with the python sdk and it was succesfull
...
- 328
0
votes
1
answer
191
views
IBM COS Python SDK | Create client/resource object authenticated via HMAC keys
Using IBM COS SDK, is it possible to create a COS client/resource without specifying the field ibm_api_key_id, instead specifying some combination of aws_access_key_id and aws_secret_access_key with ...
- 75
0
votes
1
answer
164
views
What do i set my STATIC_ROOT to when using django-storages, boto, boto3?
I am using linode object storage and django project deployed on a linode server. I have everything set up as far as the linode bucket and aws settings in my settings.py but what do I set my ...
0
votes
0
answers
16
views
Cloud Object Storage Account add Another Cloud's Bucket
Can one cloud's object storage user account able to add another s3 service provider's bucket, just like adding bucket of another account from the same cloud?
I have a app need to access raw data in ...
- 1
0
votes
1
answer
683
views
Accessing object storage bucket with a pre-authenticated request using oci java sdk
I have only preauthenticated URL,namespace and bucket name. Is there any way to upload and download data using key with help of oci java sdk. Is there a proper documentation for this, and If there is ...
1
vote
1
answer
3k
views
Is there a faster way to delete a MinIO bucket than `mc rb`
I have a bucket in MinIO with 1M objects of about 8KiB.
Which is stored on a single node (ec2 m5.large) with a single ec2 io2 disk .
If I delete the bucket using the following command, it takes an ...
3
votes
1
answer
2k
views
Testing thanos object storage upload without waiting 2 hours
I was working on a docker-compose file using Prometheus to remote write data to a Thanos receiver. This data would then be queried by Thanos Querier as well as uploaded to minIO. When configuring the ...
- 213
0
votes
1
answer
84
views
Trouble Retrieving Metadata Information from OpenStack Swift Object using REST APIs: Need Help Adding Metadata from Horizon UI or REST API properly
I am new to OpenStack Swift and I am trying to add single/multiple metadata to an object using REST APIs. However, although I receive a 202 success response for the request, when I try to retrieve the ...
1
vote
2
answers
3k
views
Can't upload job artifacts to S3 object storage, 413 request entity too large
I am using Gitlab's CI/CD pipeline to build an image (2,080 GB), it's artifacts are saved in a S3 Object Storage. When trying to push the artifacts the Gitlab-runner throws following Error:
ERROR: ...
- 11
2
votes
1
answer
759
views
Clouflare R2 with Firebase
I'd like to integrate the Cloudflare R2 (object storage) with my Firebase project.
Normally you would use Firebase Storage but since Google Cloud has some insanely high egress fees, I'd prefer to use ...
- 29
1
vote
2
answers
1k
views
Offline object storage solution azure blob api compatible
I am looking for the offline storage solution compatible with azure blob storage API for production environment.
Example Minio for AWS S3.
Microsoft Azure Storage Emulator
is for testing purpose i can ...
- 11
-2
votes
1
answer
594
views
How is it possible to have folders in object storage? [duplicate]
As per my understanding, object storage has a 'flat' structure so you cannot create folders within buckets. However, in both GCP & AWS, I am able to upload regular folders to the buckets, which ...
- 49
1
vote
0
answers
35
views
Rails - Dragonfly image upload to OCI using angular frontend
I am uploading a file in angular and returning rails params in imageData and imageContent variable. Then in rails controller, I try to create a temp file using those parameters and assign it to the ...
- 11
2
votes
2
answers
2k
views
What are the required maven dependencies to upload files to oracle object storage from Spring Boot 3.0 and Java 17?
I am trying to upload files to oracle Object Storage using Spring Boot 3.0 and Java 17. I am getting the following exception
java.util.NoSuchElementException: No http provider available; add ...
- 121
0
votes
1
answer
2k
views
Stored objects/files of Minio do not appear in local disk
I have created a folder in root direcoty, i.e. /data/test and I have already given full permissions to this folder using:
chmod 775 /data/test
Than I started MinIO, and from Minio GUI console, I ...
- 1,911
1
vote
0
answers
300
views
Install Zenko Cloud Server on ubuntu 20.04 for development purpose
For a couple of days, I have been trying to install the Zenko cloud server for development purposes on my ubuntu 20.04 machine. I am new to Docker and definitely not very much comfortable with ...
- 327
2
votes
1
answer
748
views
Are Azure "Block Blobs" an example of "block storage", "object storage", or (somehow) both?
My understanding is that "block storage" and "object storage" are two different, and mutually exclusive kinds of storage. Yet the name of Azure "block blobs" strongly ...
1
vote
1
answer
86
views
Are object storage objects technically object oriented class instances?
While every guide describes object storage objects as data with metaproperties, can they also be referred to as objects (i.e. class instantiation) in a OOP technical sense, of how they are implemented?...
user14467343
0
votes
1
answer
450
views
Limit storage per user on minio
I am setting up minio for the first time and I wonder if there is a way to limit the storage of each user, for exemple the users gets 50GB max to create buckets and store the files.
In the docs I only ...
- 1
0
votes
0
answers
532
views
Upload Object (image file) to Linode Object Storage using PHP (Laravel) with S3Client
What could be the possible solution for this? I'm trying to upload an Object (Image file) to my Linode Object Storage, I have already set up the configuration and credentials but I'm still having ...
