19 questions
0
votes
0
answers
44
views
Ceph storage: erasure code used space calculation
I have an EC pool used for object storage in Ceph. The EC is 3+2 scheme, and the pool has stored 289 TiB.
As I understand, the total storage will be used after doing EC will be:
289 TiB * 5 / 3 = 481 ...
- 54
0
votes
0
answers
51
views
How Does Reed-Solomon Erasure Code Split Data in Rust?
Body:
I'm working on a project in Rust where I'm using the reed_solomon_erasure crate to implement Reed-Solomon erasure codes for data encoding and fault tolerance. I'm struggling to understand how ...
- 1
1
vote
1
answer
898
views
trino : io.trino.spi.trinoexception error reading from hdfs at position caused by java.io.ioexception 4 missing blocks , the stripe is : AlignedStripe
I have trino to query hdfs with hive connector.
not always but sometimes it gets this error :
io.trino.spi.trinoexception error reading from hdfs at position caused by java.io.ioexception 4 missing ...
- 7,366
1
vote
1
answer
154
views
Is there solution for viewing chunks distribution in ceph's pools with erasure code?
According to the picture from this page ceph splits objects on chunks and writes them on osds.
For viewing objects distribution with replication algorithm I can use commands like ceph pg dump or ceph ...
- 13
0
votes
1
answer
307
views
Python erasure coding library that can handle larger strings?
I'm looking for an python erasure coding library that works for larger inputs. So far I've checked out:
unireedsolomon: fails for 256-byte inputs, unmaintained
reedsolo/reedsolomon: fails for a 300-...
- 862
-1
votes
1
answer
5k
views
ceph active+undersized warning
Setup:
6 node cluster with 3 hosts with 12 hdd osd(s) each (36 total) and other 3 hosts with 24 ssd osd(s) each (72 total).
2 erasure code pool that takes 100% of data one for ssd class and the other ...
- 331
1
vote
1
answer
559
views
SeaweedFS EC and cold data
During testing seaweedfs, questions began to appear. Can you explain me, what is hot, warm and cold data in seaweedfs logic?
And second question about erasurecoding: does it work only with warm data? ...
- 31
0
votes
0
answers
217
views
Using ReedSolomon library to erasure code a file
Using [ReedSolomon JavaScript Library][1] to erasure code a local file, it gives me following error :
shardSize 5432
bufferoffset 0
bufferSize 5432
paritysize 10864
sources 3
targets 12
/Users/...
- 43
2
votes
0
answers
398
views
How to use Ronomon Reed-Solomon Library to split file in chunks
I am trying to use Ronomon/Reed-Solomon JavaScript Library, I want to chunk files based on the number of data shards and parity shards. Any example demonstrating the use of ReedSolomon.encode will ...
- 43
0
votes
0
answers
373
views
How to use io.Pipe() for chunking a Large file using Erasure Coding through ReedSolomon Library
Using https://github.com/klauspost/reedsolomon, erasure coding a large file into smaller chunks, I want to use io.Pipe() to create a stream of pipes which connects each other. For example: one pipe ...
- 43
1
vote
1
answer
3k
views
Hadoop 3 : how to configure / enable erasure coding?
I'm trying to setup an Hadoop 3 cluster.
Two questions about the Erasure Coding feature :
How I can ensure that erasure coding is enabled ?
Do I still need to set the replication factor to 3 ?
...
- 54
0
votes
1
answer
500
views
Hadoop 3.0 erasure coding: impact on file read performance?
I am trying to understand the impact erasure coding could have on read performance of a file.
Before that a brief summary of Hadoop 3.0 erasure coding using the Reed-Solomon method. If a file split ...
- 3,640
1
vote
1
answer
430
views
Hadoop 3.0 erasure coding - determining the number of acceptable node failures?
In hadoop 2.0 the default replication factor is 3. And the number of node failures acceptable was 3-1=2.
So on a 100 node cluster if a file was divided in to say 10 parts (blocks), with replication ...
- 3,640
1
vote
1
answer
1k
views
How to configure the erasure coding feature in hadoop3 and is it used for storing cold files only by default?
As per the Hadoop 3.x release notes, they have introduced Erasure coding to overcome the problems with storage.
Erasure coding is a method for durably storing data with significant space savings ...
- 13
1
vote
1
answer
169
views
HDFS Erasure Coding File Creation
I am looking for a way to create a file (e.g. using copyFromLocal) in Apache HDFS and set the Erasure Coding policy in the process.
According to this page, I can use hdfs ec --setPolicy -path <...
0
votes
1
answer
123
views
Erasure Encoding with little resource usage
Hey I'm pretty new to a lot of the erasure encoding concepts. I've mostly only read about Reed-solomon, but it does not fit what I need.
I need to find a technique that can create parity shards on ...
0
votes
2
answers
372
views
Erasure Code for chunked file
Is there an erasure code, which can be applied to multiple chunks (maybe 100 or 200, each few hundred kB) by (somehow) adding redundancy chunks ?
I heard about Reed-Solomon, but it doesn't look like ...
- 2,239
0
votes
1
answer
381
views
survey about the encoding techniques used in cloud computing
I am trying to perform a survey on the encoding techniques which can be applied to a file before storing it in the cloud.
So far I have only encountered Reed Solomon erasure coding mechanism, in ...
- 2,291
0
votes
1
answer
135
views
What is erasure – what 4 restrictions does erasure place on generics? [closed]
Java -
What is erasure – what 4 restrictions does erasure place on generics?
- 25