All Questions
320 questions
0
votes
0
answers
33
views
Incompatible configuration used between Spark and HBaseTestingUtility
We are using the MiniDFSCluster and MiniHbaseCluster from HBaseTestingUtility to run unit tests for our Spark jobs.
The Spark configuration that we use is :
conf.set("spark.sql....
0
votes
0
answers
14
views
Why hbase StoreFileSize (from jmx) and hdfs dfs -du -h are different?
i want to get the hbase table StoreFileSize,
but result of the command curl hostip:port/jmx?qry=Hadoop:service=HBase,name=RegionServer,sub=Tables are different from hdfs dfs -du -h /hbase/data/default/...
- 43
0
votes
0
answers
36
views
Unable to Submit MapReduce Job from Java Client to Hadoop Cluster Running in Pseudo-Distributed Mode
I'm working on a project where I need to perform aggregations on the result of an HBase table scan using MapReduce and store the result in another HBase table. To achieve this, I've set up a Hadoop ...
- 13
1
vote
0
answers
61
views
How new version of data is read from Hfile or Memstore rather than old version of value from block cache in Hbase?
Let's say I have a key 100 with its value in HFile. During the read the key and its value is pulled from HBase and set in BlockCache and send to the client. Next when the value for key 100 is read it ...
- 825
1
vote
0
answers
250
views
hbase import module don't succeed
I have to move some hbase tables from one hadoop cluster to another. I have extracted the tables using
bin/hbase org.apache.hadoop.hbase.mapreduce.Export \ <tablename> <outputdir> [<...
- 11
-2
votes
1
answer
434
views
Java Program to load the multiple csv files from HDFS, read column values and load into HBASE?
I am new to big data and hadoop.
I have a requirement where I have to upload 100 csv files with information in it ( ex. People information ie. Name, age, city) to hdfs
and then using a java program to ...
- 19
0
votes
1
answer
412
views
Is there a way to Implement conditional statements in Static Block
I have a class HDFSHbaseInstance which will load Hbase config in static block
public class HDFSHbaseInstance {
private static FileSystem hadoopFS = null;
private static Configuration config = ...
- 1,002
0
votes
1
answer
270
views
Why aggregations are not possible in HBase?
I am wondering why aggregation operations such as SUM / AVG are not natively supported by HBase. I got to know that there are several workarounds for achieving the same, but while there are these ...
- 89
-1
votes
1
answer
185
views
How to send a csv file from machine 1 to machine 3 using nifi in machine 2?
I have 3 machines for my scenario
Machine 1: has a .csv file
Machine 2: Nifi is installed and running
Machine 3: HDFS and Hbase is installed and running.
Now I have to send the .csv file from machine ...
- 11
-2
votes
1
answer
274
views
How to send a csv from my local machine HBASE using nifi?
I am very new to DBMS and nifi platforms.
I am learning it and now I want to send a .csv file with some data from my local machine to HBASE using nifi.
It would be good to know the steps to be ...
- 11
1
vote
2
answers
607
views
HBase store file storage location
Each Region Server in HBase is assigned some regions of some tables. These Region Servers may have many regions, and their corresponding Store files have replicas on other HDFS Datanodes.
If one ...
- 381
1
vote
1
answer
203
views
How to get the count of columns in each column family in HBASE
How to get the count of columns in each column family in HBASE.
I have a table_name T, and columnfamily 'f' and 'm', have 10 columns each. how to count the number of columns in each column family.
- 21
0
votes
1
answer
151
views
Cannot create directory while running `sbt IntegrationTest/test` with HBaseTestingUtility.startMiniDFSCluster
When creating a mini HDFS cluster in integration tests with the help of HBaseTestingUtility.startMiniDFSCluster, tests are running fine in IntelliJ IDEA but fail when running via sbt IntegrationTest/...
1
vote
1
answer
388
views
Is there a maximum storage space configuration in HDFS or HBase?
Is there a maximum storage space configuration in HDFS or HBase?
I've found
dfs.data.dir: "Comma separated list of paths on the local filesystem of a DataNode where it should store its blocks&...
user3091996
0
votes
1
answer
309
views
Truncating a table in HBase did not free up space for HDFS
I was having problems with HBase and HDFS, as I understood HBase runs on top of HDFS and thus I assumed it used HDFS to store its data. The problem was that HDFS was close to its maximum space ...
user3091996
0
votes
1
answer
233
views
HBase pre-splitting
I want to pre-split HBase table for e.g. on 5 regions like this hbase(main):001:0> create 'table','info',SPLITS => ['1000','2000','3000','4000']
If one of the regions (e.g., the region is 1000 ~ ...
- 21
2
votes
0
answers
222
views
java.lang.UnsatisfiedLinkError when I start hbase
There are three node3(node1,node2,node3) in my cluster.I have installed hadoop-2.5.2,zookeeper-3.5.4-cdh5.10.0,jdk-7u67-linux-x64 on my Centos server ,they all work well.
Then I install hbase-0.98.6-...
- 21
1
vote
1
answer
762
views
HBase logcleaner does not delete the oldWALs files
In my lab, HBase archive Write Ahead Logs i.e. oldWALs files are not deleted and oldWALs directory is growing quickly in terabyte.
8.1 K 24.4 K /hbase/.hbase-snapshot
0 0 /hbase/.hbck
...
- 59
0
votes
0
answers
121
views
Can't connect to HBase container on remote SSH server
I want to connect to HBase container on remote server that I connect to with ssh through VPN. Let's say it's 10.0.0.10.In /etc/hosts i placed:
10.0.0.10 hbaseaddr
In my java code I use hbase-client:
...
- 27
0
votes
1
answer
415
views
How to enable short-circuit read in HBase? (hbase-site.xml)
So, I have HDFS configured to do short-circuit read and I checked the short-circuit read is enabled and performed very well, when I use HDFS read APIs. (I can see the logs that the feature is ...
- 157
0
votes
0
answers
109
views
Hadoop Hdfs DFSAdmin - Cannot initialize Cluster or hdfs points to file:/// not hdfs://
I'm using hadoop DFSAdmin api to report the dead blocks for an hdfs backup utility.
enter link description here
But by invoking my jar in the env (included the hadoop jars in classpath)
java -...
- 1,179
0
votes
2
answers
339
views
Is HBase block cache dropped when restart HBase?
I am currently confused with the HBase block cache.
I have come to aware about the existence of HBase block cache and it is for read performance. The question is that, if I stop the HBase by command,
...
- 157
0
votes
2
answers
523
views
Is it possible to use HDFS storage types/policies together with HBase?
HDFS has a feature called storage types/policies - it makes possible to store files on stores with different properties (fast SSD or slow but cheap archival storage).
I wonder if it's possible to use ...
- 15.5k
6
votes
1
answer
5k
views
Why do we need column families in Hbase?
What is the reason for having column families? Example:
Scenario 1 :
Table Row-Key ColumnFamily1 ColumnFamily2 ColumnFamily3
Scenario 2 :
Table1 Row-Key Column1...ColumnN
Table2 Row-Key Column1......
user12318711
0
votes
1
answer
79
views
HBase table size not changing after inserting data
i am trying to monitor my hbase table size while inserting data. at first i had 2 rows and size was 12.7k; after adding 10 rows it increased to 24.5 then i inserted 2000 rows but the size was still 24....
1
vote
0
answers
1k
views
HBase installation in cluster - Master is initializing error
I am installing a multi node HBase cluster - 3 master and 3 region servers.
After installation, all the nodes are up and connected, the HBase UI, everything connected and all fine.
I am able to HBase ...
4
votes
1
answer
7k
views
HBase:Difference between Minor and Major Compaction
I am having trouble understanding why major compaction is differ from minor compaction. As far as I know, minor compaction is that merge some HFiles into one or little more HFiles.
And I think major ...
- 157
0
votes
1
answer
81
views
how to set dummy variable as path in multipleinput in map-reduce
I have passed hbase table data using scan to the mapper. Is there any way to set Dummy path or variable as path argument in MultipleInput()
List<Scan> scans = new ArrayList();
scans.add(data);
...
- 163
0
votes
1
answer
257
views
How do we set TTL for cell in Load Data to HBase Table using completebulkload
I have a use case where I need to use Bulk Import data to HBase.
I was following this link -> http://hbase.apache.org/book.html#completebulkload
from the official site, everything is working fine ...
0
votes
1
answer
659
views
HBase oldWALs, pv2-(sequence number).log are not cleaned up
I'm wondering if someone has any idea about the following problem.
The oldWALs directory on hdfs only seems to be cleaned partially, the (quite small) pv2-(sequence number).log files just keep ...
- 25
0
votes
0
answers
174
views
Hbase - Is it possible to create inputsplits for a mapreduce job based on column count of hbase row?
We have to run mapreduce job on a hbase table which has millions of rows and cells.
We know each region will be assigned with one mapper for processing. If we use a job to scan a table with 100 small ...
- 109
0
votes
1
answer
269
views
Can one row key exist in multiple hbase regions?
As per my understanding, HBase Tables are divided horizontally by row key range into “Regions.” A region contains all rows in the table between the region's start key and end key.
Suppose I have a ...
- 109
0
votes
0
answers
849
views
HDFS files under data/hbase/oldWALs folder growing too much
Our HDFS file system is growing a lot. Here is the result after I run hdfs dfs -du -h /data/hbase, you can see most of the spaces are in "oldWALs" folder:
0 0 /data/hbase/.tmp
...
0
votes
1
answer
718
views
How do I write hbase data to HDFS as text
There are already 300,000 pieces of data of that day in HBASE, how to write the data into HBASE in TXT format, and each TXT file corresponds to the data of a day, such as:
2020.03.25/high.TXT（High is ...
- 1
0
votes
1
answer
2k
views
Can not find or load main class org.apache.hadoop.hbase.util.HBaseConfTool
Im new to hbase and ive been scouring the internet for the answer to this.I believe this is a class path issue. Im running java 1.8, hadoop 3.2, and hbase 2.2. The hbase documentation says to sets
...
0
votes
1
answer
709
views
Using NiFi to ingest json data in HBase
I'm trying to write a pretty simple XML file stored in HDFS to HBase. I'd like to transform the XML file into json format and create one row in HBase for each element within the json array. See ...
- 85
0
votes
1
answer
272
views
Is it possible to migrate data from one HBASE DB to another in a different HDFS cluster?
I have an HBASE database (with N-tables) in two cluster (West and East). Both with the same DB schema. West and East are in different domains.
How do I migrate data, from each table, with a specific ...
- 101
-1
votes
1
answer
33
views
Which hadoop framework(s) can be used in near RDBMS scenario
We are in analysis phase for a project in which we are replacing old storing system to be based on hdfs and hive used for reporting and viewing.
But we got a request if we can make use of hdfs to use ...
- 29
0
votes
1
answer
109
views
Location of ingested data through geomesa-hbase ingest command
I am totally new to geomesa ,I am using geomes-hbase-2.11-2.3.1 and I have followed below site for data ingestion.
https://www.geomesa.org/documentation/tutorials/geomesa-examples-gdelt.html
It ...
5
votes
3
answers
5k
views
HBase - hbase:metadata holds info about non existing RegionServer ID - "Master startup cannot progress, in holding-pattern until region onlined."
I cannot start Hbase Master because I am getting this error:
[Thread-18] master.HMaster: hbase:meta,,1.1588230740
is NOT online; state={1588230740 state=OPEN, ts=1569328636085, server=...
- 1,788
2
votes
0
answers
733
views
Connecting to HDFS namenode running in docker container from outside host VM
I have a HBase + HDFS setup, in which each of the HBase master, regionservers, HDFS namenode and datanodes are containerized.
When running all of these containers on a single host VM, things work ...
- 1,036
1
vote
1
answer
146
views
Can I decrypt HDFS data while ingesting into HBase without having an intermediate decrypted file?
I am considering using HBase for my project. I have some encrypted files (they are encrypted at the file-level, not column-level or row-level encryption), and would like to store their decrypted ...
- 930
0
votes
0
answers
105
views
WARN : Your Hadoop installation does not include the StreamCapabilities class from HDFS-11644
I run a Kerberised Hbase Cluster over HDFS and I get this warning in the master log when I start hbase :
WARN [Thread-15] util.CommonFSUtils: Your Hadoop installation does not include the ...
- 151
1
vote
1
answer
796
views
How to process all Hbase data with Hive
I have a HBase with the 750GB data. All data in the HBase are time series sensor data. And, my row key design is like this;
deviceID,sensorID,timestamp
I want to prepare all data in the hbase for ...
- 59
0
votes
1
answer
2k
views
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat
I am using sparklauncher to start a spark application. In spark application I am inserting data into hive table and using some hbase-hive linked table in join query. I have added hive-hbase-handler-1....
- 31
1
vote
0
answers
30
views
Hbase coporcessor connot be loaded the new one? [duplicate]
I have created a coprocessor and uploaded it to hdfs. I loaded this coprocessor with alter command for my table,
alter 'my:table', METHOD => 'table_att', 'coprocessor$1' => 'hdfs://master/user/...
- 11
1
vote
1
answer
581
views
Hive - Huge 10TB table repartitioning (Adding new partition columns)
Techies,
Background -
We have 10TB existing hive table which has been range partitioned on column A. Business case has changes which now require adding of partition column B in addition to Column A....
- 11
0
votes
1
answer
2k
views
How to load csv file from hdfs to hbase table using Dimporttsv
I am trying to load csv file into an hbase table using shell command Dimporttsv.
The csv files reside in a dir in my hdfs (/csvFiles)
the csv file was generated from a mysql table with the following ...
- 418
0
votes
0
answers
103
views
Check RPC Encrypted Data in motion on HDFS/YARN/Spark/Hbase
We have specific requirement to check/validate data in motion when RPC encrypted data flows for all communication with client as well as within big data components like HBase/Phoenix/HDFS/YARN/Hive ...
- 1,073
0
votes
1
answer
29
views
Moving Large and frequently updating MySQL table with composite key to HDFS
I have MySQL Inventory table which don't have Auto Increment Id but has composite key and last modified date(YYYY-mm-DD HH:MM:SS) and will update very frequently.It has last 3 years data around 10 ...
