Questions tagged [hdfs]
The hdfs tag has no usage guidance.
11 questions
1
vote
0
answers
23
views
Hadoop + warnings as slow block-receive from data-node machines
We have Hadoop cluster with 487 data-nodes machines ( each data-node machine include also the Service node-manager ) , all machines are physical machines ( DELL ) , and OS is RHEL 7.9 version.
Each ...
- 13.7k
0
votes
0
answers
54
views
Hadoop Namenode heap size tuning
NameNode process is executed in Java virtual machine,
and Java object which NameNode creates is managed in Java virtual memory. As the fles or directories are created, inode objects and block objects ...
- 13.7k
1
vote
0
answers
307
views
Clear RAM Memory Cache and buffer on production Hadoop cluster with HDFS filesystem
we have Hadoop cluster with 265 Linux RHEL machines.
from total 265 machines, we have 230 data nodes machines with HDFS filesystem.
total memory on each data-node is 128G and we run many spark ...
- 13.7k
0
votes
1
answer
1k
views
CPU LOAD AVRG + how to deal process with D state
we can see from our RHEL 7.6 server ( kernel version - 3.10.0-957.el7.x86_64 ) that following process are with D state ( they runs from HDFS user )
Note - D state code means that process is in ...
- 13.7k
1
vote
2
answers
524
views
Convert list of hdf5 files to netcdf files with same name using shell scripting
I have a list of datasets containing satellite data arranged in monthly folders as follows:
01 02 03 04 05 06 07 08 09 10 11 12
These folder are further divided into daily data folder for ...
- 13
0
votes
0
answers
126
views
How to move the last n files in hdfs
I have a folder in HDFS contains 830000 files, and I want to move the last "8797" files enter code here to another folder in HDFS? I tried using xargs but didn't work fine. Any other ideas?
...
- 101
0
votes
0
answers
133
views
how to run script from user hdfs without password
we create the following script on rhel 7.6
/home/run_tasks
and in visudo we configured
%sudo ALL=(ALL:ALL) ALL
root ALL=(ALL) ALL
hdfs ALL = (ALL) ALL
hdfs ALL= (root) NOPASSWD: /home/...
- 13.7k
2
votes
0
answers
1k
views
ssh: connect to host localhost port 22: Connection refused
I have installed hadoop and ssh. hadoop was working fine, then today I am getting the error below when I run the command sbin/start-dfs.sh:
Starting namenodes on [localhost]
localhost: ssh: connect ...
- 31
1
vote
0
answers
307
views
master: ssh: connect to host master port 22: Connection refused
i am trying to start my hadoop cluster using the command "start-dfs.sh" but am getting errors as shown below
Starting namenodes on [master]
master: ssh: connect to host master port 22: Connection ...
- 31
-2
votes
1
answer
90
views
Hadoop cluster + designing number of disks on data node machine and min requirements
we are using HDP version - 2.6.5 , and HDFS Block replication is 3
we are try to understand data nodes disks min requirements for production mode and according to the fact that Block replication=3
...
- 13.7k
1
vote
1
answer
3k
views
how to find the owner of user and group from user HDFS
we can grant the permissions as hdfs user for hive as the following
su hdfs
$ hdfs dfs -chown hive:2098
but how to do the opposite way?
in order to verify the owner of hive and hive group?
- 13.7k