Questions tagged [split]
The split tag has no usage guidance.
281 questions
0
votes
0
answers
34
views
Rocky 9 split screen settings
Rocky 9 has this nice feature that allows windows to snap into split screen layouts, although by default it's dividing the screen into 3 parts.
Also the borders are a bit thick.
Is there a setting ...
- 1
0
votes
1
answer
67
views
Archiving stdout to multiple tapes
I have large files which are generated on the fly to stdout, one every 24hours. I would like to archive these files progressively on tapes, ideally in a single archive which potentially spans multiple ...
- 1
0
votes
1
answer
81
views
Slow down a `split`
I have a really large archive consisting of really small files, concatenated into a
single text file, with a "" dilimiter. For smaller archives, I would split
the archive using "" ...
- 1,175
0
votes
3
answers
88
views
Read file till special char, copy that section into another file, and continue till eof
I am trying to read a file in Linux and as soon as a "&" character is encountered, I am writing the output to another file, sending that file to another folder and then continuing to ...
- 1
2
votes
1
answer
155
views
Is it safe to slicing byte directly to split a big file?
In my case, the big file is tar.gz, I have myBigFile.tar.gz with size 52GB, I splitted it with chunk size 2GB therefore I have 27 parts file.
Here is the code I program from scratch:
from time import ...
10
votes
3
answers
5k
views
What is the state of the art of splitting a binary file by size?
Some background you can happily skip
Twenty years ago or so, when navigating the web costed a lot, when I was a Windows-only user, and when CDs/DVDs were a large storage means, and when sharing video ...
- 1,747
2
votes
1
answer
170
views
Merge the split file vs Original compressed file
On Ubuntu 22.04,
I found these 2 methods will get different sha256 of archive_tgz
tar czf /a/archive_tgz .
tar czf /dev/stdout . | split -d -b 200M - /a/archive. && cat /a/archive.* > /a/...
- 43
1
vote
1
answer
831
views
What is the max size limit for using split and cat combination?
I have backed up a file in zip format on Ubuntu whose size is 6.5GB. Trying to transfer it to a USB stick so needed to split since I got error “file size too large” while copying. (Ideally I want my ...
- 121
1
vote
3
answers
575
views
Split large file in realtime whilst it is still being written to
I've got a program that will be generating 4 large binary files (400GB+ each) that I need to upload to AWS S3 as quickly as possible.
I'd like to begin uploading before the files are completely ...
1
vote
2
answers
188
views
Does splitting a file in more files necessarily mean that some/all of the overall content will not be where it was?
I guess that given a file of a certain size, not all of its bytes will be contiguous on disk (or will they? Just for existence of the phrase "defragmenting a disk" I assume they will not). ...
- 1,747
0
votes
2
answers
263
views
Linux - split a big file into small files according to specific character and how many it repeats
I have a really big file that looks like this:
>name1
ACGTACGTACGT
ACGTACGTACGT
ACGTACGTACGT
ACGTACGTACGT
>name2
ACGTACGTACGT
ACGTACGTACGT
ACGTACGTACGT
>name
ACGTACGTACGT
ACGTACGTACGT
...
- 3
0
votes
1
answer
13k
views
"gzip: stdin: unexpected end of file" after using tar with split
I have a folder of roughly 600MB of images in a folder output_test. I then create 100MB chunks of tar.gz files using the following command:
tar -czf - output_test/ | split --bytes=100MB -d -a 3 - ...
1
vote
2
answers
198
views
Divide a fasta file with scaffolds into same lenght files respecting the scaffold ID and the sequence
I am currently working with a large fasta file (3.7GB) that has scaffolds in it. Each scaffold has a unique identifier that starts with > on the first line and on the consecutive line it has the ...
- 13
0
votes
0
answers
76
views
How do I prepend variable to open file stream when using split to create csv's?
I have a bash file that takes a large csv and splits the csv into smaller csv's based on this blog https://medium.com/swlh/automatic-s3-file-splitter-620d04b6e81c. It works well as it is fast never ...
- 101
0
votes
0
answers
68
views
How to store multiple files in an array
I am using the split command in cshell to split a file into smaller files. Since the split command names the output files xaa,xab,xac.... i have to rename them. I want to store the split commands ...
1
vote
0
answers
2k
views
How to use Clamscan to scan for files larger than 4GB
I need to scan files larger than 4GB, I tried the following solutions:
1- I tried to edit the clamd.conf and set the
MaxScanSize 100000M
MaxFileSize 100000M
(Just any arbitrary size that's larger than ...
- 11
9
votes
2
answers
1k
views
Split two concatenated files
I have Afile made by the equivalent of cat a.gif b.7z > Afile.
How may I split Afile into the original a.gif and b.7z files?
- 105
0
votes
2
answers
1k
views
rename the filename
The following files exist in the current directory
GCF_000901975.1_ViralProj181986_genomic.fna.gz
GCF_001885505.1_ViralProj344311_genomic.fna.gz
GCF_000901995.1_ViralProj181990_genomic.fna.gz
...
- 53
1
vote
0
answers
1k
views
Split serial port into two virtual ports
This question is related to this post here:
Cloning/Splitting a serial port (COM) port in Ubuntu
I followed the steps to make the virtual ports but the problem is that I need full-duplex ...
- 111
0
votes
0
answers
115
views
FOLD not working on very long lines
I have a fold command as below to split a file with very long line into multiple lines. It works if the file is small. But if the length of the file is too long, it stops adding new line character ...
- 25
0
votes
1
answer
183
views
How to archive and compress large dataset into parts efficiently?
Currently using following model, but one needs double the amount of disk space to restore compressed archive given one has to pipe all parts to tar before one can delete them.
$ COPYFILE_DISABLE=true ...
- 317
1
vote
0
answers
177
views
How can I split a string containing a Bash command without expanding parameters?
I have a single string that contains a shell command line. I need to remove a specific argument from this string (if present), and print the result, while keeping it a valid shell command. How can I ...
- 3,809
16
votes
1
answer
2k
views
Splitting a small file into 512 byte segments changes it, but splitting it in 1k segments doesn't
So I'm trying to split a 64MB file FileCarve.001 into 512 byte segments (each block is 512 bytes long). I need to make sure the file has the same data when split into smaller files, so I cat all the ...
- 171
1
vote
1
answer
325
views
How to forward output of split to sftp?
I have a server that is running out of disk space. I want to move some TB sized directories into a long term archive. This is a tape robot with preferred file size of 16GB. Therefore I want to split ...
- 25
8
votes
1
answer
2k
views
Split ultrawide monitor in 3 virtuals screens
For my work, I try to split my new ultrawide monitor in 3 virtual screens on Ubuntu (I don't want to use tile tools as gTile).
My first test using "xrandr" was OK when I selected "xorg&...
- 81
1
vote
1
answer
616
views
How to split, add text for each split and gzip file
I want to take a huge file data.txt and:
Split it by 256m
Add the first 3 lines of data.txt to each split file
gzip it
data.txt looks like:
aaa
bbb
ccc
<data>
<data>
<data>
...
<...
- 1,395
3
votes
3
answers
1k
views
I don't really understand expansion and quoting on the shell (zsh/bash)
I've read up on a few answers here in reference to quoting variables on the shell.
Basically, here's what I'm running:
for f in "$(tmsu files)"; do echo "${f}"; done
this has the ...
- 2,318
0
votes
2
answers
294
views
Split a file into multiple files, using the first X positions of a line, using sed or awk
I have a large text file, containing up to 2300 sections.
Each section begins, one line before, "PAGE 1"
The page number is at very end of the line,
Example:
T5271-AX1 TITLE ...
- 125
1
vote
1
answer
140
views
Is it with GNU parallel possible to split on character count, but provide full lines only?
I would like to split an input file on character count (ASCII is fine), combined with new lines as well. That is, every group of 10000 character should be seen as one record to be piped into the child ...
-1
votes
1
answer
315
views
I want to replace a specific word in a text file and save the results in multiple text files using perl or ubuntu command?
I have a file results.txt that contains the following:
the word: word1 is a not detected
And I have a txt file contain the list of words as following:
word1 word2 word3 ...
And I want to generate ...
2
votes
2
answers
3k
views
Automating the splitting of a large mp3 file with FFmpeg into multiple files in different time points provided in a text file
I have a large mp3 file; I want to split it into 480 mp3 files in different time points. I want to know if there is an easy way to do it other than splitting it one by one. Can I provide the different ...
- 23
0
votes
1
answer
2k
views
Use output of cat with split command and specified output directory
I'm trying to split a long csv into files of 500 lines each. I want the output files in a specific directory, and I want to leave off the first line of the csv.
I can use split and leave off the ...
- 155
0
votes
1
answer
7k
views
Using split() with awk
I am trying to use split() with awk. I am splitting the contents of $7 with split() into an array, but not sure how to print the contents starting from reverse order. The field $7 is of variable ...
0
votes
1
answer
44
views
Less, but only last element of every line?
So, I want to monitor live logs as they come with less +F, but the logs have a LOT of noise before the actual message - fortunately the line is segmented by | character, with the timestamp being ...
- 109
9
votes
1
answer
2k
views
Save current vim session state and restore it later (e.g. buffers, splits etc.)
As an old VIM user, attracted to it in the first place especially because of its straightforward regex substitutions that allow to work really fast, I'm not really expecting VIM to have such a feature....
- 93
1
vote
4
answers
1k
views
streaming split/merge command?
Does a streaming version of split exist somewhere in Linux?
I'm trying to back up a large amount of data via SSH, but SSH's single threaded encryption is limiting the transfer. This machine doesn't ...
- 1,321
0
votes
1
answer
406
views
how to get split commands output to file
I want to print output of below command to file, can you help me I tried 2>&1 even 2>> also
dd if=/dev/random bs=1 count=1000 status=progress |split -b 10 -a 10
I have other dd command ...
- 85
1
vote
1
answer
192
views
How to create folders based on part of a filename and also extract & rename specific columns using awk
the following code extracts data from csv files contained in a folder, creates new folders (based on filename and date), creates and stores multiple csv files in the said folders. more info on this is ...
- 151
4
votes
2
answers
2k
views
How to split CSV file and create multiple CSV files based on a column
I have a csv file with the format:
Input.csv:
TIMESTAMP,Data1,Data2,Data3,Data4
"2021-01-03 00:00:00",80953,3.243183,2.943338,358.0123
"2021-01-03 00:01:00",80954,2.173187,1.990327,...
- 151
0
votes
0
answers
161
views
I have a `raspivid` stream, which I'm piping to `ffmpeg`, now i'd like to also stream a raw version of it to a socket?
I have a process outputing an MJPEG video stream, which I pipe into ffmpeg to reduce framerate and then to a socket.
raspivid -t 999999 -cd MJPEG -w 1920 -h 1080 -o - | ffmpeg -i - -f mjpeg -r 2 - | ...
- 131
1
vote
0
answers
176
views
Can GNU tar run a filter before adding a file to an archive?
I am trying to archive a directory containing many subdirectories and files, most of which are JPEG files. I would like to run each file through JPEG XL before adding to the tar archive, to reduce ...
- 1,475
-1
votes
3
answers
381
views
Bash Split Output Groups to Columns
I have similar output like below goes so long. I want to create columns with this output but with like groups/blocks. And process output after a pipe.
id: c2b751c227111edfgdghfhfg19079a466e1916f6df4
...
- 1
2
votes
1
answer
2k
views
Split large file into chunks and delete original?
I'm having trouble with a large text file (30GB)
I would like to create smaller files from it (5GB lets say)
But sadly I don't have no more storage (only ~10G is available).
This line:
split -b 5g &...
3
votes
3
answers
2k
views
How to create split tar archive in multiple stages to save space?
I have a very large folder that I am trying to create a tar archive of. The issue is I don't have enough extra free space to store the entire archive so I want to create say 100-200GB chunks of the ...
0
votes
0
answers
869
views
Split command to chunk a large CSV - with new line character
I'm splitting a large CSV file with small chunks using the split command. Each file should contain 1000 rows only.
The command I used is,
cat large.csv | split -a3 -l1000 --filter='gzip > $FILE.gz'...
- 385
0
votes
1
answer
1k
views
Split file from a streaming output with custom name and gzip
I have a script with a while loop that will print a text. I want to save it into files with custom name.
Script:
#!/bin/bash
while true
do
echo "Press [CTRL+C] to stop.."
done
I can run ...
- 385
4
votes
1
answer
946
views
Split large files into a number of files in unix
i have a file that contains several lines(9074 line), i want to split the file into 10 files contain the same number of lines except the last file that contain the remainder number of lines
split -l `...
- 41
-1
votes
1
answer
2k
views
linux + how to split ISO file to separate pieces [duplicate]
we are using the following ISO file for rhel upgrade
4.2G RHEL-7.6.iso
since we have problem to copy this ISO file between WIN to Linux machine
and that because sometime network problem cause ISO ...
- 13.7k
2
votes
2
answers
956
views
Split long SQL expression at delimiter
Context
I'm trying to import a dump that have some long lines (8k+ character) with SQL*Plus, so I face the error SP2-0027: Input is too long (> 2499 characters). This is a hard-coded limit and ...
- 123
0
votes
1
answer
370
views
How to split single rows of a CSV file into multiple depending on delimiter in 2nd column
My base.csv file has several entries a sample of which is below.
NAME,IP_ADDRESS,PRI_IP,APP_CODE,APP_NAME,OWNER
test2,10.9.7.12/10.222.101.11,10.9.9.12,545,myapp1,owner1
mytest,10.9.7.12,10.9.9.12,545,...
- 519