Questions tagged [awk]
AWK is a text-processing language. It is mainly used to manipulate and process structured data, and to generate reports.
637 questions
0
votes
2
answers
132
views
print (only) line preceding a match
Consider a text-input that looks like this:
a
b
pattern
c
pattern
d
e
pattern
a
--
pattern
z
I would like to get only the lines that precede a match.
E.g. if my pattern is pattern, the output should ...
- 493
-2
votes
2
answers
59
views
i am not able to print sum of two columns from a live stream of data using awk
so i am trying to monitor how much bandwidth get used by transmission while torrenting. I searched and find an utility nethogs.if i run
sudo netogs -t | grep transmission-gtk
i get
transmission-gtk/...
0
votes
1
answer
71
views
How to append line which matches pattern to previous line
Attempting to format tcpdump.txt file:
Lines containing date, interface, protocol, etc.
Lines containing network source device.port, destination device.port, flags, etc.
Extra lines (discard)
Input:
...
- 21
1
vote
1
answer
151
views
Linux awk script commands
I'm working on a project to extract network info from tcpdump that requires many awk pipe commands for formatting prior to import into a database used for reporting, i.e.,:
tcpdump -tttt -vi any -c ...
- 21
1
vote
2
answers
88
views
How to extract IP address and port from column in csv file using awk?
My goal is to use awk to parse tcpdump cvs file to extract network tracking and reporting information from columns #22 and #23 (columns and sample data) below:
1,2,3,4,5,6,,....,,22,23,,...
20-09-24,...
- 21
1
vote
1
answer
29
views
Print more than one column of `lsof' output while sorting and counting uniq values using one of those columns
I want to print only chosen columns of lsof output
$1,$2,$3,$5,$8,$10
and use uniq -c on the PID column while printing all of the initial columns of output.
I am on macOS. I am using awk, sort, uniq ...
- 446
0
votes
0
answers
37
views
Search a pattern from one file and replace next line with another pattern in another file in Shell script
I have two files file test.txt and ref.txt. I want to read ref.txt which has comma seperated 3 values. I want to match pattern in test.txt with first value in ref.txt, if found replace next line if ...
- 1
0
votes
2
answers
147
views
bash + how to insert date before line
lets say I want to pipe the date output in the beggining of some text
for example
echo "this line is test line" | date
and expected output should be
Wed May 22 14:55:10 UTC 2024 this line ...
- 931
0
votes
2
answers
134
views
How to extract a text in between two patterns (potentially containing floating lookbehind) with only one invocation?
I'm trying to match any characters (except line break characters) in between xxx<1+ whitespaces>" and whitespace.
The format of the input string is the following:
xxx<1+ whitespaces>&...
- 392
0
votes
0
answers
39
views
Read git label in version format on Windows
I have the following bash script which read the last label from git which is in version format (1.2.3.4).
git tag --sort=-refname | awk 'match($0, /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/)' | head -n 1
I ...
- 123
0
votes
3
answers
94
views
Awk command to find the most common word in a file?
Lets say I have a file that is simply a list full of dates and names, And I need to look for the "team name" that is repeated most in that list? Player name is located at position $1 and ...
2
votes
1
answer
349
views
How to Print, when columns have spaces - awk?
Problem:
desired columns have text with spaces SAMSUNG MZNLN128HCGR-000H1
undesired columns have text with spaces, see VENDOR = M.2 SSD
awk only printing the first word.
awk not printing the column ...
- 499
-1
votes
1
answer
72
views
AWK replace some special new line to one line
i have some log like this:
2023-11-15T08:59:28.000000+00:00 database-1 # Time: 231115 8:59:28
# User@Host: rdsadmin[rdsadmin] @ localhost []
# Thread_id: 3 Schema: QC_hit: No
# Query_time: 0....
- 1
0
votes
2
answers
830
views
awk string replacement - properly escape '&' character
I am working on one of our applications to update the docker image from using static configs for each environment to a single template using dynamic variable insertion with awk. Our Dockerfile calls a ...
2
votes
2
answers
509
views
I want to get every command that has 3 letters and only three letters
I don't get the output I expect using apropos -k . |awk '/^.../ {print $1}' am I completely overthinking this? In this sample output only zsh should have come back.
I also tried this
apropos -k . | ...
- 474
0
votes
1
answer
860
views
Moving files to a directory with spaces in its name
I'm trying to create a directory that has in its name the beginning of the names of the files it contains. I need this in order to archive it later. I have run into problems when moving files if the ...
alex2020
1
vote
1
answer
118
views
How to parse upload and download speed with ifstat and printf?
In bash I'm trying to display rounded (no decimals) download and upload speed in readable format i.e something like that:
Download 16KB Upload 200KB
I got this but its not quite working:
ifstat 1 1 |...
- 15
0
votes
2
answers
1k
views
Find and replace string in a file using 'sed' or 'awk'
Let's say I have the following input file:
yhara.runner.mng1.nna1.X9HCJG.1 yhara.runner.mng1.nna1.Z2HCJG.1 100.000 100
yhara.runner.mng1.nna1.X9HCJG.1 yhara.runner.mng1.nna1.AUEM0K.1 94.144 72
...
- 103
0
votes
1
answer
735
views
How can I combine "Search for multiple strings (patterns)" with "Print rows after a hit" in a string/search and output a common file?
I am using the windows version of grep "GNU Grep".
syntax: grep [OPTIONS] PATTERN [FILE...]
How can I combine different options with different patterns?
Both lines listed here work on their ...
- 3
0
votes
1
answer
418
views
Loop with awk output field
the output of my script
grep 'Interface GigabitEthernet' * | \
sed 's/104-8-20-23.lightspeed.gnvlsc.sbcglobal.net/104.8.20.23/g' | \
sed 's/0.0.0.0: //g' | \
sed 's/.log:/ /g' | \
sed 's/Line protocol ...
2
votes
0
answers
55
views
SFTP user login details real-time filtering
I have enabled the SFTP login log into the default logfile /var/log/syslog and tried to filter the login time of each user and insert it into the database.
But the filtering is not worked as I ...
- 121
1
vote
2
answers
392
views
Multiply numerical value after a string in a json file by 3?
I'm trying to replace a number in a set of text files. The number will be different for each file, but it will always occur after a specific string.
Here's the original file:
"structures&...
- 11
3
votes
4
answers
757
views
How to remove all newlines from a file?
I have the command cat file.txt | gawk '{print $2}' > test which displays:
1
1
0
0
1
1
Afterwards:
vi test
:%s/\n//
Which outputs: 110011
My question is I don't know how I could incorporate this ...
-1
votes
1
answer
175
views
write a efficient for loop in bash for multiple files (.PDB files)
I have problem with this script:
#!/bin/bash
for filename in /home/hb/pg/ensemble/pdbs/P24941/raw_pdb/*.pdb; do
grep COMPND $filename | grep "CHAIN:" -B 1 | sed 's/COMPND//g' | sed 's/...
- 9
3
votes
3
answers
582
views
How do I delete every ocurrence of a word that is followed by another, without altering the whitespace?
Suppose my file looks like this:
foo bar
foo bar bar
foo foo bar foo
What I want to do is delete every foo that is followed by a bar (i.e. there is only whitespace between them), without ...
- 61
1
vote
0
answers
269
views
pipe awk output in sed command
I have multiple lines in a file names.txt, let's say
my name is Jim your name please
what is good Harry potter
how is he Jhony and me
and so on , more than 500 lines.
I use awk command to search/ ...
- 43
0
votes
0
answers
294
views
bash loop to create m3u file with sed awk grep
I am trying to auto create m3u file for my IPTV software with the help of sed/awk or grep command,but don't have much experience in this. expert help required to do this. working on Windows-10 PC ...
- 43
1
vote
2
answers
774
views
Are linux commands (grep, awk etc) part of the system, or the shell?
I'm a developer, so I know my way around a terminal, but I never really dove into it. I can write bash scripts (albeit on a beginner level, not my area of expertise), but I now need to understand ...
- 13
0
votes
1
answer
57
views
Linux - Find string and then end of section
I have an XMLTV file. I need to get all of the programme sections for a specific channel. An example snippet:
<programme start="2023031305000 -0400" end="2023031305300 -0400&...
0
votes
1
answer
249
views
AWK: "invalid regexp: Invalid collation character" -- how do I make it valid?
I have an awk script that must process millions of records, but I need to remove any containing a multibyte character.
In one environment where I work, the following simplified shell sequence ...
- 181
1
vote
1
answer
435
views
How to find the "pattern" in FASTA file and record the cordinates along with header
I am looking for a solution to search for a string of 17 base pairs in the FASTA file that is the human reference genome.
To clarify, in simple words and put the resources, i have tried grep function ...
- 13
1
vote
2
answers
870
views
Delete lines between two patterns (with multiple occurrences) around a third pattern (single occurrence)
I'm trying to edit an uncompressed PDF on macOS Monterey, and I want to delete objects that contain a certain pattern in their description. Such objects start with "X 0 obj" (where X is the ...
- 11
0
votes
1
answer
2k
views
Awk exact string match in a specific column only returns header row
I have a 6-7M row CSV file that end users wish to review with Excel or Numbers. Each user only needs to review a subset of the file, so I was hoping to offer an awk one-liner that they can use until ...
- 105
0
votes
2
answers
186
views
How to group a flat list into categories and create a tab-separated value file using awk?
Crosspost: https://www.nixcraft.com/t/converting-a-list-into-a-tab-separated-file-grouped-by-values/4517
I have a text file with a list of values. The goal is to create a tab separated values file, ...
0
votes
3
answers
286
views
Get a valid path with random directory name from a text file
How to to get the valid path from a string?
For example I have the following text in file.txt that will always generate with a random path using the following patern:
Some error occurred. rerun the ...
- 103
0
votes
0
answers
204
views
Simplify NETSTAT monitoring output
I'm monitoring suspicious incoming connections using
sudo watch -n 30 "netstat -antp"
which provides more information than I need, so based on the answer I found here,
I've tried to ...
1
vote
1
answer
494
views
How to copy first line of text file to start or end of all another lines?
I am currently editing down multiple large text files (maybe 500 different text files) , and need to copy first line of text files to start or end of all another lines
For example this is one of my ...
- 19
-1
votes
2
answers
2k
views
Convert CSV to text and automatically remove quotes and a specific word - Linux
I need to convert .csv to .txt format via CL (Command line) and automatically remove quotes and a specific word - Linux.
Example:
I have multiple exported csv files that have:
At first row I have: &...
- 1
2
votes
0
answers
1k
views
Find and replace multi-line block in multiple files (bash/linux)
I'm trying to find a way to find a text block that meets the criteria:
block starts with "google = {"
block ends with next "}" character found - ignoring everything else.
Example
...
- 21
0
votes
0
answers
453
views
Awk regex: how to remove all punctuation except when in the middle of a word?
Following information gathered on this thread, I wanted to use this code to fill my needs, however the code provided wasn't working for more than one punctuation character (e.g. '...'). I thus wanted ...
1
vote
1
answer
169
views
get the top urls from the logs
I have a numbers of log files:
adsfs.demo.com_2022-07-11-0000-0001_cn.tgz
adsfs.demo.com_2022-07-11-0000-0002_cn.tgz
adsfs.demo.com_2022-07-11-0000-0003_cn.tgz
adsfs.demo.com_2022-07-11-0000-...
- 665
0
votes
1
answer
182
views
Insert new line if file contain string
I need to insert 'import packagename' to the line after imports. In files which contains the specific word.
What I try:
find iOS/Chat \
-type f \
-name "*.swift" \
-exec gsed -e "/...
- 101
0
votes
1
answer
977
views
Inserting string between two known strings with sed or awk
I've used sed a bit before in the past for updating strings but I haven't been successful with inserting a string in between two known strings.
I attempted to use this but it didn't update anything.
...
- 103
0
votes
2
answers
5k
views
How to group and count all lines in a file that contain specific string
I want to filter all lines from a file that contain mySearchString and after that group them together and count them.
Example find all lines that contain 9791
AB-9791___Foo
AB-9791___Foo
DE-9791___Bar
...
- 285
0
votes
1
answer
510
views
Summing real numbers from a text file
We have the following file:
more /home/avrg.bc.txt
3.4
54.2
4.5
3.5
2.3
89.6
.
.
.
We want to calculate the sum of the numbers in the file as 3.4 + 54.2 + 4.5 + ....
- 931
0
votes
1
answer
65
views
keep each line containing x only if column y is the first containing x. remove the rest containing x
I'm trying to reduce a results file that contains multiple results for sometimes the same query and I only want the top result from each query.
Using bash and bash tools like awk, sed, etc. I would ...
- 11
0
votes
1
answer
286
views
Removing Duplicate from File
I have two files fileA and fileb
FILEA
1.1.1.1
2.2.2.2
3.3.3.3
4.4.4.4
5.5.5.5
FILEB
4.4.4.4
2.2.2.2
Output should be
1.1.1.1
3.3.3.3
5.5.5.5
Size of File A is around 60GB and Size of File B is 1GB
...
- 143
0
votes
1
answer
318
views
Enclose the year in parenthesis in a filename
I have a group of files in a folder that I want to rename; each file has a 4-digit year as part of the filename, I would like to enclose the years in parenthesis.
"Ghostbusters 3 2016.avi" ...
2
votes
2
answers
127
views
Find and delete near duplicate lines with certain punctuation
I have a text file with millions of lines. Some lines contain the same alphanumeric sequence, but differ in case and punctuation. I consider these lines duplicates. I want to remove any duplicate line ...
- 35
1
vote
1
answer
1k
views
Extract last 5mins log from log file using shell
I need to extract last 5mins log from log file.
Here is my log file
[2022-02-08 13:26:21:352] [ERROR] [iBus Connection LifeCycle - CCMHost_DummyDevice_Backup_AD2::Management:::NRMCMO_FLPROD2] [com....
- 11