GNU Linux Bash -> FIND SEARCH FILE CONTENT TEXT STRING – output file list – iterate over file list colorful mark search term lines

12.Jun.2015

Administration / Server, Bash / Terminal / Scripts, GNU-Linux

searching effectively and easily for files and content of those files is key.

this kind of content-search might take a while, because it searches EVERY file.

this will output/generate a list of files with full path – that you then can further process:

search via filenames:

+ case in sensitive (A and a)

time find / -iname "*FileName*" -not -path "/proc/*" -not -path "/sys/*" -not -path "/dev/*"

if you want to exclude external mounted drives (centos7 it is /run/*, debian/ubuntu it is /mnt/*)

time find / -iname "*FileName*" -not -path "/proc/*" -not -path "/sys/*" -not -path "/dev/*"  -not -path "/run/*"  -not -path "/mnt/*" -not -path "/media/*"

iterate over found files:

basically what one can do, is pass all found /path/to/filename/that/matches.file to another command or – in this case: script:

# search current dir and all sub dir for file.m4a
# pass all file.m4a to /scripts/mp3.sh
# which will convert them into mp3
find . -type f -iname *.m4a -exec /scripts/mp3.sh {} \;
# the mp3 script looks like this and requires working ffmpeg:
ffmpeg -i "$1" -c:a libmp3lame -ac 2 -q:a 2 "$1.mp3";
# (modern almost identical alternative to ffmpeg is avconv)

search via content:

log files:

# colorful search for this
grep -a -r -i -E --color=auto "this" /var/log

# sort and filter duplicate messages
# the "color" get's lost in the process
grep -a -r -i -E --color=auto "this" /var/log | cut -d " " -f 4- | sort | uniq -c

# qnap (busybox) tested/approved
su; # become root or use sudo alternatively

# search in all files (also on mounted drives)
# will search all files, except those under /proc /sys /dev
# and meassure time it took to search
time find / -type f -not -path "/proc/*" -not -path "/sys/*" -not -path "/dev/*" | xargs grep --color=auto -s -l -i "STRING THAT YOU WANT"

# explanation of options taken:
# -a = Process a binary file as if it were text (otherwise grep might just display "binary file matches")
# -type f = generate a list of files, not directories
# -s = remove the "grep no such file or directory" error messages littering console
# -i = ignore case (case in-sensitive searcH)
# -l, --files-with-matches
# Suppress normal output; instead print the name of each input file from which output would normally have been printed. The scanning will stop on
# the first match. (-l is specified by POSIX.)

output example:

output list of /path/file and content – colorful please!

now – very cool colorful way to output the location of the files and the lines in which the keyword was found. (i think it only lists the first occasion of SEARCH_TERM then goes on to the next file)

let’s say for the reason of curiosity you would like to search your whole harddisk for “Linus” or “Torvalds”… you could use:

grep -a -r -i -E --color=auto "Linus" /*; # colorful path to file and outputs line

debian8 output screenshot:


grep -i -r --color "Torvalds" /* 2> /dev/null;
# search recursively in all files and subfolders of /root for files that contain case-insensitive Torvalds

grep -i -r --color "Torvalds" /* 2> /dev/null | tee search_for_torvalds.txt;
# this will search your entire system for "Torvalds" or "torvalds" and outputs findings on screen as well saving it into search_for_torvalds.txt
# unfortunately tee destroys the color-full output of grep - so black and white only

output example for SUSE12:

more examples:

grep is capable to –color highlight what it found

fgrep – fast grep – is faster but lacks ability of regular expressions and –color.

let’s grep some text – lets do some grepping exercise:

\WPATTERN\W makes grep search for alone standing word PATTERN (surrounded by whitespace).

suse12:~ # grep '\W465\W' /etc/services
urd                465/tcp      # URL Rendesvous Directory for SSM
igmpv3lite         465/udp      # IGMP over UDP for SSM  [Toerless_Eckert]

# atlernatively you could use
suse12:~ # grep " 465/" /etc/services
urd                465/tcp      # URL Rendesvous Directory for SSM
igmpv3lite         465/udp      # IGMP over UDP for SSM  [Toerless_Eckert]

^ marks the ^beginning of a file – so this pattern matches all lines that start with https

suse12:~ # grep ^https /etc/services
https              443/tcp      # http protocol over TLS/SSL  [Kipp_E_B_Hickman]
https              443/udp      # http protocol over TLS/SSL  [Kipp_E_B_Hickman]
https              443/sctp     # HTTPS  [Randall_Stewart] [RFC4960]
https-wmap         8991/tcp     # webmail HTTPS service  [Fred_Batty]
https-wmap         8991/udp     # webmail HTTPS service  [Fred_Batty]

# you can either use:
find . -type f | xargs grep -s -l "STRING THAT YOU WANT";

cd /etc/grub.d;
grep -r -i --color windows; # search for term "windows" (-i case insensitive) in all files in all subfolders (-r recursive)
test/test.txt:windows
30_os-prober: Windows*)
30_os-prober:menuentry '$(echo "${LONGNAME} $onstr" | grub_quote)' --class windows --class os \$menuentry_id_option 'osprober-chain-$(grub_get_device_id "${DEVICE}")' {
...

vim artikel.txt; # create an file called artike.txt first

# and fill it with this content
Affenschwanzbaum
Affenbrotbaum
affenbrotbaum
Affenkletterbaum
Heute ist Dienstag
Das wichtigste ist Heute
Paul
Peter
Maier
Meier
Das ist mir wichtig

mkdir test; # create a new folder
cp artikel.txt ./test; # create copy in that folder

vim termine; # create an file called artike.txt first

# and fill it with this content
24.12. Weihnachten
04.01. Geburtstag
05.01. Schnee schieben

# save and quit
date +%d.%m." Ganter zuhören" >> termine; # adding one more line with the current date in the format DD.MM.

#### now the testing ####

grep --color ist artikel.txt; # search for "ist" in file "artikel.txt"
Heute ist Dienstag
Das wichtigste ist Heute
Das ist mir wichtig

grep --color M[ae]ier artikel.txt; # search for Maier but also for Meier
Maier
Meier

grep --color [A-Z] artikel.txt; # search and highlight all CAPITAL LETTERS
Affenschwanzbaum
Affenbrotbaum
Affenkletterbaum
Heute ist Dienstag
Das wichtigste ist Heute
Paul
Peter
Maier
Meier
Das ist mir wichtig

grep --color [^A-Z] artikel.txt; # search and highlight for all non-capital letters
Affenschwanzbaum
Affenbrotbaum
affenbrotbaum
Affenkletterbaum
Heute ist Dienstag
Das wichtigste ist Heute
Paul
Peter
Maier
Meier
Das ist mir wichtig

grep --color ^Heute artikel.txt; # search for all lines that start with "Heute"
Heute ist Dienstag

grep --color Heute$ artikel.txt; # search for all lines that end with "Heute"
Das wichtigste ist Heute

grep --color wichtig artikel.txt; # search for all lines that contain "wichtig"
Das wichtigste ist Heute
Das ist mir wichtig

grep --color "\<wichtig\>" artikel.txt; # search for whole word "wichtig" as whole word
Das ist mir wichtig

grep --color "Affen.*baum" artikel.txt; # search for all lines containing "Affen... arbitrary amount of chars ...baum" in file artikel.txt
Affenschwanzbaum
Affenbrotbaum
Affenkletterbaum

grep --color "Affen\(brot\|schwanz\)baum" artikel.txt; # search for all lines containing "Affen... followed by either brot or schwanz ...baum" in file artikel.txt
Affenschwanzbaum
Affenbrotbaum

grep -c --color "Affen\(brot\|schwanz\)baum" artikel.txt; # count lines containing "Affen... followed by either brot or schwanz ...baum"
2

grep -i --color "affen\(brot\|schwanz\)baum" artikel.txt; # search for lines containing "affen... or Affen followed by either brot or schwanz ...baum" case in-sensitive
Affenschwanzbaum
Affenbrotbaum
affenbrotbaum

grep -l --color "affen\(brot\|schwanz\)baum" *; # search in all files in the current directory - output the filenames that have matching content
artikel.txt

grep -r --color ei *; # search in all files in the current directory
artikel.txt:Meier
meintext.txt:sofort los schreiben.
test/artikel.txt:Meier

grep -r -l --color ei *; # search in all files in the current directory - output only the filenames that contain "ei"
artikel.txt
meintext.txt

grep -r --color baum *; # search recursively in all files and subfolders of the current directory - for the pattern "zweite"
artikel.txt:Affenschwanzbaum
artikel.txt:Affenbrotbaum
artikel.txt:affenbrotbaum
artikel.txt:Affenkletterbaum
test/artikel.txt:Affenschwanzbaum
test/artikel.txt:Affenbrotbaum
test/artikel.txt:affenbrotbaum
test/artikel.txt:Affenkletterbaum

grep -r --color -v baum *; # search recursively in all files and subfolders of the current directory - that do NOT contain the pattern "zweite"
artikel.txt:Heute ist Dienstag
artikel.txt:Das wichtigste ist Heute
artikel.txt:Paul
artikel.txt:Peter
artikel.txt:Maier
artikel.txt:Meier
artikel.txt:
artikel.txt:Das ist mir wichtig
public_html/.directory:[Desktop Entry]
public_html/.directory:Icon=folder_html
public_html/.directory:Type=Directory
scripts/md5sumBenchmark.sh:#!/bin/bash
...

grep --color "/bin/bash"$ /etc/passwd; # output all users that use /bin/bash
at:x:25:25:Batch jobs daemon:/var/spool/atjobs:/bin/bash
bin:x:1:1:bin:/bin:/bin/bash
daemon:x:2:2:Daemon:/sbin:/bin/bash
ftp:x:40:49:FTP account:/srv/ftp:/bin/bash
games:x:12:100:Games account:/var/games:/bin/bash
lp:x:4:7:Printing daemon:/var/spool/lpd:/bin/bash
man:x:13:62:Manual pages viewer:/var/cache/man:/bin/bash
news:x:9:13:News system:/etc/news:/bin/bash
nobody:x:65534:65533:nobody:/var/lib/nobody:/bin/bash
root:x:0:0:root:/root:/bin/bash
uucp:x:10:14:Unix-to-Unix CoPy system:/etc/uucp:/bin/bash
user:x:1000:100:user:/home/user:/bin/bash

grep --color -v "/bin/bash"$ /etc/passwd; # output all users that do NOT use /bin/bash (inverted pattern from last example)
ftpsecure:x:486:65534:Secure FTP User:/var/lib/empty:/bin/false
gdm:x:483:482:Gnome Display Manager daemon:/var/lib/gdm:/bin/false
mail:x:8:12:Mailer daemon:/var/spool/clientmqueue:/bin/false
messagebus:x:499:499:User for D-Bus:/var/run/dbus:/bin/false
nscd:x:496:495:User for nscd:/run/nscd:/sbin/nologin
ntp:x:74:489:NTP daemon:/var/lib/ntp:/bin/false
openslp:x:494:2:openslp daemon:/var/lib/empty:/sbin/nologin
polkitd:x:497:496:User for polkitd:/var/lib/polkit:/sbin/nologin
postfix:x:51:51:Postfix Daemon:/var/spool/postfix:/bin/false
pulse:x:487:487:PulseAudio daemon:/var/lib/pulseaudio:/sbin/nologin
rpc:x:495:65534:user for rpcbind:/var/lib/empty:/sbin/nologin
rtkit:x:488:488:RealtimeKit:/proc:/bin/false
scard:x:484:484:Smart Card Reader:/var/run/pcscd:/usr/sbin/nologin
srvGeoClue:x:490:65534:User for GeoClue D-Bus service:/var/lib/srvGeoClue:/sbin/nologin
sshd:x:498:498:SSH daemon:/var/lib/sshd:/bin/false
statd:x:489:65534:NFS statd daemon:/var/lib/nfs:/sbin/nologin
systemd-bus-proxy:x:491:491:systemd Bus Proxy:/:/sbin/nologin
systemd-timesync:x:492:492:systemd Time Synchronization:/:/sbin/nologin
vnc:x:485:485:user for VNC:/var/lib/empty:/sbin/nologin
wwwrun:x:30:8:WWW daemon apache:/var/lib/wwwrun:/bin/false

grep `date +%d.%m.` termine; # search inside termine for the current date (today is 26.04.2017)
26.04. Ganter zuhören

grep $(date +%d.%m.) termine # same as last example, but different writing
26.04. Ganter zuhören

Search for filename: Display modification date:

update from Firefox 60 to 68 created new profile, basically disgarded all bookmarks.

luckily firefox does automatic backups of bookmarks in a strange “,jsonlz4” format under ~/.mozilla/firefox/profile/bookmarkbackups/ (your profile folder will be different, type firefox into url-address-bar: “about:support” and one can restore all bookmarks from this backup file like this -> https://support.mozilla.org/en-US/kb/restore-bookmarks-from-backup-or-move-them

# find all firefox bookmark files
# display modification date, so it's easy to find the most recent backup 
find . -iname "~/*bookmarks*" -exec stat  -c "%n %y" {} \;