What to do if you can't find well with grep's -f option

Introduction

If you are using grep -f and it happens that only **" the line at the end of the search string "is hit *** The line feed code may be the cause. → The content is that if you correct the line feed code with sed or ʻecho`, it will work.

(* For Windows. Maybe it behaves differently for Mac?)

• If you want to use each line in the file as a search string, you can use methods such as grep -f or while statement to read one line at a time (while read line).

-Grep -f option

grep -f [search string file] [search target file]

・ Use while statement

while read line do grep "$ line" [Search target file] done <[search string file]

(* In this writing method, it is necessary to break the line at the last line of ** [Search string file] ** (otherwise only the last line will not be read). Lines that meet multiple conditions will appear many times. Please note that it will be done)

What's happening

As shown below, when searching for each line in the file, the phenomenon that only "the line with the search string at the end" is selected may occur (or rather, it happened).

Search string file (file.txt）

AAA
BBB

Search target file (test.txt）

AAAxxxxxxxx
xxxxxAAAxxx
xxxxxxxxAAA
BBBx BBB xx
xxxxxxxxBBB
xxAAAxxBBBx
xxxCCCxxxxx

Only the line with the last search string is selected

$ grep -f file.txt test.txt
xxxxxxxxAAA
xxxxxxxxBBB

$ while read line
> do
> grep "$line" test.txt
> done < file.txt
xxxxxxxxAAA
xxxxxxxxBBB

Example of workaround

1. sed command

Anything is fine, so if you use sed to recreate ** [search string file] **, it will work fine.

$ sed 's/^//' file.txt > file2.txt
$ grep -f file2.txt test.txt
AAAxxxxxxxx
xxxxxAAAxxx
xxxxxxxxAAA
BBBx BBB xx
xxxxxxxxBBB
xxAAAxxBBBx

2. echo command

For the while statement, rereading $ line with ʻecho` also works.

$ while read line
> do
> grep `echo $line` test.txt
> done < file.txt
AAAxxxxxxxx
xxxxxAAAxxx
xxxxxxxxAAA
xxAAAxxBBBx  ##
BBBx BBB xx
xxxxxxxxBBB
xxAAAxxBBBx  ##Appears many times if multiple conditions are met

Other remedies

Examples of other line feed code conversion methods ・ Conversion of line feed code

Commentary

The cause of this behavior was that the line feed code was different between Windows and Unix. In other words, if you use a file created on Windows as a search string, the ** \ r ** part will interfere with the search (search for "** search string + \ r **"), so search at the end. It only hit if there was a string. (In fact, if you remove the line breaks on the second line of ** file.txt **, "BBB" will be searched normally, and conversely, ** test.txt ** will be sed. If you change the line feed code from CRLF to LF by executing, nothing will be output.)

Quote: Check line feed code

file.When there is no line break in the second line (BBB) of txt

$ grep -f file.txt test.txt
xxxxxxxxAAA
BBBx BBB xx
xxxxxxxxBBB
xxAAAxxBBBx

Search target file (test.When sed is executed in txt) (nothing is output)

$ sed 's/^//' test.txt > test2.txt
$ grep -f file.txt test2.txt

Using sed or ʻecho` will convert the line feed code from ** CRLF (\ r \ n) ** to ** LF (\ n) ** so that the search will work. Become.

Before and after the sed command

## ----------------------Before sed(CRLF)
$ file file.txt
file.txt: ASCII text, with CRLF line terminators

$ od -c file.txt
0000000   A   A   A  \r  \n   B   B   B  \r  \n
0000012

## ----------------------After sed(LF)
$ file file2.txt
file2.txt: ASCII text

$ od -c file2.txt
0000000   A   A   A  \n   B   B   B  \n
0000010

Before and after the echo command

$ cat hoge.txt
hoge

$ while read line
> do
> echo `echo $line` > hoge2.txt
> done < hoge.txt

## ----------------------before echo(CRLF)
$ od -c hoge.txt
0000000   h   o   g   e  \r  \n
0000006

## ----------------------After echo(LF)
$ od -c hoge2.txt
0000000   h   o   g   e  \n
0000005

Reference: ・ [Sed] Command (Basic) -Edit text file -How to replace the changed line feed code with Windows sed and return it from LF to CRLF

In the case of Mac, CR was adopted in the previous MacOS, but after MacOSX it is said that it is the same LF as Unix-like OS.

[Linux] Convert line breaks