You have to be careful about the commands you use every day in the production environment.

Introduction

It is the 13th day of Ateam cyma Advent Calendar 2019. Today, Ateam Co., Ltd. EC Business Headquarters cyma engineer @ihsiek will be in charge.

"People who have done it in a production environment Advent Calendar 2019" is very popular. Recently, the CI / CD environment has been enhanced and the chances of shell access to the production environment are decreasing, and the failure stories of others (other companies) will be a cane that will not fall down so as not to follow the same rut. I would like the engineers of the company to read such failure stories.

So, I would like to write a story of failure in a production environment due to an operation that I was completely unaware of the risks I took in my previous job.

TL; DR

• When using the --force option of the Linux command, it is better to check the behavior of the version to be used.
• Operation verification in the production environment is useless.
• OS update operation is required.
• I recommend "Advent Calendar 2019", which allows you to use the failure stories of others as your own food, so by all means I want you to read it.

What happened

• Files stored on NAS (Network Attached Storage) suddenly disappeared.

Background

The day before

• Released a certain system
• At that time, there was no CI / CD environment, and it was an operation to build after pulling master with the git command.
• I forgot the reason, but only for this release, I made a backup for each directory in the previous section and rolled back the backup with the cp command.
• Run cp -rf [backup] [destination] to verify that the backup works properly
• I released it again and returned home because the operation verification was completed successfully.

On the day

• Telephone inquiries from on-site members
• "The files that should have been uploaded to the NAS have been lost for the past few years, why?"
• Investigating but unknown cause
• When I checked because the access log to the file was left on the NAS, there was a history of deleting the file near the release time of the previous day.
• Since I haven't executed a command to delete the file in the first place, I have no idea what to do, so I decided to scrutinize the access log of the file for a while.
• By observing closely, it was confirmed that only the file with the symbolic link disappeared from the production server and the others remained, and the access log showed that it behaved like a cp command, so let's experiment.
• Create an appropriate directory (/ tmp / hoge) in / tmp
• Prepare directories and files with symbolic links
• Execute cp -rf / tmp / hoge / tmp / hoge
• Confirm that the contents of the directory with the symbolic link have completely disappeared.
• To identify the cause
• It seems that cp -rf is bad, so I checked the specifications of the -f and --force options with man cp and found the cause.

Commentary

The following is the result of man cp in the environment of occurrence.

If you try to copy the file to itself(Except for issuing an error message)do nothing.
When copying to an existing file, the destination file is ‘open(path,O_WRONLY|O_TRUNC)’Opened at.
When copying to a new file, the new file is ‘open(path,O_WRONLY|O_CREAT,mode)’Created with.
The copy destination file exists and-If opening a file in this format fails when the f option is given,
cp deletes existing files(Or unlink)Try.
Then, if the deletion is successful, the instruction is continued as a copy to a new file.

Although it is on the second line from the bottom, the destination of the symbolic link is looking at the same file in the backup and release destination, so

1. Failed to open the file
2. Delete existing files
3. Continue the instruction as a copy to a new file
4. Failed in 3 because the copy source file disappeared in 2.

That seems to have been the end of the matter. By the way, the behavior of the cp command seems to be different for each distribution, but I have confirmed that such a problem did not occur even at that time if the version was upgraded properly.

Summary

I think that the trouble was caused by the fact that I was careful that the cp command was just a copy command and that it would be okay, that I had prepared a way to verify only in production, and that I was not able to thoroughly update the OS. .. Here are my lessons learned:

• Recognize the assumption that commands behave differently for each distribution and version.
• Avoid using the -f, --force options as much as possible, and check the specifications if necessary.
• If you can only verify in the production environment, make a copy of the production environment and create a place where you can verify.
• If you are using services such as AWS and GCP, it is recommended that you easily create a dummy for the production environment from the copied image.
• I thought it might be troublesome if I was doing container orchestration, but I wouldn't encounter such a problem if I was operating a container in the first place.
• OS update operation is necessary, so let's do it systematically.

in conclusion

How was the 13th day of Ateam cyma Advent Calendar 2019? Since it was a story of my previous job, I looked back while thinking that it would not be possible in my current workplace, but I pray that everyone will not follow the same rut. We apologize for the inconvenience caused to all the people involved at that time: bow_tone1:

Also, in the sense that I want you to feed on the mistakes of others, "Advent Calendar 2019" is really Since it is recommended, I would like various people to read it, including those who have never experienced production operation. Tomorrow is @namedpython, a new graduate engineer who is a little messy returnee. I'm looking forward to posting newcomers who are expected to build a solid data analysis platform.

Ateam Co., Ltd. is looking for colleagues with a strong spirit of challenge to work with.

If you are interested in working as an engineer, please see cyma's Qiita Jobs.

For other occupations, see Ateam Group Recruitment Site.