Easily edit performance videos with ffmpeg using Ruby

1.First of all

As a recent trend of music players, there is a trend that people cannot gather in one place and play together, so they take a video for each and edit it later to make one video and publish it. I don't have much knowledge of music and basically I don't play it myself. However, there are people involved in music around me, and I am often asked to edit videos. To deal with this, I used Ruby to run ffmpeg and created an environment where video editing can be done semi-automatically.

ruby_movie_ffmpeg.png

2. Processing flow

Now, even people without knowledge can easily shoot performance videos using smartphones. … But that also means that everyone sends videos in different formats, video sizes, and frame rates. I think there is a better way to prevent the deterioration of the video, but it is processed by a step-by-step method as follows.

3. Source

Since it is supposed to run in a Windows environment, those who refer to it should be careful about how to use the character .

movie_on_playing.rb


control_file = ARGV[0]
cfile = File.open(control_file, "r")

def system_log( exe, workd )
  fho = File.open("#{workd}\\log.txt", "a")
  fho.write("#{exe}\n")
  fho.close
  res = system( exe )
  unless(res)
    puts "# Error! ffmpeg failed. See #{workd}\\log.txt to confirm command."
    exit
  end
end

out_width  = 0
out_height = 0
movie      = []
resize     = []
volume     = []
fadein     = []
delay      = []
delay_f    = []
to_time    = []
blackf     = []
crop_xy    = []
crop_size  = []
skip       = []
row_div    = []
row_flag   = true
ri = 0

for s1 in cfile
  next if s1 =~ /^#/
  a1 = s1.chomp.split(" ")
  if(a1[0] == "ffmpeg")
    ffmpeg = a1[1]
    next
  end
  if(a1[0] == "work")
    workd = a1[1]
    next
  end
  if(a1[0] == "out")
    out_file_name = a1[1]
    system("title #{out_file_name}")
    next
  end
  if(a1[0] == "movie")
    movie << a1[1]
    next
  end
  if(a1[0] == "resize")
    resize << a1[1]
    next
  end
  if(a1[0] == "volume")
    volume << a1[1]
    next
  end
  if(a1[0] == "fadein")
    fadein << a1[1]
    next
  end
  if(a1[0] == "delay")
    delay << a1[1]
    delay_f << a1[2] if(a1[1].to_f > 0)
    next
  end
  if(a1[0] == "time")
    to_time << a1[1]
    next
  end
  if(a1[0] == "crop")
    crop_xy << a1[1]
    crop_size << a1[2]
    if(row_div.size == 0)
      out_width = out_width + a1[2].split("x")[0].to_i
    end
    if row_flag
      out_height = out_height + a1[2].split("x")[1].to_i
      row_flag = false
    end
    next
  end
  if(a1[0] == "skip")
    skip << a1[1]
    next
  end
  if(a1[0] == "row_div")
    row_div << movie.size
    row_flag = true
    next
  end
end

cfile.close

Dir.mkdir(workd) unless File.exist?(workd)
efile_mp4 = []
efile_mp3 = []
ovl_vx = 0
ovl_vy = 0
ovl_a = []

fho = File.open("#{workd}\\log.txt", "a")
fho.write("\n#{Time.now}\n")
fho.close

out_width_e = out_width
out_height_e = out_height
fi = 0
w_reset = false
row_div << 9999

for i in 0...movie.size
  filen = movie[i].split("\\")[-1].split(".")[0]
  filee = movie[i].split("\\")[-1].split(".")[1]
  file_rsz = "#{filen}_#{resize[i]}"
  rx = resize[i].split("x")[0]
  ry = resize[i].split("x")[1]
  ####################
  #### resize part ###
  ####################
  
  if(delay[i].to_f > 0.0)
    fade_time = 0
  else
    fade_time = -1 * delay[i].to_f
  end
  if(delay[i].to_f > 0)
    if(fadein[i].to_i > 0)
      fade_in = "fade=in:0:#{fadein[i]},"
    else
      fade_in = ""
    end
    system_log( "#{ffmpeg} -y -i \"#{movie[i]}\" -vf \"#{fade_in}scale=#{rx}:#{ry}\" \"#{workd}\\#{file_rsz}.#{filee}\"", workd ) if(skip[i] == "off")
  else
    system_log( "#{ffmpeg} -y -i \"#{movie[i]}\" -vf \"scale=#{rx}:#{ry}\" \"#{workd}\\#{file_rsz}.#{filee}\"", workd ) if(skip[i] == "off")
  end
  ######################
  ### black mov part ###
  ######################
  if(delay[i].to_f > 0.0)
    dname = delay[i].gsub(".","p")
    bfile = "black_#{resize[i]}_#{dname}.#{filee}"
    if(delay_f[fi] == nil)
      frate = "30000/1001"
    else
      frate = delay_f[fi]
    end
    system_log( "#{ffmpeg} -y -f lavfi -i \"color=c=black:s=#{resize[i]}:r=#{frate}:d=#{delay[i]}\" -f lavfi -i \"aevalsrc=0|0:c=stereo:s=44100:d=#{delay[i]}\" \"#{workd}\\#{bfile}\"", workd ) if(skip[i] == "off")
    blackf << bfile
    fi = fi + 1
  else
    blackf << "black_0"
  end
  #######################
  ### concat/cut part ###
  #######################
  if(blackf[i] != "black_0")
    # add plus delay
    con_filen = "#{workd}\\concat_#{i}.txt"
    con_file = File.open(con_filen, "w")
    con_file.write("file #{workd}/#{blackf[i]}\n")
    con_file.write("file #{workd}/#{file_rsz}.#{filee}\n")
    con_file.close
    file_cn = "#{file_rsz}_con"
    system_log( "#{ffmpeg} -y -safe 0 -f concat -i \"#{con_filen}\" -c:v copy -c:a copy -map 0:v -map 0:a \"#{workd}\\#{file_cn}.#{filee}\"", workd ) if(skip[i] == "off")
  elsif(delay[i].to_f < 0)
    # add minus delay
    cut_time = -1 * delay[i].to_f
    file_ct  = "#{filen}_cut"
    system_log( "#{ffmpeg} -y -i \"#{workd}\\#{file_rsz}.#{filee}\" -ss #{cut_time} \"#{workd}\\#{file_ct}.#{filee}\"", workd ) if(skip[i] == "off")
    file_cn = file_ct
  else
    # not add delay
    file_cn = file_rsz
  end
  ###########################
  ### fade, crop, to part ###
  ###########################

  crop_x = crop_xy[i].split("x")[0]
  crop_y = crop_xy[i].split("x")[1]
  crop_w = crop_size[i].split("x")[0]
  crop_h = crop_size[i].split("x")[1]
  cropt = "crop=#{crop_w}:#{crop_h}:#{crop_x}:#{crop_y}"
  
  if(w_reset)
    out_width_e = out_width
    out_height_e = out_height_e - crop_h.to_i
    w_reset = false
  end

  if( out_width_e - crop_w.to_i > 0 || out_height_e - crop_h.to_i > 0)
    padt = ",pad=#{out_width_e}:#{out_height_e}:0:0"
  else
    padt = ""
  end
  if(to_time[i].to_f > 0)
    tot = "-to #{to_time[i].to_f + delay[i].to_f}"
  else
    tot = ""
  end
  file_crp = "#{file_rsz}_cropped"
  if(delay[i].to_f > 0)
    system_log( "#{ffmpeg} -y -i \"#{workd}\\#{file_cn}.#{filee}\" -vf \"#{cropt}#{padt}\" #{tot} \"#{workd}\\#{file_crp}.#{filee}\"", workd ) if(skip[i] == "off")
  else
    if(fadein[i].to_i > 0)
      fade_in = "fade=in:0:#{fadein[i]},"
    else
      fade_in = ""
    end
    system_log( "#{ffmpeg} -y -i \"#{workd}\\#{file_cn}.#{filee}\" -vf \"#{fade_in}#{cropt}#{padt}\" #{tot} \"#{workd}\\#{file_crp}.#{filee}\"", workd ) if(skip[i] == "off")
  end
  
  out_width_e = out_width_e - crop_w.to_i
  if(row_div.size > 0)
    if(i == row_div[ri] - 1)
      ovl_vx = 0
      ovl_vy = ovl_vy + crop_h.to_i
      ri = ri + 1
      w_reset = true
    else
      ovl_vx = ovl_vx + crop_w.to_i
    end
  else
    ovl_vx = ovl_vx + crop_w.to_i
  end
  ovl_a << [ovl_vx, ovl_vy]
  ############################
  ### mp3 out, volume part ###
  ############################
  vol_e = volume[i].to_f / 100
  system_log( "#{ffmpeg} -y -i \"#{workd}\\#{file_crp}.#{filee}\" -vn -acodec libmp3lame -ar 44100 -ab 256k -af \"volume=#{vol_e}\" \"#{workd}\\#{filen}.mp3\"", workd ) if(skip[i] == "off")
  efile_mp3 << "#{workd}\\#{filen}.mp3"
  efile_mp4 << "#{workd}\\#{file_crp}.#{filee}"
end

####################
### overlay part ###
####################
in_file = ""
for efile in efile_mp4
  in_file = "#{in_file} -i \"#{efile}\""
end
ovlt = ""
for i in 0...ovl_a.size - 1
  if(ovlt == "")
    ovlt = "overlay=x=#{ovl_a[i][0]}:y=#{ovl_a[i][1]}"
  else
    ovlt = "#{ovlt},overlay=x=#{ovl_a[i][0]}:y=#{ovl_a[i][1]}"
  end
end
system_log( "#{ffmpeg} -y#{in_file} -filter_complex \"#{ovlt}\" -an \"#{workd}\\out_movie.#{filee}\"", workd )
######################
### add audio part ###
######################
in_file = ""
for efile in efile_mp3
  in_file = "#{in_file} -i \"#{efile}\""
end
system_log( "#{ffmpeg} -y -i \"#{workd}\\out_movie.#{filee}\" #{in_file} -filter_complex \"amix=inputs=#{efile_mp3.size}:duration=longest:dropout_transition=2\" \"#{out_file_name}\"", workd )

3.1 Control file A control file was created and read to specify the location of the video file and the editing method. The following is an example. In addition, the video after entering the keyword "row_div" will be connected vertically instead of horizontally.

control.txt


ffmpeg C:\ffmpeg\bin\ffmpeg.exe
work   work
out    out.mp4

movie  org_data\a-san.mp4
resize 1280x720
volume 100
fadein 60
delay  1.5
time   208.5
crop   0x0 640x720
skip   off

movie  org_data\b-san.mp4
resize 1280x720
volume 120
fadein 60
delay  -2.5
time   212.5
crop   320x0 640x720
skip   off

ruby movie_on_playing.rb control.txt

With commands like, ffmpeg commands are sequentially generated and executed based on the control information.

4. Explanation of processing

The processing of each part is explained according to the ○○ part of the annotation part.

4.1. Resize part Resize the video according to the "resize" setting in the control file. Put the resize value after "scale =" and generate the following command.

ffmpeg.exe -y -i "org_data\a-san.mp4" -vf "scale=1280:720" "work\a-san_1280x720.mp4"

If the value of "delay" is +, the video that does not show anything before will be connected, so fade in will also be set here.

ffmpeg.exe -y -i "org_data\a-san.mp4" -vf "fade=in:0:30,scale=1280:720" "work\a-san_1280x720.mp4"

4.2. Black movie part If the value of "delay" is + (to delay the start of the video), the video will be created without sound for the specified number of seconds (Note: not without sound). Put the value after "d =" and generate the following command.

ffmpeg.exe -y -f lavfi -i "color=c=black:s=1280x720:r=30000/1001:d=1.5" -f lavfi -i "aevalsrc=0|0:c=stereo:s=44100:d=1.5" "work\black_1280x720_1p5.mp4"

Since this process tends to cause problems in the video after concatenation, the frame can be changed in the 3rd column with "delay [seconds] [frame rate]". However, this method may not work well, so when editing a video using ffmpeg, it is easier to shoot with a sufficient margin of the original video and then cut it. If you want to have a video shot from now on, tell the performer, "Press the record button, wait about 5 seconds, and then play."

4.3. Concat/cut part If the value of "delay" is-(to start the video early), put it in the value of "-ss" and generate the following command.

ffmpeg.exe -y -i "work\a-san_1280x720.mp4" -ss 2.5 "work\a-san_1280x720_cut.mp4"

If the value of "delay" is +, it will be linked with the previous video without video.

ffmpeg.exe -y -safe 0 -f concat -i "work\concat_1.txt" -c:v copy -c:a copy -map 0:v -map 0:a "work\a-san_1280x720_con.mp4"

4.4. Fade, crop, to part Here, you can crop the video and specify the end time. If the value of "delay" is-, set fade in here. In ffmpeg, the size after cropping and the crop position are specified in this order, but since it was a little difficult to understand personally, the control file uses the format "crop [crop position] [size after cropping]". It is converted like "crop (X1) x (Y1) (X2) x (Y2)" → ffmpeg "crop = (X2): (Y2): (X1): (Y1)". One of the strengths of controlling using Ruby is that you can decide the format yourself according to your preference.

The value of "pad =" is calculated from the cropped size of each video. "-to" is calculated from the values of "time" and "delay" in the control file.

ffmpeg.exe -y -i "work\a-san_1280x720_cut.mp4" -vf "fade=in:0:30,crop=640:720:0:0,pad=1280:720:0:0" -to 210.0 "work\a-san_1280x720_cropped.mp4"

4.5. Mp3 out, volume part Extracts sound-only files from timed videos (with -ss and -to set). In ffmpeg, 100% volume is "1.0", but in control file, 100% is "100". Put "setting value / 100" after "volume =" and generate the following command.

ffmpeg.exe -y -i "work\a-san_1280x720_cropped.mp4" -vn -acodec libmp3lame -ar 44100 -ab 256k -af "volume=1.0" "work\a-san.mp3"

4.6. Overlay part The individually adjusted videos are combined into one video by overlay. Calculate from the value of crop and generate the following command.

ffmpeg.exe -y -i "work\a-san_1280x720_cropped.mp4" -i "work\b-san_1280x720_cropped.mp4" -filter_complex "overlay=x=640:y=0" -an "work\out_movie.mp4"

4.7. Add audio part Finally, input all the sounds and you're done. Hold the mp3 file name as an array, connect them with -i in order, enter the size of the array as the value of "amix = inputs =", and generate the following command.

ffmpeg.exe -y -i "work\out_movie.mp4" -i "work\a-san.mp3" -i "work\a-san.mp3" -filter_complex "amix=inputs=2:duration=longest:dropout_transition=2" "out.mp4"

This "out.mp4" is the completed video.

5. Other

One of the major drawbacks of entering commands in the CUI and editing videos is that you cannot edit while watching or listening to the timing and volume. Therefore, the video created by the above method is

Etc. may be requested. At this time, ** for videos that are not adjusted, use the previous result ** to speed up, create an item called "skip" in the control file, and if this is "on", skip the processing of that video. I am doing it.

6. At the end

ffmpeg is a very sophisticated and easy-to-use tool that I've always found useful, but it's a lot of work to type dozens or hundreds of commands, especially to calculate and enter video size-related information. Is required. I hope this article has helped you to ease that effort.

Recommended Posts

Easily edit performance videos with ffmpeg using Ruby
Notes on using FCM with Ruby on Rails
Try using GPS receiver kit with RaspberryPi3 (Ruby)
Feel the basic type and reference type easily with ruby
Feel the basic type and reference type easily with ruby 2