We will proceed with the following contents --When using optical flow ――Problems in using 1 I can't find the feature points where I want --Solution 1 --Problems in using 2 Lost if you move fast
Optical flow is a technology that detects and tracks characteristic parts of an image. It's a very useful technology, but when you try it, you face various problems. Here I will write that it was a little troublesome for me to use. In particular, we will introduce how to find the feature points and what to do if you lose them.
If there is a characteristic place because it is a method of finding a characteristic part in the image, the place you really want may not be characterized. One example is strong lighting.
Let the person first specify the location of the feature point and follow it. For example, if you cut out a part of the image like the following source code and detect the feature points of the image, it will look good. An example algorithm is shown below.
When the feature points were detected like this, they tracked even if there was strong lighting in the video. The source is shown below, but since the code is only extracted, some error will occur if it is left as it is.
def extractFeatures(self, gray, rect, features):
featureList = cv2.goodFeaturesToTrack(gray,100,0.01,10)
for feature in featureList:
if rect[0] <= feature[0][0] <= rect[2] and rect[1] <= feature[0][1] <= rect[3]:
features = self.addList(features, feature[0][0], feature[0][1])
return features
def featureMouseClicked(self, event, x, y, flags, param):
if event != cv2.EVENT_LBUTTONDOWN and event != cv2.EVENT_LBUTTONUP:
return
if event == cv2.EVENT_LBUTTONDOWN:
self.rect[0]=x
self.rect[1]=y
if event == cv2.EVENT_LBUTTONUP:
self.rect[2]=x
self.rect[3]=y
self.featureRectSet=True
def addList(self,lis,x,y):
if lis == None:
lis = np.array([[[x,y]]], np.float32)
else:
lis = np.append(lis, [[[x, y]]], axis = 0).astype(np.float32)
return lis
def cut(img,x,y,width,height):
ux=0
uy=0
dx=0
dy=0
if img is None:
return None,dy,uy,dx,ux
img_height, img_width = img.shape[:2]
if y+height/2 > img_height:
uy = img_height
else:
uy = y+height/2
if y-height/2 < 0:
dy = 0
else:
dy = y-height/2
if x+width/2 > img_width:
ux = img_width
else:
ux = x+width/2
if x-width/2 < 0:
dx = 0
else:
dx = x-width/2
if not(dx<ux and dy<uy):
return None,dy,uy,dx,ux
if not(0<=ux<=img_width or 0<=dx<=img_width or 0<=uy<=img_height or 0<=dy<=img_height):
return None,dy,uy,dx,ux
return img[dy:uy,dx:ux],dy,uy,dx,ux
def nextFrame(self):
end_flag, Movieframe = self.Moviecap.read()
#End judgment
if( Movieframe is None):
return None
#Save current frame
self.nowMovieFrame = Movieframe
#Optical flow detection
#Cut out a video with a rectangle with a width and height specified around the position of the previous feature point. Here, the height of the rectangle specified first. It is the same as the width
mask = np.zeros_like(Movieframe)
cutFrame,dy,uy,dx,ux= cut(Movieframe,
int(self.points[len(self.points)-1 - i][0]),#x
int(self.points[len(self.points)-1 - i][1]),#y
2*abs(self.rect[1]-self.rect[3]),2*abs(self.rect[0]-self.rect[2]))#
mask[dy:uy,dx:ux] = cutFrame
self.grayNext = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)
self.featureNext, status, err = cv2.calcOpticalFlowPyrLK(self.grayPrev, self.grayNext, self.featurePrev, None,
(dict( winSize = (15,15),
maxLevel = 2,
criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))))
#Select feature points for which optical flow was detected (0: not detected, 1: detected)
if status != None:
goodPrev = self.featurePrev[status == 1]
goodNext = self.featureNext[status == 1]
for i, (nextPoint, prevPoint) in enumerate(zip(goodNext, goodPrev)):
prev_x, prev_y = prevPoint.ravel()
next_x, next_y = nextPoint.ravel()
if self.featureNext is None:
return 0
#Preparing for the next optical flow
self.grayPrev = self.grayNext.copy()
self.featurePrev = goodNext.reshape(-1, 1, 2)
#Returns 1 on success
return 1
If it moves slowly, optical flow will track it well, but if it moves even a little faster, it will lose sight of the feature points. By the way, there may be an opinion that "If you move fast because you specified the range in Solution 1, the place with the feature point will go out of the range and you will lose sight of the feature point", but the state where you do not specify the range However, the phenomenon of losing sight of the feature points has been seen.
As a countermeasure, there is a way to raise the fps as much as possible when recording a video. However, there are often financially impossible things. This time, if I lost sight of the feature points, I had to stop the video and start over from the range specification. It calls the nextFrame function of the above source code, pauses the playback of the video when 0 is returned, calls featureMouseClicked and extractFeatures, specifies the range again, and plays the video. Perhaps I could do more by generating intermediate frames and increasing fps, but I didn't have the power to implement that much.
opencv 2.4.13 python 2.7.11