Check squat forms with deep learning

** The video used in this article was recorded a long time ago and is currently refraining from muscle training at the gym. ** **

Introduction

This is the second "Muscle Training x Deep-Learning". The first is here (Deep learning made it dramatically easier to see the time-lapse of physical changes). "Form check" is a habit of many trainees (people who love muscle training). By fixing your smartphone during training, taking a selfie, and looking back at it later, you will be able to know your own form habits and improve the quality of subsequent training! In this article, I focused on "squat (king of muscle training)" among muscle training and wrote a story of checking the form using deep learning.

First from the result

ezgif.com-optimize.gif

The source code is available in the Google Colab notebook. https://colab.research.google.com/drive/18bFHGZ415T6emBoPHacrMyEckPGW9VCv You can also run it using your own video, so please give it a try.

table of contents

1. Posture estimation

Posture estimation is a technology that estimates the posture of the human body and animals. For example, in the case of the human body, the posture can be expressed by detecting the neck, buttocks, knees, etc. and connecting them. A well-known one is OpenPose.

https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/media/dance_foot.gif?raw=true

1-1. Try to move

This time, I referred to "[Learn while making! Deep learning by PyTorch](https://www.amazon.co.jp/%E3%81%A4%E3%81%8F%E3%82%8A" % E3% 81% AA% E3% 81% 8C% E3% 82% 89% E5% AD% A6% E3% 81% B6% EF% BC% 81PyTorch% E3% 81% AB% E3% 82% 88% E3 % 82% 8B% E7% 99% BA% E5% B1% 95% E3% 83% 87% E3% 82% A3% E3% 83% BC% E3% 83% 97% E3% 83% A9% E3% 83 % BC% E3% 83% 8B% E3% 83% B3% E3% 82% B0-% E5% B0% 8F% E5% B7% 9D-% E9% 9B% 84% E5% A4% AA% E9% 83 % 8E-ebook / dp / B07VPDVNKW /) "GitHub Page. I wanted to implement it with PyTorch x Google Colab, so the above implementation was very helpful.

For the time being, I will try to estimate the posture from a single image using free materials. The source is a schematic version. See the Google Colab notebook for more information.


def create_model(weight_path):
  """
Create a model.
The trained model and OpenPoseNet have different network layer names
  (For example, module.model0.0.weight and model0.model.0.weight)So
Correspond and load
  """

  #Model definition
  model = OpenPoseNet()

  #Load learned parameters
  net_weights = torch.load(
      weights_path, map_location={'cuda:0': 'cpu'})
  keys = list(net_weights.keys())

  weights_load = {}

  #The loaded contents can be displayed on OpenPoseNet.
  #Parameter name model.state_dict().keys()Copy to
  for i in range(len(keys)):
    weights_load[list(model.state_dict().keys())[i]
                 ] = net_weights[list(keys)[i]]

  #Give the model what you copied
  state = model.state_dict()
  state.update(weights_load)
  model.load_state_dict(state)
  return model

model = create_model(weights_path)
model.to(device)

model.eval()
with torch.no_grad():
  predicted_outputs, _ = model(img.to(device))
  pafs = predicted_outputs[0][0].cpu().detach().numpy().transpose(1, 2, 0)
  heatmaps = predicted_outputs[1][0].cpu().detach().numpy().transpose(1, 2, 0)

pafs = cv2.resize(
    pafs, (test_img.shape[1], test_img.shape[0]), interpolation=cv2.INTER_CUBIC)
heatmaps = cv2.resize(
    heatmaps, (test_img.shape[1], test_img.shape[0]), interpolation=cv2.INTER_CUBIC)

_, result_img, _, _ = decode_pose(test_img, heatmaps, pafs)

cv2_imshow(result_img)

The results are as follows. image.png

It feels pretty good. If you do this for each frame of the video, you will be able to check the form. However, I don't want all of these joints, only the minimum necessary parts. This time,

I will focus on just that.

1-2. Heat map

In the model to be used, the posture is estimated by outputting the heat map and extracting each part from it. First, let's draw a heat map of the target part.

# 1:Neck, 8:Butt (right), 9:Knee (right), 10:Ankle (right)
necessary_parts=[1,8,9,10] 
fig, ax = plt.subplots(2, 2, figsize=(16, 10))

for i, part in enumerate(necessary_parts):
    heat_map = heatmaps[:, :, part]
    heat_map = np.uint8(cm.jet(heat_map)*255)
    heat_map = cv2.cvtColor(heat_map, cv2.COLOR_RGBA2RGB)
    blend_img = cv2.addWeighted(test_img, 0.5, heat_map, 0.5, 0)
    ax[int(i/2), i%2].imshow(blend_img)

plt.show()

The results are as follows. I have been able to extract a specific part properly.

image.png

1-3. Posture estimation

Identify the joint position from the output heat map of the specific part. This time, on the premise that only one person can be seen on one sheet, only one point is left for each part, and the specified parts are connected.

def find_joint_coords(heatmaps, necessary_parts, param = {'thre1': 0.1, 'thre2': 0.05, 'thre3': 0.5}):
  """
Detect joint coordinates
  """
  
  joints = []
  for part in necessary_parts:
      heat_map = heatmaps[:, :, part]
      peaks = find_peaks(param, heat_map)
      if len(peaks) == 0:
        joints.append(img, [np.nan, np.nan], [np.nan, np.nan])
      #If there are two or more peaks, leave only the strongest location
      if peaks.shape[0]>1:
        max_peak = None
        for peak in peaks: 
          val = heat_map[peak[1], peak[0]]
          if max_peak is None or heat_map[max_peak[1], max_peak[0]] < val:
            max_peak = peak
      else:
        max_peak = peaks[0]
      joints.append(max_peak)

  return joints

img = test_img.copy()
joints = find_joint_coords(heatmaps, necessary_parts)
for i in range(len(joints)-1):
  img = cv2.line(img,tuple(joints[i].astype(int)),tuple(joints[i+1].astype(int)),(255,0,0),3)

cv2_imshow(img)

The results are as follows. feel well.

image.png

2. Application to squat form check

Now that I know it's going to work, I'll apply it to my squat video.

2-1. Important factors in squats

The skeleton that forms the shin, thigh, and trunk varies from person to person, so the optimal form varies from person to person. I think it will be easier to imagine in the figure below.

image.png

In other words, the form that suits the person is more due to skeletal proportions than flexibility and strength balance. Therefore, if you can grasp this skeletal proportion by yourself, it will help prevent injuries and you can safely increase the weight of the squat. For more information, check out YouTube and a number of other great web pages below.

IMAGE ALT TEXT HERE Squats Part 1: Fold-Ability and Proportions

2-2. Make it look like a form check

Since the coordinates of the joints can be taken, find the joint angle and moment arm and arrange them in chronological order. The angle of the joint is the angle of the two vectors that sandwich the joint. The moment arm simulates the ideal center of gravity (midfoot) and uses it as the distance of the perpendicular line passing through the joint with respect to the straight line in the vertical direction. Calculate this for each frame and connect them to complete. Finally, plot the angle and moment arm produced in each frame with matplotlib, paste it at the top of the screen, and add a graph that looks like it is being analyzed nicely.

** * The source code will be long, so I will omit it. Please refer to the notebook. ** **

The completed example is as follows (the frame is selected appropriately).

image.png

Summary

We applied posture estimation using deep learning and performed a squat form check. I think we were able to detect the joints with high accuracy. From this result, you may be able to further improve the quality of squats by finding a player with a skeleton close to you and comparing the player with the form.

The challenges are as follows.

As mentioned above, muscle training that you can do at home is not just about moving your body! Have a fun muscle training hack life!

Recommended Posts

Check squat forms with deep learning
Try deep learning with TensorFlow
Deep Kernel Learning with Pyro
Generate Pokemon with Deep Learning
Try Deep Learning with FPGA-Select Cucumbers
Cat breed identification with deep learning
Make ASCII art with deep learning
Solve three-dimensional PDEs with deep learning.
Categorize news articles with deep learning
Forecasting Snack Sales with Deep Learning
Deep Learning
Make people smile with Deep Learning
Classify anime faces with deep learning with Chainer
Try Bitcoin Price Forecasting with Deep Learning
Try with Chainer Deep Q Learning --Launch
Try deep learning of genomics with Kipoi
Deep Learning Memorandum
Python Deep Learning
Deep learning × Python
99.78% accuracy with deep learning by recognizing handwritten hiragana
First Deep Learning ~ Struggle ~
Learning Python with ChemTHEATER 03
"Object-oriented" learning with python
A story about predicting exchange rates with Deep Learning
Learning Python with ChemTHEATER 05-1
Python: Deep Learning Practices
Deep learning / activation functions
Deep Learning from scratch
Deep learning image analysis starting with Kaggle and Keras
Deep learning 1 Practice of deep learning
Deep learning / cross entropy
First Deep Learning ~ Preparation ~
First Deep Learning ~ Solution ~
Learning Python with ChemTHEATER 02
I tried deep learning
Extract music features with Deep Learning and predict tags
Domain check with Python
Deep learning large-scale technology
Check version with python
Deep learning / softmax function
[Evangelion] Try to automatically generate Asuka-like lines with Deep Learning
Create an environment for "Deep Learning from scratch" with Docker
(Now) Build a GPU Deep Learning environment with GeForce GTX 960
[Deep learning] Image classification with convolutional neural network [DW day 4]
I captured the Touhou Project with Deep Learning ... I wanted to.
Deep Learning with Shogi AI on Mac and Google Colab
I tried to divide with a deep learning language model
HIKAKIN and Max Murai with live game video and deep learning
Easy deep learning web app with NNC and Python + Flask
Sine curve estimation with self-made deep learning module (python) + LSTM
Machine learning learned with Pokemon
Deep Learning from scratch 1-3 chapters
Play with reinforcement learning with MuZero
Deep Learning Gaiden ~ GPU Programming ~
Ensemble learning summary! !! (With implementation)
Check python coverage with pytest-cov
Deep learning image recognition 1 theory
Deep running 2 Tuning of deep learning
Reinforcement learning starting with Python
About learning with google colab
Deep learning / LSTM scratch code