Real-time display of server-side processing progress in the browser (implementation of progress bar)

Overview

Description

The title may be suspicious and difficult to understand, but in simple terms, it is an implementation of a function that displays the progress of batch processing performed on the server side in real time in the form of a progress bar. As for the method, it seems that there are various methods when I try to google, but among them, there was no implementation method like me this time, so I will post it for the time being.

Process flow

--Browser: ** Parameter input ** (Language: php) --Server: -** Start management script ** (Language: python) --Launch batch processing script and monitoring script --Pass the monitoring result to the progress bar -** Batch script ** (Language: shell) --Batch processing is performed according to the parameters from the browser. -** Monitoring script ** (Language: shell) --Monitor the log file generated by batch processing in real time and output the progress status -** Progress bar generation and display ** (Language: php, javascript) --Get the result of the monitoring script in real time and display it on the progress bar. We will still notify the administrator of the processing result by email.

Implementation

Browser: Parameter input part

This time I needed to enter the ISBN of the product, so enter the ISBN number and send it to the server in the form of post.

convert.php


<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>Convert WAV File!</title>
</head>
<body>
<form action="result.php" method="post">
  <p><label>ISBN (Numeric Only) : <input type="number" name="isbn"></label></p>
  <p><input type="submit" value="Submit"></p>
</form>
</body>
</html>

Server: Management script part

On the server side, it is necessary to perform batch processing according to the parameters obtained from the browser side, and at the same time, run a monitoring script to grasp the situation at the same time. Therefore, two scripts, batch processing and monitoring, will be started in parallel.

Note: When the batch processing is finished, in order to end the monitoring script, the batch processing status is notified by passing the ID of the batch processing process to the monitoring script.

process.py


#! /usr/bin/env python3
# -*- coding: utf-8 -*-
 
import os, sys, subprocess, shlex
from multiprocessing import Pool, Process, Queue

def f(logPID, q, scriptName, isbn):
	q.put([os.getpid()])
	os.system("sh "+scriptName+".sh "+logPID + " " + isbn)

def main():
	jobs = []
	q = Queue()

	isbn = sys.argv[1] 
	
	#Batch processing part
	mainJob = Process(target=f, args=("",q,"test", isbn, ))
	jobs.append(mainJob)
	mainJob.start()

	logPID = str(q.get()[0])
        
	#Monitoring part
	monitorJob = Process(target=f, args=(logPID,q,"monitor", "", ))
	jobs.append(monitorJob)
	monitorJob.start()
		
	for job in jobs:
		job.join()

main()

Server: batch processing

Originally, the purpose of this program is to perform batch processing based on ISBN information, but this time, as a test case, we will replace it with a script that outputs the processing progress as a numerical value at regular time intervals.

test.sh


#!/bin/sh

isbn=$1

touch log.txt
for i in 1 2 3 4 5 6 7 8 9 10; do
   sleep 2
  echo $i >> log.txt
done

Server: Monitoring process

The monitoring script uses the tail command to retrieve and analyze the lines added to the log file in real time. This time, since the progress status is directly output as a number in batch processing, no processing is done, but in reality, since various information is mixed in the log file as well as the progress status, only the progress status is used in the monitoring script. Processing to extract is required.

Note: By adding the batch processing process ID to the tail command, it is possible to terminate the monitoring script when the batch processing is completed.

monitor.sh


#!/bin/sh

pID=$1 #Batch processing process ID
tail -f -n 1 --pid=$pID log.txt | while read line
do 
  if [ -n "$line" ] ; then 
     echo $line
  fi
done

Server: Progress bar display

The monitoring result is acquired in real time and displayed on the progress bar. Finally, we will notify the administrator by e-mail that it has been processed.

Note: Use the `` `popen``` command to get the python processing result in real time in php.

result.php


<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html lang="en">
<head>
    <title>Progress Bar</title>
</head>
<body>
<!-- Progress bar holder -->
<div id="progress" style="width:500px;border:1px solid #ccc;"></div>
<!-- Progress information -->
<div id="information" style="width"></div>
<?php
$isbn = $_POST['isbn']; 
echo "ISBN: ".$isbn;
// Total processes
$total = 10;
// Loop through process
$cmd = "python /var/www/html/service/progressbar/process.py ".$isbn;
$proc = popen($cmd, 'r');
//rea one line of the last oparation, and do nothing
$i = (int) fread($proc, 4096);
while (!feof($proc)) {
    // Calculate the percentation
	$i = (int) fread($proc, 4096);
    $percent = intval($i/$total * 100)."%";    
	// Javascript for updating the progress bar and information
    echo '<script language="javascript">
    document.getElementById("progress").innerHTML="<div style=\"width:'.$percent.';background-color:#ddd;\">&nbsp;</div>";
    document.getElementById("information").innerHTML="'.$percent.' processed.";
    </script>';
	// Send output to browser immediately
    flush();
	// Sleep one second so we can see the delay
    sleep(1);
	
	if($percent=="100%")
		break;
}
// Tell user that the process is completed
    echo '<script language="javascript">
    document.getElementById("information").innerHTML="Process completed!";
    </script>';

//send mail when the process is over
    $to = "[email protected]";
    $subject = "Batch processed";
    $message = "Admin,\n Item number: 「".$isbn."It has been processed.\n Thank you.";
    $headers = 'From: [email protected]' . "\r\n";
    if(mail($to, $subject, $message, $headers))
        echo "Notification email has been sent!";
    else
        echo "The notification email could not be sent.\n Please inform Admin directly of the item number."
?>
</body>
</html>

Remarks

The source code of this program can also be obtained from the link below. Implementation of progress bar

Recommended Posts

Real-time display of server-side processing progress in the browser (implementation of progress bar)
Make progress of dd visible in the progress bar
How to display the progress bar (tqdm)
Django ~ Let's display it in the browser ~
Display Python 3 in the browser with MAMP
I want to display the progress bar
View the result of geometry processing in Python
The one that displays the progress bar in Python
I want to display the progress in Python!
Use the progress bar with Click: (Easy to use, improve the display of tasks that take more than 24 hours, notes when using in parallel processing)
Read the csv file and display it in the browser
About testing in the implementation of machine learning models
What to do if the progress bar is not displayed in tqdm of python
A reminder about the implementation of recommendations in Python
How to display the regional mesh of the official statistics window (eStat) in a web browser
100 language processing knock-42: Display of the phrase of the person concerned and the person concerned
Display the timetable of the Morioka bus location system in Pythonista
The image display function of iTerm is convenient for image processing.
Implementation of quicksort in Python
[Cloudian # 9] Try to display the metadata of the object in Python (boto3)
A function that measures the processing time of a method in python
for, continue, break Explain the flow of iterative processing in Python3-Part 1
Python in the browser: Brython's recommendation
The story of participating in AtCoder
Implementation of login function in Django
Progress bar in pop-up in Python Kivy
Paper: Music processing in the brain
Implementation of life game in Python
Waveform display of audio in Python
The story of the "hole" in the file
The meaning of ".object" in Django
Implementation of original sorting in Python
Summarize the findings of reading Go's HTTP implementation ~ Implement FIFO in 5 ways ~
Explanation and implementation of the XMPP protocol used in Slack, HipChat, and IRC
[Android] Display images on the web in the info Window of Google Map
Receive a list of the results of parallel processing in Python with starmap
Display the status of COVID 19 infection in Japan with Splunk (GitHub version)
Trial to judge the timing of the progress display of the for loop using 0b1111 ~
I tried to display the altitude value of DTM in a graph