I made a Docker image of SDAPS for Japanese

Introduction

SDAPS is OMR (optical mark recognition) software that creates and aggregates questionnaires for questionnaires using general copy paper and document scanners that do not require special mark sheet paper.

It is not suitable for applications such as entrance exams that require strict results, but it has advantages such as no need to fill (this can be changed to fill with checkmode = fill) and no need to prepare special paper. ..

Since SDAPS does not support Japanese, I created a questionnaire containing Japanese text and created a Docker image that executes a modified sdaps command so that it can output a report.

The created Docker image is published on Docker Hub, and the Docker file and changed parts are published on GitHub.

Deliverables

The changed parts are published on the following Github and Docker Hub.

• https://github.com/YasuhiroABE/docker-sdaps-ja
• https://hub.docker.com/repository/docker/yasuhiroabe/sdaps-ja

Reference material

Here is a list of URLs that I referred to when writing this article.

• https://sdaps.org/
• http://fabiorehm.com/blog/2014/09/11/running-gui-apps-with-docker/
• https://qiita.com/YasuhiroABE/items/75639f4b95731171b894

Development environment

For development, build and push Docker images are performed in the following environment. The created Docker container is also based on Ubuntu 20.04 for this development environment.

• Host OS
• Ubuntu 20.04.1 on VMware Workstation 16 Pro
• Docker
  • 19.03.12 (docker-ce 5:19.03.12~3-0~ubuntu-bionic DEB package)
• SDAPS
  • 1.9.7 (sdaps 1.9.7-0.2ubuntu2 DEB package)

testing environment

We are testing the ability to create questionnaires, read TIFF images, and modify them in the following environment.

• Linux Distribution
• Xubuntu Desktop 20.04.1 amd64 version
• Windows10 (2004) 64bit version
• WSL2 + Docker Desktop for Windows (19.03.12)
• Windows Terminal v1.3.2651.0
• macOS (10.15.6)
  • Docker Desktop for Mac (19.03.12)
  • iTerm2 v3.3.12

The following equipment is used to print and read the questionnaire.

• Laser printer (OKI B841)
• General PPC paper (white plain paper)
• Print quality-High definition (1200x1200)
• Document Scanner (Brother ADS-3600W)
• Settings-TIFF multi-page, 300dpi, monochrome, long edge binding

About the use of a copy machine

So far I haven't used a copy machine to prepare the questionnaire. In the past, the default print quality was low, so we have experienced problems, but in recent years, we have been able to handle it without problems even when we temporarily increase the number of sheets using a copier at the destination where we requested distribution. However, since print quality strongly depends on the settings of the copier, it is recommended to perform sufficient tests in advance and save the parameters to ensure reproducibility so that the output can be obtained with the highest possible definition. ..

How to use

There may be some bugs derived from sdaps and an error message may appear, but I will introduce how to use the command including them.

When executing the command, use the volume function of Docker to create a working directory where the TeX file of the answer sheet and the project directory are placed, and be sure to mount it on / proj in the container.

• Working directory in Docker content: / proj

Workflow

Roughly, they are used in the following order.

1. Create a working directory and move it (e.g. dev.proj1 /; cd dev.proj1)
2. Create a TeX file for the questionnaire (e.g. example-ja.tex)
3. Execute the setup command to create the project directory
4. Scan the completed questionnaire and place the TIFF file in the working directory (e.g. 01.tif, 02.tif, ...)
5. Repeat the add command to register the TIFF file
6. Execute the recognize command to extract the data
7. Create a report file with the report_tex command (e.g. report_1.pdf)
8. Extract raw data with csv command (e.g. data_1.csv)

preparation work

Pull the Docker image to be used before executing the command.

Execution example of pull

$ sudo docker pull yasuhiroabe/sdaps-ja:ub2004-2
ub2004-2: Pulling from yasuhiroabe/sdaps-ja
e6ca3592b144: Pull complete
...
Digest: sha256:26458bd39267149a3ccebc36f60a3082e0f519545596166accede1eac4eed8f7
Status: Downloaded newer image for yasuhiroabe/sdaps-ja:ub2004-2
docker.io/yasuhiroabe/sdaps-ja:ub2004-2

About the working directory

Although it is not registered on GitHub, dev.proj1 /, dev.proj2 / work directories are created for work, and sample answer sheet TeX files and project directories are placed.

For reference, I will post the final state of the dev.proj2 directory.

Output result of tree command

$ tree dev.proj2
dev.proj2
├── 01.tif
├── 02.tif
├── 20200310_survey
│   ├── 1.tif
│   ├── 2.tif
│   ├── data_1.csv
│   ├── data_2.csv
...
│   ├── questionnaire.pdf
...
│   ├── report_1.pdf
│   ├── report_2.pdf
...
│   ├── translator-sdaps-dictionary-English.dict
...
└── 20200310_survey.tex

1 directory, 43 files

Sample Japanese questionnaire

The files used for the test are listed. Please refer to the original as some comments and contents have been deleted.

example.Example of embedding Japanese based on tex

\documentclass[
  english,
  a4paper, 
  checkmode=check,
  pagemark,
  stamp]{sdapsclassic}
\usepackage[utf8]{inputenc}
% For demonstration purposes
\usepackage{multicol}

\usepackage{xltxtra}
\setmainfont{IPAPMincho}
\setsansfont{IPAPGothic}
\setmonofont{IPAGothic}
\XeTeXlinebreaklocale "ja"

\author{author}
\title{title}

\begin{document}
  \begin{questionnaire}
    \begin{info}
      \texttt{info}You can add any text by using the environment.
    \end{info}

    \addinfo{Date}{10.03.2013}

    \section{5 grade evaluation}
    \singlemark{How often do you use SDAPS??}{rare}{every day}

  \end{questionnaire}
\end{document}

It is possible to replace some of the predefined fixed phrases.

translator-sdaps-dictionary-English.dict example

\ProvidesDictionary{translator-sdaps-dictionary}{English}

\providetranslation{infotext}{How to fill in/Instructions to fill out the form.}
\providetranslation{standard-deviation}{Standard-Deviation}
\providetranslation{info-cross}{Mark example/Check}
\providetranslation{info-correct}{Modification example/Uncheck to correct}
\providetranslation{answers}{Answers}
\providetranslation{questionnaireid}{Questionnaire-ID:}
\providetranslation{surveyid}{Survey-ID:}
\providetranslation{draft}{draft}
\providetranslation{info-mark}{For questions with a range (1-5) choose the answer the mark that fits best.}
\providetranslation{info-select}{For things like satisfaction, mark one on a scale of five.}
\providetranslation{mean}{Mean}

Here's how example-ja.tex and translator-sdaps-dictionary-English.dict are placed in the working directory (dev.proj1).

python

$ tree dev.proj1/
dev.proj1/
├── example-ja.tex
└── translator-sdaps-dictionary-English.dict

0 directories, 2 files

Follow the steps below to generate a questionnaire.

python

$ sudo docker run --rm -v `pwd`/dev.proj1:/proj \
  --name sdaps-ja \
  yasuhiroabe/sdaps-ja:ub2004-2 \
  setup --add translator-sdaps-dictionary-English.dict \
  work/ example-ja.tex

The questionnaire is placed at dev.proj1/work/questionnaire.pdf in this example.

[Supplement] How to get the original translator-sdaps-dictionary-English.dict

If you can install it from a package on Ubuntu etc., you can copy it from / usr / share / sdaps / tex /, but if that is not possible, make a note of how to extract it from the Docker image.

Since "docker cp" cannot be used to extract files from images, it is a little troublesome, but you can extract the original file by the following method.

Copy files inside the container to your local file system

##Execute the sdaps command("-rm"No options)
$ docker run yasuhiroabe/sdaps-ja:latest

##Check the container ID
$ docker ps -a |grep sdaps-ja
a9e5dcb2a026        yasuhiroabe/sdaps-ja:latest   "/run.sh --help"         17 seconds ago      Exited (0) 16 s
econds ago                       eloquent_haibt

##Copy the required files using the container ID
$ docker cp a9e5dcb2a026:/usr/share/sdaps/tex/translator-sdaps-dictionary-English.dict my-translator.dict

setup command

After creating the TeX file of the answer sheet, let the sdaps command create the project directory. Creating a directory with the project name in advance will result in an error.

setup execution example

$ make proj1-setup
sudo docker run --rm \
        -v `pwd`/dev.proj1:/proj \
        --name sdaps-ja \
        yasuhiroabe/sdaps-ja:ub2004-2 \
        setup \
        --add translator-sdaps-dictionary-English.dict \
        work/ example-ja.tex

Command execution message

This is XeTeX, Version 3.14159265-2.6-0.999991 (TeX Live 2019/Debian) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
This is XeTeX, Version 3.14159265-2.6-0.999991 (TeX Live 2019/Debian) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
...
Running xelatex now multiple times to generate the questionnaire.
Running xelatex now multiple imes to generate the questionnaire.
title
Author:author
Date: 10.03.2013

Questionnaire.pdf will be placed in the specified project directory (dev.proj1 / work /).

In this example, translator-sdaps-dictionary-English.dict is specified. (Optional) This file will be copied if you create the project directory, but if you want to get it in advance, you can do it in the following ways.

add command

Scan the questionnaire, place a TIFF multi-page file (01.tif in this example) in your project directory, then run add.

add execution example

$ make proj1-add
sudo docker run --rm \
        -v `pwd`/sdaps.proj:/proj \
        yasuhiroabe/sdaps-ja:ub2004-2 add work 01.tiff

Command execution message

Processing 01.tiff
Done

The equipment you are using can only store 25 pages of answers in one file, so if you exceed this, execute the add command multiple times.

recognize command

When the registration of the answer sheet is completed, it is the phase to analyze the registered sheet.

recognize

$ make proj1-recognize 
sudo docker run --rm \
        -v `pwd`/dev.proj1:/proj \
        yasuhiroabe/sdaps-ja:ub2004-2 \
        recognize work

Command execution message

Connection Error (Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory)
Connection Null
...
Warning: 1.tif, 1: Matrix not recognized.
Warning: No page number for page 1.tif, 1 exists.
Warning: Got a wrong survey ID (1.tif, 1)! It is None, but should be 2449560734.
Warning: 1.tif, 3: Matrix not recognized.
Warning: No page number for page 1.tif, 3 exists.
Warning: Got a wrong survey ID (1.tif, 3)! It is None, but should be 2449560734.
4 sheets
Processed 4 of 4 sheets, took 1.852894 seconds
‘‘‘

## report_tex command

It is used to extract the report PDF file based on the analysis result.


#### **`report_tex execution example`**
```bash

$ make proj1-reporttex 
    sudo docker run --rm \
    -v `pwd`/dev.proj1:/proj \
    yasuhiroabe/sdaps-ja:ub2004-2 \
    report_tex work/ 

Command execution message

Unable to init server: Could not connect: Connection refused
Unable to init server: Could not connect: Connection refused
This is XeTeX, Version 3.14159265-2.6-0.999991 (TeX Live 2019/Debian) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
...
Running xelatex now multiple times to generate the report.

When the process is finished, the project directory(dev.proj1/work/)In the report_.A pdf file has been generated. report_In the tex command option,--create-tex*There is an option and there is a function to extract TeX files, but this is/Since the progress is output to tmp, if you want to use it, select an appropriate directory./Mount it in tmp. This example is in the GNU makefile, proj2-Described in the reporttex task.

report_1.The result is output to pdf as follows."Answers:", "Mean:", "Standard-Deviation:"If you want to translate such things into Japanese, translator-sdaps-dictionary-English.Please change the dict.

##csv command

It is used when you want to extract raw data in CSV format. Used to verify that the content of the report is correct. When calculating deviations in Excel, slightly different values may be displayed due to differences in error processing methods.

csv execution example

$ make proj1-csv 
sudo docker run --rm \
   -v `pwd`/dev.proj1:/proj \
   yasuhiroabe/sdaps-ja:ub2004-2 \
   csv export work/ 

The generated CSV file is the data in the project directory._*.It is output to csv.

text:dev.proj1/work/data_2.Contents of csv

questionnaire_id,global_id,empty,valid,recognized,review,verified,1_review,1_1_review,1_1
None,None,0,1,1,,0,,,3
None,None,1,0,1,,0,,,-1
None,None,0,1,1,,0,,,4
None,None,1,0,1,,0,,,-1

That is all for the main usage examples.

#Problems encountered while working

Make a note of the last problem you encountered while working. please come if you have interest.

##I get an error when I run the sdaps report

Only when executed from inside the container"Unable to init server: Could not connect: Connection refused"Message is displayed.

Command line when an error occurs

## In dev.proj2 / in advance, the TeX file of the questionnaire and the 20200310_survey / directory with sdaps added the answers are placed.
$ sudo docker run --rm -v `pwd`/dev.proj2:/proj \
        --name sdaps-ja \
        yasuhiroabe/sdaps-ja:ub2004-2 report 20200310_survey/

I didn't know the timing of the error, so I created a startup script and observed the process with strace.

Debug run.sh

!/bin/bash

cd "${WORKING_DIR}"
strace sdaps "$@"

A message is output there

python

connect(6, {sa_family=AF_UNIX, sun_path=@"/home/sdaps/.cache/broadway1.socket"}, 38) = -1 ECONNREFUSED (Connection refused)

Apparently, the library that is loading somehow processing the image behind the scenes seems to be failing to connect to the X server during the initialization process. Even in a local environment where sdaps can be executed properly, if you point the DISPLAY environment variable to the person who is not running, it will fail.

Error reproduction

$ env DISPLAY="" sdaps report 20200310_survey
------------------------------------------------------------------------------
- SDAPS -- report
------------------------------------------------------------------------------
Unable to init server: Could not connect: Connection refused
Unable to init server: Could not connect: Connection refused

For this reason, when you docker run the DISPLAY environment variable--I decided to add it with the env option. Ubuntu 20.Since 04's X server has only unix domain socket open(Port 6000 is closed)、/tmp/.X11-I am mounting unix. This area is based on the output of lsof and the link in the reference information.

Even so, the error around DBus still remains, but I will pass it through for the time being.

Next error around DBus

$ make proj2-report
sudo docker run --rm \
        -e DISPLAY=:0.0 \
        -v /tmp/.X11-unix:/tmp/.X11-unix \
        -v `pwd`/dev.proj2:/proj \
        --name sdaps-ja \
                sdaps-ja \
        report 20200310_survey/  

(sdaps:1): dbind-WARNING **: 13:41:40.375: Couldn't connect to accessibility bus: Failed to connect to socket 
/tmp/dbus-5MKrJQOOU3: Connection refused

Even if you do not connect to the X server, the process ends normally with an error, so these variables are not set during actual use.