Automatic minimum testing in web development

Introduction

On New Year's Day, if I'm too lazy, I can't return to society, so I tried programming. It seems that neither the client nor the partner who is requesting development is very good at testing in the Web development project that has been PM since last year, so if I keep silent, it will not work at all! It's an atmosphere that seems to be able to deliver something like this. However, that is not possible, and I tell my partner to do more tests properly, but I do not want to take time (money), so he has a program that automatically performs the tests that should be cleared at least. BeautifulSoup/requests) I wrote it. It's not so much an automated test, but it's a level feeling.

Click here for the finished product ▶ kurab/simple_web_inspector

I don't want to explain it after testing and distributing it casually, so I decided not to use DB or create a class and divide it into files this time.

At least the items you want to test

This time, I decided to mainly check broken links, temporary links, etc. because it is the minimum. In particular,

I want you to check this as a matter of course, but you can't do it manually. I think it's usually done automatically, but apparently it's not normal. I hope that this will be cleared as a matter of course and the minimum level will rise.

First make a URL list

I can't create a URL list manually, so I'll just grab it. The policy is to manually add pages such as LP that cannot be traced from within the site.

internal_url_collector.py

What you are doing is get all the hrefs for the a tag in the page and

Formatted to full path and output excluding. Do the same for the resulting URLs and add non-duplicate URLs. Repeat.

In addition, the protocol is assumed to be unified with https, and redirected ones are not followed up to the next page.

Just that. I thought it would be good to repeat it for the hierarchy of the site, but if there is pagenation etc., it will be detected a lot. Adjust appropriately while looking at the results.

It's not good to use two temporary files, and I'd like to consider a smarter method if I have time.

Inspect based on URL list

simple_web_inspector.py

I am inspecting what I want to test at a minimum as mentioned above. This time, the output of the result is Markdown. When I tried it, I thought that it was not good enough, or I thought that I needed a little more ingenuity, but the reason why I chose Markdown is that I always think that it is better to manage the version of these comprehensive test results. I wanted it to be a text file, not an excel spreadsheet, not a csv. It's hard to read if it's just text, so I chose Markdown.

As an image of use, when the develop branch of git is updated, hook it and run a test. An image in which tests and other related documents are saved according to the source code history. I dream of such a thing.

What is Markdown? You can output it to PDF and give it to the other party.

Report sample

The page-by-page report looks like this.

Basic Information

item value
url https://www.yourdomain.com/dir/
httpStatus 200
title Home Page
description (not set)
keywords test, automation, web

Link Analysis

Invalid item count
Empty Link('' or no href attr) 2
Temporary Link suspected(#) 10
Dead Link(404) 3
Redirected Link(301, 302) 0

Dead Link List (404 Not Found)

Media Analysis

type total dead moved
css 4 0 0
js 25 0 0
font 5 0 0
favicon 1 0 0
img 65 0 0

in conclusion

If there is such a thing, it may be possible to propose improvements to the client (such as compiling because there are too many js in the above sample), it may be said that it is dangerous in the first place, and for development partners, it is like this Are you going to deliver things? It can be said that, above all, a product with few bugs can be made, and it is full of good things.

In 2021, I hope that many new coronaviruses and bugs will disappear from the world. happy New Year.

Recommended Posts

Automatic minimum testing in web development
Web application development memo in python
Web application development in Go language_Hands-on day 1
Framework development in Python
Development environment in Python
Slackbot development in Python
WEB application development using django-Development 1-
Run automatic jobs in python
Web stamping automation in Chrome
Seven trends in software development in 2020
Web application development with Flask
Trade-offs in web scraping & crawling
Python development in Visual Studio 2017
Participate in nl command development
Minimum spanning tree in C #
Python development in Visual Studio
[Pytest] [mock] Web development beginners summarized unit test and mock in python.