I want to perform aggregation processing with spring-batch

Background

In a WEB application made with Spring-Boot I want to implement such batch processing that aggregates the contents of the prepared API execution log file and stores the result in the DB. However, there is no site that can be used as a reference for implementing that requirement, and it is ridiculously clogged, so I will leave it as a memorandum. (I don't know if this is the best)

Main requirements

Prerequisite data (example)

Before: Log format to be aggregated

ʻURL / HTTP method / HTTP status code / execution time / execution date and timeat the time of access are listed in TSV format. Example: / api / ・ ・ ・ GET 200 48 2020/08/14 11:05:42 701 / api / ・ ・ ・ GET 200 27 2020/08/14 11:05:43 352 / api / ・ ・ ・ ・ / 41 DELETE 401 10 2020/08/14 11:05:46 780 / api / ・ ・ ・ / 42 PUT 200 108 2020/08/14 11:06:16 824 / api / ・ ・ ・ POST 500 806 2020/08/14 11:06:30 252 ・ ・ ・ `

After: Format when storing DB

Implementation

policy

point

code

Dto class

@Data
public class LogCollectedDto {
  //API name
  private String apiName;
  //HTTP method
  private String httpMethod;
  //HTTP status code
  private String httpCode;
  //Execution time(ms)
  private String executionTime;
  //Aggregation date and time
  private String collectedDate;
}

Reader

Defined in Bean

  @Bean
  public FlatFileItemReader<LogCollectedDto> reader() {

    final String READ_FILE_PATH = <Log file name to read>;

    FlatFileItemReader<LogCollectedDto> reader = new FlatFileItemReader<>();
    reader.setResource(new FileSystemResource(READ_FILE_PATH));
    reader.setEncoding(StandardCharsets.UTF_8.name());
    reader.setLinesToSkip(0);
    reader.setLineMapper(
        new DefaultLineMapper() {
          {
            setLineTokenizer(
                new DelimitedLineTokenizer(DelimitedLineTokenizer.DELIMITER_TAB) {
                  {
                    setNames(
                        new String[] {
                          "apiUrl", "httpMethod", "httpCode", "executionTime", "collectedDate"
                        });
                  }
                });
            setFieldSetMapper(
                new BeanWrapperFieldSetMapper<LogCollectedDto>() {
                  {
                    setTargetType(LogCollectedDto.class);
                  }
                });
          }
        });
    return reader;
  }

Processor

Cut out to another class

public class CustomItemProcessor implements ItemProcessor<LogCollectedDto, LogCollectedDto> {

  @Override
  public LogCollectedDto process(LogCollectedDto item) throws Exception {

    //Validation of the retrieved Item, skip that line if false
    if (!validateItem(item)) {
      return null;
    }

    //Hold the acquired Item once in another variable for processing later
    //(If you modify the argument directly, the Item acquired when interrupting and restarting may become the processed data)
    LogCollectedDto afterItem = item;
    //Data content processing (separate method omitted)
    afterItem.setApiName(getApiName(・ ・ ・));

    return afterItem;
  }

//(Omitted)

}

Writer

Cut out to another class

@RequiredArgsConstructor
public class CustomItemWriter extends JpaItemWriter<LogCollectedDto> {

  private final JpaItemWriter<Log> jpaItemWriter;

  @Override
  public void write(List<? extends LogCollectedDto> items) {

   //Aggregate the Items received from the Processor and pass it to another Writer instance
    Map<String, List<LogCollectedDto>> groupingMap = groupingItems(items);
    jpaItemWriter.write(collectItems(groupingMap));
  }

  /**
   *Grouping Items received from Processor
   *API name and HTTP status as composite key
   *
   * @param list
   * @return
   */
  private Map<String, List<LogCollectedDto>> groupingItems(List<? extends LogCollectedDto> list) {
    //Create a composite key
    Function<LogCollectedDto, String> compositeKey =
        logDto -> {
          StringBuffer sb = new StringBuffer();
          sb.append(logDto.getApiName()).append("-").append(logDto.getHttpMethod());
          return sb.toString();
        };

    Map<String, List<LogCollectedDto>> grpByComplexKeys =
        list.stream().collect(Collectors.groupingBy(compositeKey));

    return grpByComplexKeys;
  }

  /**
   *Generate a list of Entity by aggregating grouped items
   *
   * @param groupingMap
   * @return
   */
  private List<Log> collectItems(Map<String, List<LogCollectedDto>> groupingMap) {

    List<Log> recordList = new ArrayList<>();

    for (List<LogCollectedDto> dtoList : groupingMap.values()) {
      //Instantiation of Entity class
      Log record = new Log();
      //Aggregation processing
      record.setApiName(dtoList.stream().findFirst().get().getApiName());
      record.setHttpCode(dtoList.stream().findFirst().get().getHttpCode());
      record.setHttpMethod(dtoList.stream().findFirst().get().getHttpMethod());
      record.setAccesses(dtoList.size());
      record.setAverageTime(
          dtoList.stream()
              .collect(
                  Collectors.averagingDouble(dto -> Double.parseDouble(dto.getExecutionTime()))));
      record.setCollectedTime(LocalDateTime.now());
      recordList.add(record);
    }

    return recordList;
  }
}

Concern

Summary

There are concerns, but I personally think it's the cleanest way to write. We are always looking for opinions. Conclusion: Stream API is the strongest! !! !!

Recommended Posts

I want to perform aggregation processing with spring-batch
I want to perform asynchronous processing and periodic execution with Rail !!!
I want to perform Group By processing with Stream (group-by-count, group-by-sum, group-by-max)
[Java] I want to perform distinct with the key in the object
I want to test Action Cable with RSpec test
I want to use java8 forEach with index
I want to play with Firestore from Rails
[Rails] I want to load CSS with webpacker
I want to dark mode with the SWT app
I want to monitor a specific file with WatchService
I want to authenticate users to Rails with Devise + OmniAuth
I want to transition screens with kotlin and java!
I want to get along with Map [Java beginner]
I want to redirect sound from Ubuntu with xrdp
I want to convert characters ...
Dynamically switch the database to connect to
Replace with a value according to the match with a Java regular expression
I want to perform aggregation processing with spring-batch
I tried to increase the processing speed with spiritual engineering
I want to push an app made with Rails 6 to GitHub
I want to perform high-speed prime factorization in Ruby (ABC177E)
I want to make a list with kotlin and java!
I want to make a function with kotlin and java!
Even in Java, I want to output true with a == 1 && a == 2 && a == 3
I want to manually send an authorization email with Devise
I want to distinct the duplicated data with has_many through
I want to implement various functions with kotlin and java!
I want to pass the startup command to postgres with docker-compose.
[Java] I want to test standard input & standard output with JUnit
Swift: I want to chain arrays
I want to use FormObject well
I want to convert InputStream to String
I tried to interact with Java
I want to docker-compose up Next.js!
Manually perform processing equivalent to @ConfigurationProperties
I want to make a button with a line break with link_to [Note]
I want to connect SONY headphones WH-1000XM4 with LDAC on ubuntu 20.04! !!
I want to add a browsing function with ruby on rails
I want to understand the flow of Spring processing request parameters
I want to return to the previous screen with kotlin and java!
I want to INSERT Spring Local Time with MySQL Time (also milliseconds)
I want to avoid OutOfMemory when outputting large files with POI
[Rails] I want to add data to Params when transitioning with link_to
I want to extract between character strings with a regular expression
I want to develop a web application!
I want to write a nice build.gradle
I want to eliminate duplicate error messages
I tried migrating Processing to VS Code
I want to make an ios.android app
I want to display background-ground-image on heroku.
I tried to get started with WebAssembly
I want to select multiple items with a custom layout in Dialog
Even in Java, I want to output true with a == 1 && a == 2 && a == 3 (PowerMockito edition)
[iOS] I tried to make a processing application like Instagram with Swift
I want to RSpec even at Jest!
I want to write a unit test!
I want to install PHP 7.2 on Ubuntu 20.04.
[Note] I want to get in reverse order using afterLast with JdbcTemplate
I want to create a dark web SNS with Jakarta EE 8 with Java 11
I want to mess with Permission of Windows directory from WSL (ubuntu)
I want to display a PDF in Chinese (Korean) with thin reports
I was addicted to the Spring-Batch test
I want to stop Java updates altogether