[JAVA] When I regained my mind about Tribuo released by Oracle, the person inside was a hot person.

Preface

Since it is updated in real time, I have summarized the contents of the tutorial again to organize the information. This article was written based on the contents of here.

About materials

See here for the original story. If you need Japanese, please see here. Click here to download iris data. See here for the source code. See here for JavaDoc.

change point

The CSV data loading bug has been fixed. v4.0.1 *

Fixed an issue where CSVReader could not read files with extra line breaks at the end. Added IDXDataSource to read IDX (ie MNIST) format datasets. Updated the configuration tutorial to read MNIST from IDX files instead of libsvm files.

In the story of the person in the middle, he said I overlooked the v4.0.0 written in the document, I fixed it.

Because I'm aware of it, the third line of the code below that appears in the introduction requires a model as the first argument.

1. var trainSet = new MutableDataset<>(new LibSVMDataSource(Paths.get("train-data"),new LabelFactory()));
2. var model    = new LogisticRegressionTrainer().train(trainSet);
3. var eval     = new LabelEvaluator().evaluate(new LibSVMDataSource(Paths.get("test-data"),trainSet.getOutputFactory()));

var eval = new LabelEvaluator()
                .evaluate(model, new LibSVMDataSource(Paths.get("test-data"), trainSet.getOutputFactory()));

Postscript 2020/09/30 I got a reply saying that I fixed it and committed it.

Other material

That, This is [on git Issues](https: / /github.com/oracle/tribuo/issues/47) I tried to give it, but it was answered seriously. I intended to write a story, but thank you for taking the time to deal with me seriously.

According to the person in the middle, Tribuo's official page has already been fixed. That is. Null check and javadoc will be fixed this week. Correspondence is quick.

People in this Tensorflow java committer. The person himself said.

For GPU-based computations on FP16, the JVM already supports aarch64, so the Java APIs for Tensorflow Java and ONNX Runtime should support it soon, and Tribuo will follow them too. ](Https://github.com/Craigacp). Said. Even now, he said that it can be done using Tensorflow Java or ONNX Runtime.

~~ These people have a great amount of heat. I want to do this in Japan. OpenAI is not open at all. ~~

Setup (Getting Started)

I have changed the version since the last time.

In maven, set as follows.

<dependency>
    <groupId>org.tribuo</groupId>
    <artifactId>tribuo-all</artifactId>
    <version>4.0.1</version>
    <type>pom</type>
</dependency>

In gradle, the jar cannot be downloaded successfully by the method described in the document. Only the pom file will be downloaded. It is necessary to set as follows.

    api ('org.tribuo:tribuo-all:4.0.1@pom') {
        transitive = true
    }

It seems that it works differently with kotlin. I'll add a tab to the document and write it, but I'm angry that I don't want to bother to create a tab for gradle and write it twice in Groovy and kotlin.

Get iris data.

Download destination of iris data

https://archive.ics.uci.edu/ml/machine-learning-databases/iris/bezdekIris.data

Create a class with a normal main method and implement it as Getting Started.

SampleTribuo

package org.project.eden.adam;

import java.io.IOException;
import java.nio.file.Paths;

import org.tribuo.DataSource;
import org.tribuo.Model;
import org.tribuo.MutableDataset;
import org.tribuo.Prediction;
import org.tribuo.classification.Label;
import org.tribuo.classification.LabelFactory;
import org.tribuo.classification.dtree.CARTClassificationTrainer;
import org.tribuo.classification.evaluation.LabelEvaluation;
import org.tribuo.classification.evaluation.LabelEvaluator;
import org.tribuo.classification.sgd.linear.LogisticRegressionTrainer;
import org.tribuo.data.csv.CSVLoader;
import org.tribuo.evaluation.TrainTestSplitter;

/**
 * @author jashika
 *
 */
public class SampleTribuo {

    /**
     * @param args main method arguments.
     * @throws IOException Throws on file read error.
     */
    public static void main(String[] args) throws IOException {

        //Read labeled iris data
        var irisHeaders = new String[] { "sepalLength", "sepalWidth", "petalLength", "petalWidth", "species" };
        DataSource<Label> irisData = new CSVLoader<>(new LabelFactory()).loadDataSource(Paths.get("bezdekIris.data"),
                irisHeaders[4], irisHeaders);
        //* Since the path of the data to be read is embedded, please match it to your own environment.

        //70 read iris data for training%, 30 for testing%Divided into
        var splitIrisData = new TrainTestSplitter<>(irisData, 0.7, 1L);
        var trainData = new MutableDataset<>(splitIrisData.getTrain());
        var testData = new MutableDataset<>(splitIrisData.getTest());

        //Decision tree learning can be used
        var cartTrainer = new CARTClassificationTrainer();
        Model<Label> tree = cartTrainer.train(trainData);

        //You can also use logistic regression
        var linearTrainer = new LogisticRegressionTrainer();
        Model<Label> linear = linearTrainer.train(trainData);

        //Ultimately, make predictions from unknown data
        //Forecasts are the output name (ie label) and the score/It becomes a probability.
        Prediction<Label> prediction = linear.predict(testData.getExample(0));

        //The complete test dataset may be evaluated to calculate accuracy, F1, etc.
        LabelEvaluation evaluation = new LabelEvaluator().evaluate(linear, testData);

        //You can also manually inspect the rating
        double acc = evaluation.accuracy();

        // 0.Returns 978
        //Display the formatted evaluation string.
        System.out.println(evaluation.toString());
    }
}

I just created one class with the name SampleTribuo. Run it as is.

Execution result

September 29, 2020 5:12:51 pm org.tribuo.data.csv.CSVIterator getRow
warning: Ignoring extra newline at line 151
September 29, 2020 5:12:51 pm org.tribuo.classification.sgd.linear.LinearSGDTrainer train
information: Training SGD classifier with 105 examples
September 29, 2020 5:12:51 pm org.tribuo.classification.sgd.linear.LinearSGDTrainer train
information: Labels - (0,Iris-versicolor,34), (1,Iris-virginica,35), (2,Iris-setosa,36)
Class                           n          tp          fn          fp      recall        prec          f1
Iris-versicolor                16          16           0           1       1.000       0.941       0.970
Iris-virginica                 15          14           1           0       0.933       1.000       0.966
Iris-setosa                    14          14           0           0       1.000       1.000       1.000
Total                          45          44           1           1
Accuracy                                                                    0.978
Micro Average                                                               0.978       0.978       0.978
Macro Average                                                               0.978       0.980       0.978
Balanced Error Rate                                                         0.022

Description of each item

In the created SampleTribuo code, the following part is the header part of the iris data.

//Read labeled iris data
var irisHeaders = new String[] { "sepalLength", "sepalWidth", "petalLength", "petalWidth", "species" };

The name of the header is as follows.

・ SepalLength = length of calyx ・ SepalWidth = width of calyx ・ PetalLength = Petal Length ・ PetalWidth = width of petals ・ Species = types

Also, if you pay attention to the types of iris data, you can see that the following three types of data are included. The following is an excerpt from Downloaded Iris Data.

・ 5.8,4.0,1.2,0.2, Iris-setosa ・ 6.7,2.5,5.8,1.8, Iris-virginica ・ 5.5,2.6,4.4,1.2, Iris-versicolor

The types are as follows. It seems to be a kind of iris.

・ Iris setosa = Iris setosa ・ Iris versicolor = barge color ・ Iris virginica = Virginia (Virginica?)

In the Tribuo sample code, the last output by system.out.println is the LabelEvaluation class. And the LabelEvaluation class receives the trained Model of linearTrainer and the test data. The following is the relevant part. The output result already mentioned in the previous section is the evaluation result of the data set.

SampleTribuo

Omission

//You can also use logistic regression
var linearTrainer = new LogisticRegressionTrainer();
Model<Label> linear = linearTrainer.train(trainData);

//The complete test dataset may be evaluated to calculate accuracy, F1, etc.
LabelEvaluation evaluation = new LabelEvaluator().evaluate(linear, testData);

Omission

//Display the formatted evaluation string.
System.out.println(evaluation.toString());

Looking at the Java Doc of the LabelEvaluation class, the following content was written. The function to output individually or output in HTML is implemented.

Excerpt from JavaDoc Label Evaluation item

double	accuracy()Overall accuracy of the evaluation.

double	accuracy(Label label)Accuracy per rating label.

double	AUCROC(Label label)Area below the ROC curve.

double	averageAUCROC(boolean weighted)The area under the ROC curve averaged between labels.

double	averagedPrecision(Label label)Accuracy by taking a weighted average of accuracy at a given threshold-Summarizing the recall curve, the weights represent the recalls achieved at that threshold.

LabelEvaluationUtil.PRCurve	precisionRecallCurve(Label label)Calculates a single-label precision recall curve.

static String	toFormattedString(LabelEvaluation evaluation)This method produces a nicely formatted string output with the appropriate tabs and line breaks for display in the terminal.

default String	toHTML()Returns an HTML-formatted string that represents this rating.

static String	toHTML(LabelEvaluation evaluation)This method produces HTML-formatted string output with the appropriate tabs and line breaks, suitable for embedding in a web page.


static String	toFormattedString(LabelEvaluation evaluation)This method produces a nicely formatted string output with the appropriate tabs and line breaks for display in the terminal.

static String	toHTML(LabelEvaluation evaluation)This method produces HTML-formatted string output with the appropriate tabs and line breaks, suitable for embedding in a web page.

The following part is the part that predicts unknown data. That said, I don't have any unknown data, so it's 30% of the divided data.

Omission

//Ultimately, predictions are made from unknown data.
//Forecasts are the output name (ie label) and the score/It becomes a probability.
Prediction<Label> prediction = linear.predict(testData.getExample(0));

//Added.
//Unknown data(test data)Output the contents of.
System.out.println("Test data to enter: " + testData.getExample(0));

//Output prediction data from Model.
System.out.println("Prediction result: " + prediction.getOutput());
Omission

Original material is implemented as it is, but since the first test data is predicted and nothing is done as it is, this is output. I will try.

September 29, 2020 9:34:34 pm org.tribuo.data.csv.CSVIterator getRow
warning: Ignoring extra newline at line 151
September 29, 2020 9:34:34 pm org.tribuo.classification.sgd.linear.LinearSGDTrainer train
information: Training SGD classifier with 105 examples
September 29, 2020 9:34:34 pm org.tribuo.classification.sgd.linear.LinearSGDTrainer train
information: Labels - (0,Iris-versicolor,34), (1,Iris-virginica,35), (2,Iris-setosa,36)
Test data to enter: ArrayExample(numFeatures=4,output=Iris-setosa,weight=1.0,features=[(petalLength, 1.3)(petalWidth, 0.3), (sepalLength, 4.5), (sepalWidth, 2.3), ])
Prediction result: (Iris-setosa,0.8752021370239595)

The input data are the data of Iris-setosa, each of which has a sepal length of 4.5, a sepal width of 2.3, a petal length of 1.3, and petals. The width (petal Width) is 0.3.

On the other hand, the prediction result was 87% probability, and it was predicted to be Iris-setosa. You're right.

This alone is not very interesting, so let's predict 30% of the data. I changed the code as below.

SampleTribuo

Omission
for (var example : testData) {
    //Correct answer
    var correctAnswer = example.getOutput().getLabel();
    //Prediction result
    var predictResult = linear.predict(example);

    if (correctAnswer.equals(predictResult.getOutput().getLabel())) {
        System.out.println("Prediction result->Correct answer: " + predictResult.getOutputScores());
    } else {
        System.out.println("Prediction result->Incorrect answer: " + predictResult.getOutputScores() + "Correct answer->" + correctAnswer);
    }
}
Omission

Execution result

September 29, 2020 10:35:18 pm org.tribuo.data.csv.CSVIterator getRow
warning: Ignoring extra newline at line 151
September 29, 2020 10:35:18 pm org.tribuo.classification.sgd.linear.LinearSGDTrainer train
information: Training SGD classifier with 105 examples
September 29, 2020 10:35:18 pm org.tribuo.classification.sgd.linear.LinearSGDTrainer train
information: Labels - (0,Iris-versicolor,34), (1,Iris-virginica,35), (2,Iris-setosa,36)
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.12465430241942166), Iris-virginica=(Iris-virginica,1.4356055661867684E-4), Iris-setosa=(Iris-setosa,0.8752021370239595)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.022985338660181973), Iris-virginica=(Iris-virginica,4.886599414569194E-6), Iris-setosa=(Iris-setosa,0.9770097747404035)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.8743791900480306), Iris-virginica=(Iris-virginica,0.11972594367759447), Iris-setosa=(Iris-setosa,0.005894866274374883)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.13752628567026762), Iris-virginica=(Iris-virginica,0.8624557718013333), Iris-setosa=(Iris-setosa,1.794252839916067E-5)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.754662316591029), Iris-virginica=(Iris-virginica,0.24288192724655125), Iris-setosa=(Iris-setosa,0.00245575616241976)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.1114591917841135), Iris-virginica=(Iris-virginica,0.8885339140853985), Iris-setosa=(Iris-setosa,6.89413048798891E-6)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.018368556099769975), Iris-virginica=(Iris-virginica,2.8979851163953406E-6), Iris-setosa=(Iris-setosa,0.9816285459151136)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.8336662376434194), Iris-virginica=(Iris-virginica,0.16052879318949909), Iris-setosa=(Iris-setosa,0.005804969167081375)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.7401640849157867), Iris-virginica=(Iris-virginica,0.25814459323929695), Iris-setosa=(Iris-setosa,0.0016913218449164506)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.37734213887670864), Iris-virginica=(Iris-virginica,0.6225430896188512), Iris-setosa=(Iris-setosa,1.1477150444008675E-4)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.08165259920209698), Iris-virginica=(Iris-virginica,0.9183413054167227), Iris-setosa=(Iris-setosa,6.095381180367614E-6)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.9166451164592634), Iris-virginica=(Iris-virginica,0.0710098849793183), Iris-setosa=(Iris-setosa,0.012344998561418293)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.12496197859394245), Iris-virginica=(Iris-virginica,0.8750207513327715), Iris-setosa=(Iris-setosa,1.7270073286161786E-5)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.08832616235771579), Iris-virginica=(Iris-virginica,0.9116697571921494), Iris-setosa=(Iris-setosa,4.080450134879567E-6)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.7606313121553816), Iris-virginica=(Iris-virginica,0.23595375784889286), Iris-setosa=(Iris-setosa,0.0034149299957255148)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.2838330052617929), Iris-virginica=(Iris-virginica,0.7160953733169281), Iris-setosa=(Iris-setosa,7.162142127894781E-5)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.8962905028787402), Iris-virginica=(Iris-virginica,0.076593124958417), Iris-setosa=(Iris-setosa,0.027116372162842753)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.8781294741330369), Iris-virginica=(Iris-virginica,0.11263604829308826), Iris-setosa=(Iris-setosa,0.009234477573874762)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.02598246093746204), Iris-virginica=(Iris-virginica,9.258026355622071E-6), Iris-setosa=(Iris-setosa,0.9740082810361823)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.4125332441403861), Iris-virginica=(Iris-virginica,0.5874113240653356), Iris-setosa=(Iris-setosa,5.54317942782377E-5)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.005480793710204649), Iris-virginica=(Iris-virginica,3.36057635603426E-7), Iris-setosa=(Iris-setosa,0.9945188702321598)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.7157983219006878), Iris-virginica=(Iris-virginica,0.28236588916463473), Iris-setosa=(Iris-setosa,0.0018357889346774156)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.041785356794614356), Iris-virginica=(Iris-virginica,1.63804767321482E-5), Iris-setosa=(Iris-setosa,0.9581982627286535)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.03557299656267042), Iris-virginica=(Iris-virginica,1.572396823310113E-5), Iris-setosa=(Iris-setosa,0.9644112794690964)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.7445227755500248), Iris-virginica=(Iris-virginica,0.25336879249494487), Iris-setosa=(Iris-setosa,0.0021084319550302984)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.23201927045744059), Iris-virginica=(Iris-virginica,0.7679353747675968), Iris-setosa=(Iris-setosa,4.535477496262948E-5)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.6136153688335659), Iris-virginica=(Iris-virginica,0.38487059274540525), Iris-setosa=(Iris-setosa,0.0015140384210286877)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.0037281020760574794), Iris-virginica=(Iris-virginica,1.291282392314054E-7), Iris-setosa=(Iris-setosa,0.9962717687957032)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.04755707288123218), Iris-virginica=(Iris-virginica,1.934947705107781E-5), Iris-setosa=(Iris-setosa,0.9524235776417168)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.19993977432885074), Iris-virginica=(Iris-virginica,0.8000350473542086), Iris-setosa=(Iris-setosa,2.5178316940654208E-5)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.04523347766974972), Iris-virginica=(Iris-virginica,2.000119721397452E-5), Iris-setosa=(Iris-setosa,0.9547465211330363)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.029794358762199272), Iris-virginica=(Iris-virginica,8.7737002425928E-6), Iris-setosa=(Iris-setosa,0.9701968675375582)}
Prediction result->Incorrect answer: {Iris-versicolor=(Iris-versicolor,0.5733177742115076), Iris-virginica=(Iris-virginica,0.42626475284848847), Iris-setosa=(Iris-setosa,4.1747294000396004E-4)}Correct answer-> Iris-virginica
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.029877036683050945), Iris-virginica=(Iris-virginica,7.161847786201107E-6), Iris-setosa=(Iris-setosa,0.970115801469163)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.7821104478321824), Iris-virginica=(Iris-virginica,0.21391875776045305), Iris-setosa=(Iris-setosa,0.003970794407364584)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.019534516466634153), Iris-virginica=(Iris-virginica,5.375894384912967E-6), Iris-setosa=(Iris-setosa,0.9804601076389811)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.05383775370019657), Iris-virginica=(Iris-virginica,0.9461587580877066), Iris-setosa=(Iris-setosa,3.4882120966764877E-6)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.12393648653270413), Iris-virginica=(Iris-virginica,0.8760476481453248), Iris-setosa=(Iris-setosa,1.58653219710635E-5)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.8991747020299696), Iris-virginica=(Iris-virginica,0.09631094366524456), Iris-setosa=(Iris-setosa,0.004514354304785789)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.8925507000636339), Iris-virginica=(Iris-virginica,0.10101559809853458), Iris-setosa=(Iris-setosa,0.00643370183783148)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.759603537166626), Iris-virginica=(Iris-virginica,0.23777484597390816), Iris-setosa=(Iris-setosa,0.00262161685946576)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.01560114791426806), Iris-virginica=(Iris-virginica,2.3647440157063515E-6), Iris-setosa=(Iris-setosa,0.9843964873417164)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.23490596676084197), Iris-virginica=(Iris-virginica,0.7650569443472615), Iris-setosa=(Iris-setosa,3.708889189648049E-5)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.5604116022032899), Iris-virginica=(Iris-virginica,0.4392143166585561), Iris-setosa=(Iris-setosa,3.740811381540314E-4)}
Prediction result->Correct answer: {Iris-versicolor=(Iris-versicolor,0.13060096881100045), Iris-virginica=(Iris-virginica,0.8693951745606155), Iris-setosa=(Iris-setosa,3.856628384041485E-6)}

One case is incorrect, but the probability of Iris-versicolor is 57% and the probability of Iris-virginica is 42%. On the contrary, even if the data is correct, there is a 56% chance of Iris-versicolor, and there is a fairly low probability of correct answer. Since there is little data to be trained, there will be a bias. From here, it should be fun to improve the accuracy.

SampleTribuo uses logistics regression to learn, but there are other classes available. There are many implementations of the Trainer interface, so just trying them out will take quite some time. After all, I would like to try to learn on GPU rather than learning on CPU, so it seems better to look at the ONNX cooperation part or TensorFlow cooperation part before the learning model itself. I feel like it.

~~ Well, I've only done machine learning for the last few days ... ~~

Afterword

In the sample Tribuo I created, I was looking for a way to save the trained data because I would read the test data and train it every time, but I could not find it in the document. From the source code, you can see that the method of saving the TensorflowModel using Tensorfrow and the method of saving the CSV are provided, but since it is the first code to touch, I do not understand the habit or thought. It seems that this is something that should be here ...

When I was thinking about serializing and deserializing on my own, I found it on Issues.

The content of the question is as follows.

Can tutorials and documentation include examples of serializing models and datasets to disk? It would be helpful to have additional help on how to use ONNX.

Here is the answer.

Sure, you can add serialization examples to your documentation. At this time, serialization uses the standard java.io.Serializable mechanism. An example of loading and saving a model or dataset can be found in the demo program that comes with Tribuo. Note that you should use the serialization allow list to prevent deserialization attacks.

The best example of an ONNX loader at the moment is in unit testing. We plan to add a tutorial that goes through the loading process of third-party models.

In other words, it is not listed in the document, and it seems that you can use java.io.Serializable and do it yourself. How to use ONNX loader is in the test code ... You have to search the source yourself. I searched for the test code, but it was loaded. that? How about saving?

I looked inside OnnxRuntime, but it's only loading. It says, "Static loader for JNI binding. There is no public API, but it is called by various classes in this package." It doesn't seem to be in the ai.onnx runtime package or the org.tribuo.interop.onnx package. In the first place, at the moment, it says "Serialization uses the standard java.io.Serializable mechanism", so it seems that it can be done with Tensorflow, but officially there is no such thing at the moment. ..

The function is fairly complete, but the documentation is not catching up even in English. In some places, javadoc is not included. It seems that if you ask, it will tell you, but it's faster to try it than to listen to it one by one, but it's annoying and I want to see it. At the moment, it feels like hmm, but I feel like it's going to change all at once. The person inside is quite hot.

Next, let's think about where to start.