[JAVA] Try putting a lot of data into Cosmos DB

For a little verification purpose, I created a simple sample Java code that throws a lot of data into Cosmos DB. As a matter of course, if a large number of transactions are executed, the billing amount will increase, so please use at your own risk.

https://github.com/yahanda/cosmosdb-testdata-generator

how to use

  1. Download the Yellow taxi boarding / alighting data from the Taxi and Limousine Commission page in New York City. , Place it in any folder. (Example: C: \\ Temp \\)
  2. Load pom.xml in Eclipse and open the Java perspective.
  3. Open ʻappsettings.propertiesand update the following property values for your Cosmos DB environment. ForCsvFileDir`, specify the folder where the CSV file was placed in step 1).
EndPointUrl=https://REPLACE_ME_WITH_YOURS.documents.azure.com:443/
PrimaryKey=REPLACE_ME_WITH_YOURS
Database=TestDB
Collection=TestCollection
CsvFileDir=C:\\Temp\\
  1. Run GenerateNYCTaxiData.java as a Java Application.

Execution example


Created a new database:
{"_rid":"DewXAA==","id":"TestDB","_self":"dbs/DewXAA==/","_colls":"colls/","_users":"users/","_etag":"\"00002908-0000-2300-0000-5e4cf35e0000\"","_ts":1582101343}
Press any key to continue..

Created a new collection:
{"uniqueKeyPolicy":{"uniqueKeys":[]},"geospatialConfig":{"type":"Geography"},"_rid":"DewXAIlOilc=","_sprocs":"sprocs/","_triggers":"triggers/","_udfs":"udfs/","conflictResolutionPolicy":{"mode":"LastWriterWins","conflictResolutionProcedure":"","conflictResolutionPath":"/_ts"},"_docs":"docs/","partitionKey":{"paths":["/_partitionKey"],"kind":"Hash"},"indexingPolicy":{"indexingMode":"consistent","automatic":true,"includedPaths":[{"path":"/*"}],"excludedPaths":[{"path":"/\"_etag\"/?"}]},"id":"TestCollection","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/","_conflicts":"conflicts/","_etag":"\"00001f05-0000-2300-0000-5e4cf3620000\"","_ts":1582101346}
Press any key to continue..
Read a csv file:
C:\Temp\yellow_tripdata_2019-01.csv
Press any key to continue..

Header: VendorID,tpep_pickup_datetime,tpep_dropoff_datetime,passenger_count,trip_distance,RatecodeID,store_and_fwd_flag,PULocationID,DOLocationID,payment_type,fare_amount,extra,mta_tax,tip_amount,tolls_amount,improvement_surcharge,total_amount,congestion_surcharge
Created 1 document:
{"_rid":"DewXAIlOilcBAAAAAAAAAA==","puLocationid":"151","fare_amount":"7","tpep_dropoff_datetime":"2019-01-01 00:53:20","congestion_surcharge":"","vendorid":"1","passenger_count":"1","tolls_amount":"0","improvement_surcharge":"0.3","trip_distance":"1.50","_attachments":"attachments/","store_and_fwd_flag":"N","payment_type":"1","ratecodeid":"1","total_amount":"9.95","extra":"0.5","tip_amount":"1.65","mta_tax":"0.5","id":"92ce47d1-b876-443e-8e47-d1b876643e4a","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/docs/DewXAIlOilcBAAAAAAAAAA==/","doLocationid":"239","tpep_pickup_datetime":"2019-01-01 00:46:40","_etag":"\"00003108-0000-2300-0000-5e4cf3660000\"","_ts":1582101350}
Created 2 document:
{"_rid":"DewXAIlOilcCAAAAAAAAAA==","puLocationid":"239","fare_amount":"14","tpep_dropoff_datetime":"2019-01-01 01:18:59","congestion_surcharge":"","vendorid":"1","passenger_count":"1","tolls_amount":"0","improvement_surcharge":"0.3","trip_distance":"2.60","_attachments":"attachments/","store_and_fwd_flag":"N","payment_type":"1","ratecodeid":"1","total_amount":"16.3","extra":"0.5","tip_amount":"1","mta_tax":"0.5","id":"14a612b4-4eac-4aa3-a612-b44eacdaa3b6","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/docs/DewXAIlOilcCAAAAAAAAAA==/","doLocationid":"246","tpep_pickup_datetime":"2019-01-01 00:59:47","_etag":"\"00003208-0000-2300-0000-5e4cf3660000\"","_ts":1582101350}
Created 3 document:
{"_rid":"DewXAIlOilcDAAAAAAAAAA==","puLocationid":"236","fare_amount":"4.5","tpep_dropoff_datetime":"2018-12-21 13:52:40","congestion_surcharge":"","vendorid":"2","passenger_count":"3","tolls_amount":"0","improvement_surcharge":"0.3","trip_distance":".00","_attachments":"attachments/","store_and_fwd_flag":"N","payment_type":"1","ratecodeid":"1","total_amount":"5.8","extra":"0.5","tip_amount":"0","mta_tax":"0.5","id":"69d7e305-53e1-4efd-97e3-0553e18efd36","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/docs/DewXAIlOilcDAAAAAAAAAA==/","doLocationid":"236","tpep_pickup_datetime":"2018-12-21 13:48:30","_etag":"\"00003308-0000-2300-0000-5e4cf3660000\"","_ts":1582101350}
Created 4 document:
{"_rid":"DewXAIlOilcEAAAAAAAAAA==","puLocationid":"193","fare_amount":"3.5","tpep_dropoff_datetime":"2018-11-28 15:55:45","congestion_surcharge":"","vendorid":"2","passenger_count":"5","tolls_amount":"0","improvement_surcharge":"0.3","trip_distance":".00","_attachments":"attachments/","store_and_fwd_flag":"N","payment_type":"2","ratecodeid":"1","total_amount":"7.55","extra":"0.5","tip_amount":"0","mta_tax":"0.5","id":"3cb3e88a-da63-4060-b3e8-8ada63d060b3","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/docs/DewXAIlOilcEAAAAAAAAAA==/","doLocationid":"193","tpep_pickup_datetime":"2018-11-28 15:52:25","_etag":"\"00003408-0000-2300-0000-5e4cf3660000\"","_ts":1582101350}
Created 5 document:
{"_rid":"DewXAIlOilcFAAAAAAAAAA==","puLocationid":"193","fare_amount":"52","tpep_dropoff_datetime":"2018-11-28 15:58:33","congestion_surcharge":"","vendorid":"2","passenger_count":"5","tolls_amount":"0","improvement_surcharge":"0.3","trip_distance":".00","_attachments":"attachments/","store_and_fwd_flag":"N","payment_type":"2","ratecodeid":"2","total_amount":"55.55","extra":"0","tip_amount":"0","mta_tax":"0.5","id":"ff1002e6-03f5-4bac-9002-e603f5cbacf5","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/docs/DewXAIlOilcFAAAAAAAAAA==/","doLocationid":"193","tpep_pickup_datetime":"2018-11-28 15:56:57","_etag":"\"00003508-0000-2300-0000-5e4cf3660000\"","_ts":1582101350}
Created 6 document:
{"_rid":"DewXAIlOilcGAAAAAAAAAA==","puLocationid":"193","fare_amount":"3.5","tpep_dropoff_datetime":"2018-11-28 16:28:26","congestion_surcharge":"","vendorid":"2","passenger_count":"5","tolls_amount":"5.76","improvement_surcharge":"0.3","trip_distance":".00","_attachments":"attachments/","store_and_fwd_flag":"N","payment_type":"2","ratecodeid":"1","total_amount":"13.31","extra":"0.5","tip_amount":"0","mta_tax":"0.5","id":"3cf17f8b-617e-49a4-b17f-8b617e29a49c","_self":"dbs/DewXAA==/colls/DewXAIlOilc=/docs/DewXAIlOilcGAAAAAAAAAA==/","doLocationid":"193","tpep_pickup_datetime":"2018-11-28 16:25:49","_etag":"\"00003608-0000-2300-0000-5e4cf3660000\"","_ts":1582101350}
Created 7 document:

:

image.png

Reference information

Recommended Posts

Try putting a lot of data into Cosmos DB
Try using Microsoft Azure SDK For SQL API of Azure Cosmos DB Service 4.6
Insert data into DB using Yaml file
Insert Eijiro's dictionary data into Oracle DB part2