Upload & move GCS files with Go

I tried using Golang to upload and move files to GCS. The repository can be found here. You can set up a docker container and try it out. https://github.com/greenteabiscuit/gcs-golang

Prerequisites

--I have a GCP account --Docker has been downloaded --You have downloaded the key file that has GCS viewing and editing privileges.

things to do

1. Try uploading an empty file sample.txt to GCS
2. Move the sample.txt uploaded to GCS to another directory

1. Try uploading

main.go

package main

import (
	"context"
	"io"
	"log"
	"os"

	"cloud.google.com/go/storage"
	"google.golang.org/api/option"
)

func main() {
	credentialFilePath := "./key.json" // key.json creates a service account and gets

	//Create a client
	ctx := context.Background()
	client, err := storage.NewClient(ctx, option.WithCredentialsFile(credentialFilePath))
	if err != nil {
		log.Fatal(err)
	}

	//Empty sample to write GCS object.Create txt file locally
	f, err := os.Create("sample.txt")
	if err != nil {
		log.Fatal(err)
	}

	//Create a Reader for the object
	bucketName := "experimental-bucket-tt"   // e.g. example-bucket
	objectPath := "sample-object/sample.txt" // e.g. foo/var/sample.txt

	writer := client.Bucket(bucketName).Object(objectPath).NewWriter(ctx)
	if _, err := io.Copy(writer, f); err != nil {
		panic(err)
	}

	if err := writer.Close(); err != nil {
		panic(err)
	}

	log.Println("done")
}

Set up a virtual environment with Docker and try running it in it. Don't forget to download the module.

$ docker-compose up

// In different tab
$ docker exec -it containername bash

//You need to download the following modules.

/go/src# go mod init example.com/gcs/write

/go/src# go get -u cloud.google.com/go/storage

Hopefully it will say done.

/go/src# go run main.go
2020/12/29 01:36:24 done

Sample.txt should have been uploaded in the root directory of the bucket.

2. Move the file within GCS.

Unfortunately there is no function to just move, so the order is to copy to the desired folder and delete the files in the original folder. Also, if the folder in the middle is not created, the GCS side (?) Will create it for you. This time, move it to a folder named destination-folder/30.

main.go

package main

import (
	"context"
	"io"
	"log"
	"os"

	"cloud.google.com/go/storage"
	"google.golang.org/api/option"
)

func main() {
	credentialFilePath := "./key.json" // key.json creates a service account and gets

	//Create a client
	ctx := context.Background()
	client, err := storage.NewClient(ctx, option.WithCredentialsFile(credentialFilePath))
	if err != nil {
		log.Fatal(err)
	}

	//Creating a file to write a GCS object
	f, err := os.Create("sample.txt")
	if err != nil {
		log.Fatal(err)
	}

	// ......................
	//Object upload
	// ......................
	bucketName := "experimental-bucket-tt"   // e.g. example-bucket
	objectPath := "sample-object/sample.txt" // e.g. foo/var/sample.txt

	uploadWriter := client.Bucket(bucketName).Object(objectPath).NewWriter(ctx)
	if _, err := io.Copy(uploadWriter, f); err != nil {
		panic(err)
	}

	if err := uploadWriter.Close(); err != nil {
		panic(err)
	}
	log.Println("create file: done")

	// ......................
	//Move objects
	// ......................
	dstObjectPath := "destination-folder/30/sample.txt"
	src := client.Bucket(bucketName).Object(objectPath)
	dst := client.Bucket(bucketName).Object(dstObjectPath)
	if _, err := dst.CopierFrom(src).Run(ctx); err != nil {
		panic(err)
	}
	if err := src.Delete(ctx); err != nil {
		panic(err)
	}
	log.Println("move file: done")
}

You should now be able to create a file, upload it, and move the uploaded file.

/go/src# go run main.go
2020/12/29 10:05:55 create file: done
2020/12/29 10:05:55 move file: done

It should have been moved to the destination-folder/30 folder.

reference

Reading and Writing to Google Cloud Storage Upload files to Google Cloud Storage in Go language (golang)