Migrate your data in Google Cloud Storage using Power of Talend

What is Google Cloud Storage:


Google Cloud storage allow us to store and manage any amount of data world-wide. Any one can use Google Cloud Storage for a wide of scenarios like Analytical data, Web content,  data for archival and disaster recovery, or distributing large data objects to users etc.

Cloud Storage provides fast, low-cost, highly durable storage for data accessed less than once a month. Perfect for reducing the cost of backups and archives while still retaining immediate access. Backup data in Cloud Storage can be used for more than just recovery because all Cloud Storage classes have ms latency and are accessed through a single API.

Pricing details by storage class:

Savings vs. AWS by storage class:

Lets Start using Google Cloud Storage:

In order to use google cloud services we have to go to https://console.cloud.google.com and sign in with your gmail account (Create a new gmail account if not exist already). On successful login you will be redirected to following page 

Here in the left side all the services provided by google cloud are listed related to different categories included Compute, Storage, Networking, Big Data, Artificial Intelligence and other Tools. I am going to illustrate Google Cloud Storage here.

To start using storage services first of all we have to create a project and also sign up with your personal and card details by clicking on Sign up for a free trial as below.

Once you successfully signed up you can create bucket and use the google storage also you google cloud will guide you with quick start service. 

• Create a bucket

Buckets hold the objects (any type of file) you want to store in Cloud Storage.

• Upload and share objects

Start using your bucket by uploading an object and making it publicly available.

• Clean up

As a final step, you'll delete the bucket and object you created for this tutorial.

Since you'll be deleting your bucket and object at the end, don't worry about getting charged for trying this tutorial.

For uploading a file or Folder clicking on upload files or folders in any bucket.

Now come to Talend Platform and supported components. Following is a list of Talend components used for communicating with Google Storage.

• tGSConnection: This component will be used to create a connection with Google Storage using Access Key and Secret Key.                                                                                                               

   To get the values of access key and secret key click on settings and than select Interoperability. The Interoperability API lets you use HMAC authentication and lets Cloud Storage interoperate with tools written for other cloud storage systems. Turn on this API only if you require interoperable access for the current user. This API is enabled per project member, not per project. Each member can set a default project and maintain their own access keys.

Using these values we can connect with google storage and further other talend components like tGSBucketCreate, tGSPut, tGSGet etc. can be used to create a bucket upload a file and download a file etc.

Also this access key and secret key does not change with project in one gmail account. If you are using same gmail account your access key and access secret will remain same.

Following is a small talend job for uploading file on google storage bucket.