Google has unveiled a new offering as part of its Cloud Platform providing on-demand access to Spark and Hadoop processing services, in order to enable customers to more easily derive useful insights from large datasets.
Available now as a beta service, Google Cloud Dataproc offers managed Spark and Hadoop capabilities for batch processing, querying, streaming, and machine learning applications. It enables customers to quickly create and manage clusters of nodes running Spark or Hadoop, and also disable them to avoid incurring costs when not required.
“With less time and money spent on administration, you can focus on your jobs and your data. In the time it takes you to read this blog post, you can have a Spark or Hadoop cluster created, configured, and ready to work for you,” said Google product manager James Malone, announcing the new service in a post on the firm’s blog.
Google claims that Cloud Dataproc is fast, low-cost, and easy to use when compared with building your own Hadoop processing clusters or even provisioning the same resources manually in the cloud, taking all the pain out of operating a big data environment.
Customers are charged on a minute-by-minute billing, at a price that works out at just one US cent per hour for each virtual CPU in the cluster, on top of any other Cloud Platform resources that the customer may be using, according to Google.
In addition, Cloud Dataproc clusters can be started up, scaled, and shutdown with each of these operations taking no more than 90 seconds, Google claimed, compared with five to 30 minutes to create Spark and Hadoop clusters manually.
At launch, Google Cloud Dataproc provisions customer data processing clusters with Spark 1.5 or Hadoop 2.7.1.
Users can interact with clusters and Spark or Hadoop jobs through the Google Developers Console, the Google Cloud SDK, or the Cloud Dataproc REST application programming interface (API). The service also integrates with other Google Cloud Platform offerings, including Cloud Storage, BigQuery, and Cloud Bigtable.