The Met Office has been surprised by the way in which big data consumption has turned its weather and climate forecasting data into a product, according to the organisation’s IT chief.
Charles Ewen, chief information officer at the Met Office, said at Cloud Expo 2015, attended by V3, that the progression of cloud computing and technology in general is changing how the Met manages data.
“What’s happening now in a machine-to-machine world, with predicative analytics and artificial neural networks, is the ability to take weather and climate insights and mix it with other kinds of data to do useful things. It means that increasingly data is the product,” he said.
“So we are now in a place where we have to change ourselves such that we can make this data the product.”
Ewen explained that the change has taken the Met Office by surprise. “From a technology architecture point of view, we didn’t see this coming,” he said.
“We assumed the world was going to carry on with doing all the work that we do in a highly homogenous, highly place-based [location] in Exeter, Devon – where all this happens – and we would produce products and services for others to consume,” he said.
Ewen explained that providing third parties with access to the vast volumes of data the Met Office harvests from its weather modelling is a technical challenge that can strain the capacity of networks and related systems.
The Met Office is working with a variety of technologies and techniques to ensure that only useful data is passed on to other organisations.
“It’s thinking about information not data, cutting down the size of the data by being specific and dynamic with downstream users about what data they actually need,” he said.
Using the cloud as a platform could support the Met Office in delivering data in this dynamic fashion, but the organisation will not fully migrate to the cloud given the scale of data it needs to manage in its data centre and supercomputer.
“We don’t have a strategy to move to the cloud. We will use the cloud in the context of the mixed economy to solve the IT problems for which the cloud looks like a good tool to use,” he said.
The Met Office has traditionally been structured to gather around 7TB of weather and modelling data every day and reduce it into usable information.
“The data produced at this scale is already too big to be very useful for too many people. We try and turn that data back into information, to put knowledge in somebody’s head, ultimately so that somebody can make a decision,” Ewen said.
The Met Office may not have predicted the demand for big data, but Ewen forecasted floods of big data for government organisations last year.