Blog icon

The challenge

Understanding and using data

Public data is a critically under-utilised national asset. The Australian Government's data holdings are a strategic national resource with considerable value for growing the economy - the better that the government can understand and use its data, the better it is able to make decisions and discover new opportunities. Many government agencies hold massive quantities of data, but it often gets stuck inside silos where its importance is invisible, origins untracked, and existence unknown to those elsewhere in the organisation who could improve or derive further value from it.

Outside of Government, there is considerable potential for public data to be openly published, and further utilised by the private sector, researchers and engaged citizens. The Australian Government is one of the world leaders in making government datasets open, but for the public to make use of the data, they need well-described and up-to-date data and powerful discovery tools in order to quickly find the data that is relevant to them.

Our response

Providing open-source software

Magda is a data catalog system that provides a single place where all of an organisation's data can be catalogued, enriched, searched, tracked and prioritised - whether big or small, internally or externally sourced, available as files, databases or APIs.

Magda resolves the challenge of storing and using data by providing open-source software to assist government agencies in all aspects of data management—from collection and authoring, through internal discovery and use, all the way to sharing inter-organisation or publishing on an open data portal. In this manner it unlocks the potential of Australia's data, making it easier to find, access and reuse from both inside and outside the Australian Government.

Magda is supported under the Platforms for Open Data (PFoD) program which is a National Innovation and Science Agency (NISA) initiative. PFoD partners include the Digital Transformation Agency and the Department of Agriculture, Water and the Environment.

The results

Improving data use and sharing within the Australian Government

As part of discovery, the Magda team undertook extensive research in order to understand the current state of data sharing, pain points and future potential, conducting nearly 100 interviews with data users across the public, private and research sectors. The Magda team identified twelve opportunity areas where Magda could alleviate current state pain points and gaps around data sharing and access - this research has gone on to help inform the Office of the National Data Commissioner's efforts around improving data use and sharing within the Australian Government.

Since January 2019, Magda has been powering Australia's national data catalogue, now makes over 89,000 datasets drawn from sources all over the Australian government available for search, and makes these available to 10,000-20,000 individual users per week.

As an open-source project, Magda has also been used by other teams within CSIRO, including for the Knowledge Network project in Land and Water, and the NSW Spatial Digital Twin developed by the Terria team within CSIRO's Data61.

Over the past 18 months, Magda has been extending the functionality initially developed for into a more general-purpose data cataloguing solution for use within government agencies. Working prototypes have been developed and deployed for the use of the Department of Agriculture, Water and Environment (DAWE), and will continue to be added to in an agile manner. Of particular note is tackling the problem of making metadata around datasets easier and quicker to create for non-expert users.

The Magda team are looking forward to continuing to support , enhancing its reputation as an open-source project and rounding out its new metadata generation tool to DAWE, which is of particular use to the team writing the next State of the Environment report.

Contact us

Find out how we can help you and your business. Get in touch using the form below and our experts will get in contact soon!

CSIRO will handle your personal information in accordance with the Privacy Act 1988 (Cth) and our Privacy Policy.

First name must be filled in

Surname must be filled in

I am representing *

Please choose an option

Please provide a subject for the enquriy

0 / 100

We'll need to know what you want to contact us about so we can give you an answer

0 / 1900

You shouldn't be able to see this field. Please try again and leave the field blank.