Jekyll2019-03-22T01:49:44+00:00/feed.xmlblog.data.gov.auGitHub pages for blog.data.gov.auFebruary 2019 Release of G-NAF and Administrative Boundaries now available2019-02-20T00:00:00+00:002019-02-20T00:00:00+00:00/news-media/blog/february-2019-release-g-naf-and-administrative-boundaries-now-available<p>The February 2019 release of the Geocoded National Address File (G-NAF) and Administrative Boundaries datasets are now available for download.</p>
<ul>
<li>PSMA Geocoded National Address File (G-NAF) - <a href="https://data.gov.au/dataset/ds-dga-19432f89-dc3a-4ef3-b943-5326ef1dbecc/details">https://data.gov.au/dataset/ds-dga-19432f89-dc3a-4ef3-b943-5326ef1dbecc/details<span class="sr-only"> - external site</span></a></li>
<li>PSMA Administrative Boundaries - <a href="https://data.gov.au/dataset/ds-dga-bdcf5b09-89bc-47ec-9281-6b8e9ee147aa/details?q=PSMA">https://data.gov.au/dataset/ds-dga-bdcf5b09-89bc-47ec-9281-6b8e9ee147aa/details?q=PSMA<span class="sr-only"> - external site</span></a></li>
</ul>
<p>We encourage users of the data to update to the latest versions.</p>Ann BeaumarisThe February 2019 release of the Geocoded National Address File (G-NAF) and Administrative Boundaries datasets are now available for download. PSMA Geocoded National Address File (G-NAF) - https://data.gov.au/dataset/ds-dga-19432f89-dc3a-4ef3-b943-5326ef1dbecc/details - external site PSMA Administrative Boundaries - https://data.gov.au/dataset/ds-dga-bdcf5b09-89bc-47ec-9281-6b8e9ee147aa/details?q=PSMA - external site We encourage users of the data to update to the latest versions.Saving lives with open data2018-12-20T00:00:00+00:002018-12-20T00:00:00+00:00/news-media/blog/saving-lives-open-data<p><img class="au-responsive-media-img" typeof="foaf:Image" src="/assets/img/crashboard750cluster2.png" alt="Screenshot of crash clusters on map area in City of Casey" /></p>
<p><br /><strong>The winners of the DTA's 2018 GovHack award produced a stunningly useful tool to gain insights into road crash data.</strong></p>
<h2>Project Crashboard</h2>
<p>Michael Murrell and Ashleigh Latter are the as.numeric team. Between the two of them, they created Crashboard, winning the ‘help government decide with data’ award. The interactive tool shows not only where road crashes happen, but when and why. The dashboard can also compare locations across the state to allow state government managers to ‘decide with data’ about where they allocate resources.</p>
<p>We dug a bit deeper and asked as.numeric a few open, data-relatable questions.</p>
<p><strong>1. How do you prep for something like GovHack? Apart from building a tolerance to sleep… do you come in with some datasets already mixed and matched?</strong></p>
<p>GovHack is difficult to prepare for as the award categories aren’t known in advance. A couple of things we could do is brush up on basic data analysis and app building skills, as well as to make sure any relevant software was installed and up-to-date. This helped us make a quick start once we came up with our project idea.</p>
<p><strong>2. With the datasets you used, what were the qualities that attracted you to them (format, frequency, description, etc.)</strong></p>
<p>Since we only had a weekend to complete the project, we looked for smaller, easy to understand datasets. Things like data cleanliness (e.g. minimal missing entries, unambiguous features) and familiar file formats were important to get on with the task quickly. Data with common features that could be easily matched across multiple datasets (such as postcode, date etc) were also a bonus.</p>
<p><strong>3. What type of datasets do you personally think are critical for building Australia’s open data future?</strong></p>
<p>We believe health and safety, as well as transport data are vital for the future of open data in Australia. These types of datasets can be utilised to develop valuable insights and tools that have the potential to directly impact the lives of everyday Australians in a positive way. If the public can see the benefits of using open government data, they are more likely to get on-board with future open data projects.</p>
<p><strong>4. Now about your team. What skill sets make a 'dream team'?</strong></p>
<p>Our team consisted of just the 2 of us, but we were lucky to have an assortment of skills that complemented each other. Ashleigh used her skills in data analysis, app building and project planning to manipulate and join datasets into the required form and then build the app. Michael has a background in statistics and audio engineering, which he used to develop the underlying app scoring metric and produce the project video. It is important to not only be able to use the data to build something but to also communicate why your project is useful.</p>
<p><strong>5. You have a newbie friend about to enter GovHack next year for the very first time. What three tips would you give them?</strong></p>
<ol>
<li>Come in with a plan and stick to it. Know what you should be working on and when – include things like data exploration, making the project and creating the video.</li>
<li>Focus on developing a narrative around your idea. You need to communicate why your project is useful and how it meets award criteria. Showing off your programming skills won’t help you if you can’t explain why your idea is beneficial.</li>
<li> Don’t neglect the video! This is how you showcase your idea – think of it as a sales pitch. It is more important to fully explain what your project intends to do, instead of attempting to develop a fully functioning app that isn’t well pitched. Plan to spend plenty of time on the video as it takes a lot longer than you might expect to storyboard, record and edit video and audio into a polished video submission.</li>
</ol>
<p><strong>6. Your project was about safety. Why?</strong></p>
<p>We attended the Casey event, where their award category focused on road safety. We wanted to develop something that could help not just our local community, but the rest of Australia as well. Millions of Australians use our roads every day, so if we can improve the safety of our roads and help make journeys faster, then we are benefiting a very large portion of Australians.</p>
<p><strong>7. If you had all the Australian Government Data you could hope for, and a month, what would be one ‘super project’ you’d have a crack at?</strong></p>
<p>One project we would like to attempt is a health tool with the aim of identifying areas in Australia with higher than average rates of various illnesses. We would use data from multiple sources (such as climate, health, socioeconomic, etc.) to determine what could be contributing to the increased illness rates, and provide suggestions based on identified factors to improve outcomes in those areas.</p>
<p><strong>8. To government data publishers, any friendly advice?</strong></p>
<p>- Keep datasets updated. It’s great seeing government departments publishing datasets, but if they’re not kept current, then they gradually become less and less useful for present and future projects.</p>
<p>- Work with data analysts to improve data quality and formatting. The data needs to be unambiguous and as clean as possible to encourage people to use it for developing new insights and tools.</p>
<p>Thank you, Michael and Ashleigh, View their GovHack project (including their video) on their <a href="https://hackerspace.govhack.org/projects/crashboard_177">hackerspace page<span class="sr-only"> - external site</span></a>.</p>
<p><em>The DTA was lead agency sponsor of the 2018 GovHack awards. Find out more about the annual event at <a href="http://www.govhack.org">www.govhack.org<span class="sr-only"> - external site</span></a></em></p>
<p>
</p>
<div class="video-filter">
<iframe src="//www.youtube.com/embed/k9qz77-9Jf8?modestbranding=0&html5=1&rel=1&autoplay=1&wmode=opaque&loop=0&controls=1&autohide=0&showinfo=0&theme=dark&color=red&enablejsapi=0" width="750" height="394" class="video-filter video-youtube video-center vf-k9qz779jf8" frameborder="0" allowfullscreen="true"></iframe></div>Stephen CanningsNovember 2018 Release of G-NAF and Administrative Boundaries now available2018-11-20T00:00:00+00:002018-11-20T00:00:00+00:00/news-media/blog/november-2018-release-g-naf-and-administrative-boundaries-now-available<p>The November 2018 release of the Geocoded National Address File (G-NAF) and Administrative Boundaries datasets are now available for download. </p>
<ul>
<li>PSMA Geocoded National Address File (G-NAF)<br /><a href="https://data.gov.au/dataset/geocoded-national-address-file-g-naf">https://data.gov.au/dataset/geocoded-national-address-file-g-naf</a></li>
<li>PSMA Administrative Boundaries<br /><a href="https://data.gov.au/dataset/psma-administrative-boundaries">https://data.gov.au/dataset/psma-administrative-boundaries</a></li>
</ul>
<p>We encourage users of the data to update to the latest versions.</p>
<p>Please forward any queries about the data to <a href="mailto:spatial@industry.gov.au" class="spamspan">spatial@industry.gov.au</a></p>Ann BeaumarisThe November 2018 release of the Geocoded National Address File (G-NAF) and Administrative Boundaries datasets are now available for download. PSMA Geocoded National Address File (G-NAF)https://data.gov.au/dataset/geocoded-national-address-file-g-naf PSMA Administrative Boundarieshttps://data.gov.au/dataset/psma-administrative-boundaries We encourage users of the data to update to the latest versions. Please forward any queries about the data to spatial@industry.gov.auInside the judge’s tent2018-11-05T00:00:00+00:002018-11-05T00:00:00+00:00/news-media/blog/inside-the-judges-tent<p><img class="au-responsive-media-img" src="/assets/img/circustenttop750.png" /></p>
<p><strong>We take a peek inside the GovHack 2018 judging dome. Marita Baier-Gorman ushers you inside the tent.</strong></p>
<p>As the lead agency sponsor for GovHack this year, we had the chance to design two challenges. A challenge is a way for competitors to push themselves in return for awards and prizes.</p>
<p>Many of us who’d put our hands up to contribute to GovHack this year were seasoned hackers. In the past we had mentored, organised, competed, or all of the above.</p>
<p>Over the past few years, GovHack has revealed a lot of clever entries, from developing applications for mobile to importing datasets into mapping tools so that they can be visualised.</p>
<p>This year, we wanted to do something different, and inspire competitors to take a step beyond just finding data or making it pretty. We wanted teams to think about how the wisdom gained from data could be used by government to make decisions that improve services and society.</p>
<h2>Our two challenges</h2>
<h3>1. Mix and mashup bounty award</h3>
<p>The best use of two or more data sets which seem to be completely unrelated to each other, but can be brought together to inform great solutions.</p>
<h3>2. Help government decide with data award</h3>
<p>Government bodies at local, state and national levels make many decisions each day. How can available data help them make evidence-based, informed decisions to deliver better services?</p>
<p>By the end of the weekend, we had received 118 entries for our two challenges. This was a significant undertaking for our 5 judges. Each entry has a 3-minute video, open source code and a written brief to consider, and we wanted to give each entry the space to shine.</p>
<h2>Judging process</h2>
<p>The judges (or ‘<a href="http://lexicon.ft.com/term?term=red-team">red team’</a>) created a spreadsheet to allow them to score separately to reflect their individual areas of expertise (social research, code, data analysis, emerging technology and government policy).</p>
<p>Entries were checked against the challenge descriptions and GovHack's criteria?</p>
<p>Having ranked our top entries individually, we then bunkered down for many hours over 3 days to further narrow down our selections. A small meeting room was our big top, and our Chief Technology Officer played the role of the ‘<a href="https://csrc.nist.gov/glossary/term/White-Team">white team</a>’, coming in afterward to assess entries independently. Being a long-time hacker himself, he was really excited about discovering projects that were imaginative as well as practical.</p>
<p>Although some of the entries did not meet our criteria and were excluded from judging, it didn’t stop us from going through them and enjoying the ideas they explored or wondering whether we should start hiring!</p>
<p>The rationale of the teams to put themselves in as many challenges as possible makes a lot of sense to the weekend-worn hacker; you have to be in it to win it, right? This made it important for us to have clear benchmarks for our challenges. This kept us focused during the process rather than having too much fun and getting distracted.</p>
<h2>The dilemma</h2>
<p>The judge’s dilemma was to make hard selections when there are so many worthy entries. A couple of themes emerged. We saw a lot of solutions to help prevent bankruptcy, and which locations in Australia most needed Tax Office shopfronts. There were also a lot of game-based entries, where a mobile app encouraged users to look for landmarks around their local area and update datasets in real time. It was great to see crowd-sourced data quality solutions.</p>
<p>While we can’t say who won our challenges yet, we can say that it was an honour to see the ideas and workings behind their projects. This was a privilege not lost on us, and we look forward to celebrating the incredible ingenuity of GovHack 2018 at the upcoming award ceremonies.</p>
<p><em>The National Award Ceremony is on </em><em>Saturday</em><em> November 10 in Sydney. More information on <a href="https://www.govhack.org/">the GovHack website</a>.</em></p>Marita Baier-GormanNear real-time satellite images show us the real Australia2018-10-24T00:00:00+00:002018-10-24T00:00:00+00:00/news-media/blog/near-real-time-satellite-images-show-us-real-australia<p><img class="au-responsive-media-img" src="/assets/img/natmap-sent-v2.jpg" alt="Screenshot of National Map showing Sentinel 2 flight paths" /></p>
<p><br /><strong>New open data satellite images capture our continent in a new way. Updated every few days, these ‘earth-selfies’ show us the real Australia, clouds, droughts and all.</strong></p>
<p>You are familiar with Google satellite images, right? But to see recent map and earth satellite images you probably need to live in a big U.S. city–in New York, they are updated weekly. For the rest of us, it may take months or even years before a refresh.</p>
<p>Also, these images make things look pretty and green. The endless blue skies (or put it another way, the lack of clouds) is a sure sign they are hand-selected, combined and enhanced.</p>
<p>The Sentinel-2 satellite images recently offered as open data allow us to see recent snapshots across Australia, updated every few days, to provide quite a different view to Google’s idyllic imagery.</p>
<p>While the resolution is not as high, and the purpose not the same as Google’s, you do get a couple of unique features. You can go back in time and view past images from the last 90 days (the service started 7 September 2018) and the inclusion of multi-spectral data lets us view vegetation, bodies of water and urban land cover.</p>
<p>Some of the practical applications include monitoring floods and droughts, tracking changes to vegetation, forests and coastlines and seeing the effects of fires and changes in climate. The images also contribute to disaster mapping and helping humanitarian relief efforts.</p>
<p>You can view this imagery now by loading Sentinel-2 data (from both satellites, 2A and 2B) into NationalMap, the Digital Transformation Agency’s open data visualisation tool.</p>
<p>Images are taken across the Australian continent from the same angle every 5 days. Images are often updated more frequently in overlapping images from different angles.</p>
<p> </p>
<h2>Quick start to viewing Sentinel-2 satellite images</h2>
<p>
</p>
<div class="video-filter">
<iframe src="//www.youtube.com/embed/W8aFZYzpo28?modestbranding=0&html5=1&rel=1&autoplay=1&wmode=opaque&loop=0&controls=1&autohide=0&showinfo=0&theme=dark&color=red&enablejsapi=0" width="700" height="394" class="video-filter video-youtube video-center vf-w8afzyzpo28" frameborder="0" allowfullscreen="true"></iframe></div>
<p><br /> </p>
<ol>
<li>
<p>Open <a href="https://nationalmap.gov.au/">NationalMap.gov.au</a></p>
</li>
<li>
<p>Select Add data > Satellite imagery > Sentinel-2 A</p>
</li>
<li>
<p>Then click on 'Add to map' (in top right) .</p>
</li>
<li>
<p>Repeat for Sentinel-2 B</p>
</li>
<li>
<p>Select dates to find satellite pathways over the desired area</p>
</li>
<li>
<p>Zoom in to reveal the imagery</p>
</li>
</ol>
<h3>Quick examples</h3>
<p>Here is a selection of amazing NationalMap Sentinel-2 views:</p>
<ul>
<li><a href="https://nationalmap.gov.au/#share=s-gDCqBhwzRtnXPudYNkx0IITFC3I" class="toc-filter-processed">Uluru - Topographic Map Comparison</a> (Left: Sentinel-2A Simple RGB - Oct 21 2018; Right: CesiumION Basemap)</li>
<li><a href="https://nationalmap.gov.au/#share=s-hZlEuQnVwWBcEhhfnETQxzOgmJj" class="toc-filter-processed">Tamworth vegetation</a> (green = vegetation)</li>
<li><a href="https://nationalmap.gov.au/#share=s-6euB8VV09gfygPlnQ93hK07He6Z" class="toc-filter-processed">Tamworth</a> region - use the slider to compare Sentinel-2 and Google-like images</li>
<li><a href="https://nationalmap.gov.au/#share=s-3hJhkIs6JVfhzFIBIjYwGjcQm2b" class="toc-filter-processed">Bushfire in Kakadu</a> (Left: fire, Oct 4 2018 - RGB; Right:: Oct 9 2018 - RGB)</li>
<li><a href="https://nationalmap.gov.au/#share=s-sSfbbxM8u3VjjpcBk9hy2aXgDE8" class="toc-filter-processed">Bushfire in Nitmiluk</a> (Left: fire, Oct 4 2018 - False Infrared; Right: Oct 9 2018 - False Infrared)</li>
</ul>
<p>If you want to share a NationalMap view with others simply select Share (in the top right).</p>
<p>To compare two map views with a slider, try using split view (<a href="https://www.youtube.com/watch?v=H3htpdYAE7w">video</a> tutorial). </p>
<p> </p>
<h2>About the service</h2>
<p>The satellites are run by the European Space Agency and contribute to the <a href="http://copernicus.eu/">Copernicus program</a>. Digital Earth Australia (DEA) manages the data loaded into the NationalMap; as part of Geoscience Australia they are a key contributor to the <a href="http://www.copernicus.gov.au/about-us">Copernicus Australasia</a> regional hub. </p>
<p>David Gavin, Director DEA Technologies, says the department is working with Australian Industry sectors to harness satellite-derived data to "deliver new insights into the Australian continent to the Australian public and grow the Australian economy".</p>
<p>Sentinel-2A was launched on 23 June 2015 while Sentinel-2B followed on 7 March 2017.</p>
<p>Future plans include opening the historical archive of Sentinel 2A and 2B observations to NationalMap, plus delivering the current Landsat 7 and 8 satellite data in near real-time. Both projects are in partnership with National Computational Infrastructure and expected to be available at the end of the year.</p>
<h3><br />Sentinel-2 facts</h3>
<p>These satellites possess the following capabilities:</p>
<ul>
<li>Systematic global coverage of land surfaces from 56° S to 84° N, coastal waters, all of the Mediterranean Sea and every island greater than 100 km2 </li>
<li>Revisits every 5 days under the same viewing angles which ensure that the level of sunlight is even. At high latitudes, Sentinel-2 can overlap some regions so they will be observed twice or more every 5 days, but with different viewing angles.</li>
<li>A spatial resolution of 10 m, 20 m and 60 m.</li>
<li>290 km field of view</li>
<li>A free and open data policy</li>
</ul>
<p>To track the position of the Sentinel-2 satellites in real-time, download the Sentinel App on <a href="https://itunes.apple.com/it/app/esa-sentinel/id1036738151?mt=8">iTunes</a> and <a href="https://play.google.com/store/apps/details?id=esa.sentinel&hl=it">Google</a>.</p>
<h2>Using multi-spectral data</h2>
<p>For further definition, users can choose from the styles menu to view multi-spectral data – 15 bands in the visible, near-infrared, and shortwave infrared part of the spectrum.</p>
<p>There are several views you can render, everything from Simple RGB to Spectral Bands and Normalised Indexes.</p>
<p>Find out more in the Communities of Practice: <a href="https://community.digital.gov.au/t/sentinel2-images-using-multi-spectral-data/935">Sentinel-2 images - using multi-spectral data</a></p>
<p> </p>
<h2>Discuss or ask questions</h2>
<p>We now have a communities page where you can ask questions and find the latest announcements about open data.</p>
<p><a href="https://community.digital.gov.au/c/open-data">Join the open data community</a></p>Stephen CanningsAlex gets a GovHack gong2018-09-20T00:00:00+00:002018-09-20T00:00:00+00:00/news-media/blog/alex-gets-govhack-gong<p><strong>From GovHack’s humble beginnings to a nation-wide annual event, few have been as closely connected as the DTA’s Alex Sadleir. </strong></p>
<p>This year, Alex received a Sydney ‘Spirit of GovHack’ award for team mentoring so we made him answer a few questions about what he does, and why. </p>
<p>If you didn't know, GovHack is an open data hackathon held across a weekend where thousands of competitors work in teams to solve real-world problems using government public data. The event is volunteer run, from state organisers through to the mentors.</p>
<p><strong>Alex, what does a GovHack mentor actually do?</strong></p>
<p>A mentor sits with each team and looks over their idea. I suggest how they could get that done, or if there's a better way to do it. We help them understand the tools and the data that's available to them.</p>
<p><strong>How did mentoring start in GovHack?</strong></p>
<p>Mentoring has always been a part of the GovHack experience. Originally, it happened organically. Government sponsors were coming along to present their dataset. They had the expertise with that dataset, and about working with data. People were tapping them on the shoulder, and over time it became a more formal thing to provide mentors at every location.</p>
<p><strong>I heard you won a prize at the first ever GovHack? </strong></p>
<p>In 2009 it was kind of a pilot event at my university in Canberra (Australian National University). The competition was associated with Mash Up Australia, and GovHack was set up to help people prepare for that bigger competition.</p>
<p>My team’s project ‘Lobby Lens’ was one of the award winners.</p>
<p>The prize at the time was a Meccano robot. Since then the prizes have become more, well … stylish.</p>
<p><strong>What was Lobby Lens about?</strong></p>
<p>It was about visualising the connections between political lobbyists, government contracts and government political donors.</p>
<p><strong>How were you involved in GovHack after that?</strong></p>
<p>GovHack started as a one-off, but they created this amazing brand with the GovHack logo created by the Twitter ‘fail whale’ designer, <a href="http://www.yiyinglu.com/sc/logo">Yiying Lu</a>. A couple of years later GovHack was rebooted by volunteers. This gave me the opportunity from the start to shape the event and be on its national committee. For a few years, I was also the technical director for the competition.</p>
<p><strong>Over time you must have seen a change in the kind of projects people were making?</strong></p>
<p>I think one of the things we tried to do is open it up. Because initially it was very heavily focused on app developers. </p>
<p>There's a wide range of ways to use government data like data journalism and creating art from government data. We've tried to encourage different uses over the years.</p>
<p>For example, this year I was helping <a href="https://2018.hackerspace.govhack.org/projects/placeholder_94">out a team</a> that made a 3D printed model of some data.</p>
<p>That's a bit different from the traditional, hackathon app-making.</p>
<p><br /><em>Alex Sadleir is a senior developer at the Digital Transformation Agency (DTA). The DTA is the lead agency sponsor for GovHack 2018.</em></p>
<p><em>Currently, the judges are reviewing 243 projects. This year’s winners of GovHack prizes will be announced at state and territory awards ceremonies in October, with a national Red Carpet Awards in November. </em><br /> </p>Stephen CanningsFrom GovHack’s humble beginnings to a nation-wide annual event, few have been as closely connected as the DTA’s Alex Sadleir. This year, Alex received a Sydney ‘Spirit of GovHack’ award for team mentoring so we made him answer a few questions about what he does, and why. If you didn't know, GovHack is an open data hackathon held across a weekend where thousands of competitors work in teams to solve real-world problems using government public data. The event is volunteer run, from state organisers through to the mentors. Alex, what does a GovHack mentor actually do? A mentor sits with each team and looks over their idea. I suggest how they could get that done, or if there's a better way to do it. We help them understand the tools and the data that's available to them. How did mentoring start in GovHack? Mentoring has always been a part of the GovHack experience. Originally, it happened organically. Government sponsors were coming along to present their dataset. They had the expertise with that dataset, and about working with data. People were tapping them on the shoulder, and over time it became a more formal thing to provide mentors at every location. I heard you won a prize at the first ever GovHack? In 2009 it was kind of a pilot event at my university in Canberra (Australian National University). The competition was associated with Mash Up Australia, and GovHack was set up to help people prepare for that bigger competition. My team’s project ‘Lobby Lens’ was one of the award winners. The prize at the time was a Meccano robot. Since then the prizes have become more, well … stylish. What was Lobby Lens about? It was about visualising the connections between political lobbyists, government contracts and government political donors. How were you involved in GovHack after that? GovHack started as a one-off, but they created this amazing brand with the GovHack logo created by the Twitter ‘fail whale’ designer, Yiying Lu. A couple of years later GovHack was rebooted by volunteers. This gave me the opportunity from the start to shape the event and be on its national committee. For a few years, I was also the technical director for the competition. Over time you must have seen a change in the kind of projects people were making? I think one of the things we tried to do is open it up. Because initially it was very heavily focused on app developers. There's a wide range of ways to use government data like data journalism and creating art from government data. We've tried to encourage different uses over the years. For example, this year I was helping out a team that made a 3D printed model of some data. That's a bit different from the traditional, hackathon app-making. Alex Sadleir is a senior developer at the Digital Transformation Agency (DTA). The DTA is the lead agency sponsor for GovHack 2018. Currently, the judges are reviewing 243 projects. This year’s winners of GovHack prizes will be announced at state and territory awards ceremonies in October, with a national Red Carpet Awards in November. They came, they coded … now they sleep2018-09-10T00:00:00+00:002018-09-10T00:00:00+00:00/news-media/blog/they-came-they-coded-now-they-sleep<p>They came with names like ‘Save Lives, Not Spreadsheets’, ‘A Nice Well Behaved Cheese’ and ‘I Adopted a Puppy Today But I’m Still Here Hacking’: teams with multiple skills from programming to design to video making. Some even tweeted their pets dressed in 'merch'. </p>
<p>Who were these crazy people? GovHackers, of course! The annual hackathon began at 7pm Friday 7 September and finished Sunday afternoon - 46 hours later. These teams readily consumed, along with pizza, loads of open government data. Sifting, mashing and rising up to the various challenges offered by agencies. </p>
<h2><br />DTA challenges</h2>
<p>This year, the Digital Transformation Agency (DTA) offered bounty challenges. They proved popular.</p>
<p>The 'Mix and Mashup' bounty was simple - best use of two or more data sets which seem to be completely unrelated to each other, but can be brought together to inform great solutions. 71 teams took up this challenge. </p>
<p>'More Than Maps and Apps' invited teams to help government decide with data, to help them make their big and small decisions. 59 teams responded to the brief. </p>
<p>These challenges were the most popular of the 105<a href="https://2018.hackerspace.govhack.org/challenges"> <span>challenges</span></a> offered.</p>
<p>As the lead agency for all things open data, the DTA supports its greater use – not only by government, but also in business and community. Each year, GovHack is a focal point for open data in Australia. The event helps to drive a cycle of improvement of open data and the services that unlock its value. And of course, the participation in GovHack provides a unique opportunity for people from a range of places, backgrounds and skill sets to come together and surprise us with their projects.</p>
<h2>Datasets and traffic</h2>
<p>Over the competition weekend,<a href="http://data.gov.au/"> data.gov.au</a> had a spike in traffic with 27,127 datasets viewed and 26,206 API requests made, from across the nation. Sydney was well out in front of other cities by for total Australian city traffic. But Mount Gambier in South Australia, when compared to population density, boasted the greatest number of hackers over the weekend.</p>
<p><img alt="Chart showing Mount Gambier as most engaged city during Hackathon, taking into account population" src="/assets/img/govhack-mtgambier.jpg" /></p>
<p>Of the datasets accessed, the ATO and IP Australia offered special Govhack 2018 datasets that were of keen interest (and related to their challenges). Getting fewer hits but worth noting were 78 hits on Aboriginal language word lists, and 16 hits on Tasmanian convicts asking permission to marry (1829-1857). It’s amazing what data is out there, and what historical data could be useful for present-day data-driven activities.</p>
<p><img alt="Bar chart showing most popular search terms with top results in order - Heath, data, NSW, Victoria, Australia, population, Melbourne, water, coast, postcode, transport" src="/assets/img/govhackterm_search_bar.jpg" /></p>
<h2><br />Judging and next steps</h2>
<p>The judges will soon begin to assess each and every team's project leading up to the GovHack award night in around six weeks.</p>
<p>In the meantime, catch up on the event and view the teams, challenges and projects, including project repositories and the datasets they used, all in <a href="https://2018.hackerspace.govhack.org/">Hackerspace</a>.</p>
<p>Thank you to everyone who made this event happen, the many mentors, supports and volunteers. The Open Data team was busy over the weekend providing support. They too are in need of a good sleep.</p>
<p> </p>
<p><img alt="" src="/assets/img/govhackimage.jpg" width="100%" /></p>Stephen CanningsThey came with names like ‘Save Lives, Not Spreadsheets’, ‘A Nice Well Behaved Cheese’ and ‘I Adopted a Puppy Today But I’m Still Here Hacking’: teams with multiple skills from programming to design to video making. Some even tweeted their pets dressed in 'merch'. Who were these crazy people? GovHackers, of course! The annual hackathon began at 7pm Friday 7 September and finished Sunday afternoon - 46 hours later. These teams readily consumed, along with pizza, loads of open government data. Sifting, mashing and rising up to the various challenges offered by agencies. DTA challenges This year, the Digital Transformation Agency (DTA) offered bounty challenges. They proved popular. The 'Mix and Mashup' bounty was simple - best use of two or more data sets which seem to be completely unrelated to each other, but can be brought together to inform great solutions. 71 teams took up this challenge. 'More Than Maps and Apps' invited teams to help government decide with data, to help them make their big and small decisions. 59 teams responded to the brief. These challenges were the most popular of the 105 challenges offered. As the lead agency for all things open data, the DTA supports its greater use – not only by government, but also in business and community. Each year, GovHack is a focal point for open data in Australia. The event helps to drive a cycle of improvement of open data and the services that unlock its value. And of course, the participation in GovHack provides a unique opportunity for people from a range of places, backgrounds and skill sets to come together and surprise us with their projects. Datasets and traffic Over the competition weekend, data.gov.au had a spike in traffic with 27,127 datasets viewed and 26,206 API requests made, from across the nation. Sydney was well out in front of other cities by for total Australian city traffic. But Mount Gambier in South Australia, when compared to population density, boasted the greatest number of hackers over the weekend. Of the datasets accessed, the ATO and IP Australia offered special Govhack 2018 datasets that were of keen interest (and related to their challenges). Getting fewer hits but worth noting were 78 hits on Aboriginal language word lists, and 16 hits on Tasmanian convicts asking permission to marry (1829-1857). It’s amazing what data is out there, and what historical data could be useful for present-day data-driven activities. Judging and next steps The judges will soon begin to assess each and every team's project leading up to the GovHack award night in around six weeks. In the meantime, catch up on the event and view the teams, challenges and projects, including project repositories and the datasets they used, all in Hackerspace. Thank you to everyone who made this event happen, the many mentors, supports and volunteers. The Open Data team was busy over the weekend providing support. They too are in need of a good sleep. WofG Web Reporting Service - Crawl Data2018-09-07T00:00:00+00:002018-09-07T00:00:00+00:00/news-media/blog/wofg-web-reporting-service-crawl-data<p>The Digital Transformation Agency (DTA) works to deliver better, faster, simpler digital services. As part of our charter, we’ve undertaken work to examine the adoption of DTA’s guidance, products and services by agencies on publicly-facing Australian Government websites.</p>
<p>This service is in an early beta stage, and we’ve recently gathered over 5 million URLs as part of this exercise.</p>
<h2>The dataset</h2>
<p>The DTA has published the <a href="https://data.gov.au/dataset/whole-of-australian-government-web-crawl/?utm_source=blog&utm_medium=web&utm_campaign=gh">Whole-of-Australian Government Web Crawl dataset</a> on data.gov.au as Web ARChive (WARC) files, in parts and whole.</p>
<p>The dataset is large - the largest so far on data.gov.au. We’ve made it available both as a single 66GB WARC file, and as a series, split into 57 smaller WARC files. We’d suggest you download a <a href="https://data.gov.au/dataset/whole-of-australian-government-web-crawl/resource/5e0f3ec0-e6b0-4584-9633-307c0c978668/?utm_source=blog&utm_medium=web&utm_campaign=gh">smaller WARC file</a> (approx. 1.1GB) as a sample first.</p>
<p>WARC files are a recognised <a href="https://www.iso.org/standard/68004.html">ISO standard</a> for web archiving. We’re planning to filter these WARC files further to examine metrics like:</p>
<ul>
<li>Size (number of domains, websites and URLs published by a given agency or portfolio, seeded by the <a href="https://data.gov.au/dataset/australian-government-organisations-register/">Australian Government Organisations Register</a>)</li>
<li>Technology (examining whether agencies may be leveraging whole-of-government platforms, services and products, like <a href="https://www.govcms.gov.au">GovCMS</a>, <a href="https://beta.dta.gov.au/our-projects/google-analytics-government">GA360</a> and <a href="https://designsystem.gov.au">Design System</a>)</li>
<li>Quality (examining whether agencies are applying guidance from the <a href="https://guides.service.gov.au/content-guide/">Content Guide</a> or the <a href="https://www.dta.gov.au/what-we-do/policies-and-programs/style-manual/">Style manual</a>)</li>
<li>Accessibility (examining readability levels, machine-checkable portions of WCAG 2.1, use of CAPTCHA, and non-English usage)</li>
<li>Usage (to ensure, where feasible, any efforts spent on improvements are focused on the most-frequently-used content)</li>
</ul>
<p>Once filtered, each URL’s metadata will be injected as JSON into an ElasticSearch stack, visualised by Kibana. This will allow DTA’s policy and product owners to visualise and explore a large, complex time series of reporting metrics in a browser-based reporting environment, ensuring changes to products and policies are backed by sound, reproducible evidence.</p>
<h2><img alt="Flow diagram indicating current state of service (1-4) and next steps (5-9)" src="/assets/img/wogprocessgraph-750-1-4.png" width="100%" />Future plans</h2>
<p>We’re planning to conduct three more crawls - one every 90 days - examining environmental changes over time.</p>
<p>There’s likely to be several other uses for this raw WARC store - in discussions with research organisations and universities, we’re anticipating that this snapshot will be used for reporting on Linked Data usage, the generation of government ontologies, and informing corpus work for Australian dictionaries.</p>
<p>As always, we’d love to hear your comments and feedback on this dataset - feel free to join the discussion at <a href="https://community.digital.gov.au/t/whole-of-australian-government-web-crawl-dataset/868">Open Data’s communities of practice</a>.</p>
<p><em>Gordon Grace is the product owner for the Whole-of-Australian Government Web Reporting Service at the Digital Transformation Agency.</em></p>Gordon GraceThe Digital Transformation Agency (DTA) works to deliver better, faster, simpler digital services. As part of our charter, we’ve undertaken work to examine the adoption of DTA’s guidance, products and services by agencies on publicly-facing Australian Government websites. This service is in an early beta stage, and we’ve recently gathered over 5 million URLs as part of this exercise. The dataset The DTA has published the Whole-of-Australian Government Web Crawl dataset on data.gov.au as Web ARChive (WARC) files, in parts and whole. The dataset is large - the largest so far on data.gov.au. We’ve made it available both as a single 66GB WARC file, and as a series, split into 57 smaller WARC files. We’d suggest you download a smaller WARC file (approx. 1.1GB) as a sample first. WARC files are a recognised ISO standard for web archiving. We’re planning to filter these WARC files further to examine metrics like: Size (number of domains, websites and URLs published by a given agency or portfolio, seeded by the Australian Government Organisations Register) Technology (examining whether agencies may be leveraging whole-of-government platforms, services and products, like GovCMS, GA360 and Design System) Quality (examining whether agencies are applying guidance from the Content Guide or the Style manual) Accessibility (examining readability levels, machine-checkable portions of WCAG 2.1, use of CAPTCHA, and non-English usage) Usage (to ensure, where feasible, any efforts spent on improvements are focused on the most-frequently-used content) Once filtered, each URL’s metadata will be injected as JSON into an ElasticSearch stack, visualised by Kibana. This will allow DTA’s policy and product owners to visualise and explore a large, complex time series of reporting metrics in a browser-based reporting environment, ensuring changes to products and policies are backed by sound, reproducible evidence. Future plans We’re planning to conduct three more crawls - one every 90 days - examining environmental changes over time. There’s likely to be several other uses for this raw WARC store - in discussions with research organisations and universities, we’re anticipating that this snapshot will be used for reporting on Linked Data usage, the generation of government ontologies, and informing corpus work for Australian dictionaries. As always, we’d love to hear your comments and feedback on this dataset - feel free to join the discussion at Open Data’s communities of practice. Gordon Grace is the product owner for the Whole-of-Australian Government Web Reporting Service at the Digital Transformation Agency.August 2018 Release of G-NAF and Administrative Boundaries now available2018-08-28T00:00:00+00:002018-08-28T00:00:00+00:00/news-media/blog/august-2018-release-g-naf-and-administrative-boundaries-now-available<p>The August 2018 release of the Geocoded National Address File (G-NAF) and Administrative Boundaries datasets are now available for download. <br />• <a href="https://data.gov.au/dataset/geocoded-national-address-file-g-naf">PSMA Geocoded National Address File (G-NAF)</a><br />• <a href="https://data.gov.au/dataset/psma-administrative-boundaries">PSMA Administrative Boundaries</a></p>
<p>Please note the product update: Data model changes improve G-NAF product scope <a href="https://bit.ly/2Paq6hX">https://bit.ly/2Paq6hX</a><br />We encourage users of the data to update to the latest versions.<br />Please forward any queries about the data to <a href="mailto:spatial@industry.gov.au" class="spamspan">spatial@industry.gov.au</a></p>Ann BeaumarisThe August 2018 release of the Geocoded National Address File (G-NAF) and Administrative Boundaries datasets are now available for download. • PSMA Geocoded National Address File (G-NAF)• PSMA Administrative Boundaries Please note the product update: Data model changes improve G-NAF product scope https://bit.ly/2Paq6hXWe encourage users of the data to update to the latest versions.Please forward any queries about the data to spatial@industry.gov.auFirst Open Data Meet-up report2018-08-22T00:00:00+00:002018-08-22T00:00:00+00:00/news-media/blog/first-open-data-meet-report<p><img alt="A montage of presenters and attendees at the meet-up, chatting afterwards, presenting and watching." src="/assets/img/opendata-montage750-2.jpg" width="100%" /></p>
<p><br /><strong>Our first open data meet-up was a chance for some 300 people to hear what others were doing and what was on their data-filled minds.</strong></p>
<p>One theme emerged - the perennial question of what is ‘good’ data? We heard about ways to measure the quality and quantity of data and how to make it meaningful and available.</p>
<h2><br />Linking data</h2>
<p>Experimental scientist Nicholas Car spoke about Linked Data and explained that it’s a specific type of data that can be linked to and from other data. Nicholas demonstrated linking to the<a href="https://data.gov.au/dataset/geocoded-national-address-file-g-naf"> G-NAF</a> (a dataset containing all physical addresses in Australia) so web and smart clients could make use of it by not only including addresses but coordinates and other information.</p>
<p>As part of the Australian Government Linked Data Working Group, Nicholas is working towards an integrated set of concepts and categories to assist developers of whole-of-government models. There are several he said in development made by different agencies through different lenses. The working group wants to create a methodology to integrate models to connect with each other, which he said was a world first.</p>
<p>Anyone from public or private can join the <a href="http://www.linked.data.gov.au/">Linked Data Working Group</a>.</p>
<h2><br />Making it ‘easier to find’</h2>
<p>Alex Gilleran is the CSIRO’s Data 61 tech lead for MAGDA (Making Australian Government Data Available) working in partnership with the Digital Transformation Agency (DTA). He shared the discovery and development process his team has experienced.</p>
<p>One goal of the project was to make search and discovery of the existing 70,000 datasets on<a href="http://data.gov.au"> data.gov.au</a> ‘easier to find’.</p>
<p>He conceded there was no chance of ‘pushing back on data custodians to repair the million different ways’ to write metadata. So instead the team invested in the philosophy of cleaning up existing metadata to make it work with their system.</p>
<p>Search was also improved with a stretch goal to make the search experience more Google-like, with more meaningful results based on the users search terms. Visualising data is also now part of the user journey and not an additional step requiring a separate application.</p>
<p>The results are in beta and you can use it now at<a href="http://search.data.gov.au"> search.data.gov.au</a>.</p>
<p>In future, Alex said they wanted to be more prescriptive about uploading data with better formats and consistent metadata. He floated the idea of agencies using an in-house instance of MAGDA.</p>
<p>Data in MAGDA is now also ranked on a scale, based on World Wide Web founder Tim Berners-Lee’s <a href="https://www.w3.org/2011/gld/wiki/5_Star_Linked_Data">5-star linked data system</a>.</p>
<h2><br />Ranking data</h2>
<p>Louie Jasek and Jamie Leach demonstrated an alternative system to rate open data quality based on The Open Data Institute’s (ODI) Certificate system, which on the first look uses badges rather than stars.</p>
<p>As the new open data manager in the Queensland Government, Louie was focusing on data quality and the different meanings of quality. He compared Berner-Lees on the one hand, of machine readability, open licences and linked data, with an entirely alternative meaning of quality on the other hand, of timeliness, completeness and accuracy.</p>
<p>A key challenge he says is having a shared understanding of the concept of ‘data quality’. And the focus on quantity over quality is also an issue, he reflected, ’what is next after you publish a dataset, what can be improved?’</p>
<p>Jamie championed the Open Data Certificates, set up by the UK Government but now being adopted and adapted in Queensland.</p>
<p>Data custodians can investigate the <a href="http://certificates.theodi.org/en/">certificates at ODI</a> now. An interesting feature is that it can be applied to a single dataset or even to an entire agency level portal. The certificates are partly automated but a form allows publishers to add information.</p>
<p> </p>
<h2>So how much is there?</h2>
<p>The question of quantity was on the mind of CSIRO data scientist Jonathan Yu. He shared his research findings of the quantity of both open and research data in Australia.</p>
<p>Last year, Australia lost the number one spot in the <a href="https://index.okfn.org/place/">Global Open Data Index survey</a> (coming second to Taiwan); Jonathan argues the metrics used are limited because they are criteria-based and excluded a fuller understanding of volume, velocity and variety.</p>
<p>So how much data? The numbers are as follows (as at October 2017, for Australia):</p>
<ul>
<li>
<p>Datasets: 125,819</p>
</li>
<li>
<p>Files: 10,038,968</p>
</li>
<li>
<p>Formats, unique: 1,548 (including variations of descriptions, like .xl and .xls)</p>
</li>
<li>
<p>Keywords: 667,967</p>
</li>
</ul>
<p>The volume of open government data is approximately 1.7TB.</p>
<p>However, that is overshadowed like a coconut next to a coconut palm when you include research data. Comparing the publicly offered data, research is some 700 times more than government data. The volume of open research data is approximately 944TB. And this figure excludes astronomy data published by the CSIRO. Jonathon joked that was ‘unfair’ to include as space data was collected continuously.</p>
<p>The presented quantitative survey of open data in Australia is a starting point, Jonathan says, and adds that ‘we can’t manage what we don’t measure’.</p>
<h2>Fellowships for public servants</h2>
<p>As a recent alumnus, Patrick Drake-Brockman shared back his experience in the Data Fellowship Program. The fellowship provides public servants with 3 months of advanced data training. Successful applicants work with a mentor from Data61 on a self-directed project.</p>
<p>Fellows can submit a project related to their agency. Successful submissions may have some of the following qualities: a project that can be developed within the time frame; can be applied to other agencies or upscaled to other levels of government; plus the applicable skills and experience of both the candidate and the mentor.</p>
<p>The latest round closed on 3 August but don’t despair, you can find out more about the Digital Transformation Agency’s (DTA) <a href="https://beta.dta.gov.au/help-and-advice/learning-and-development/data-fellowship-program">data fellowship program</a> and future rounds.</p>
<p>Patrick’s fellowship project was a network analysis of government procurement.</p>
<p>‘A contract between a customer and a vendor’, he says, ‘can be viewed as an edge linking two nodes in a network’. And once you can build that network you can analyse it.</p>
<p>Patrick hypothesised how a policy change could affect procurement, based on past patterns. Working with his mentor, it was clear new learnings were discovered and insights gained that will help his team better understand procurement policy in Australia.</p>
<p>The meet-up was held across every capital city and online. In Canberra, the presentations concluded with refreshments and plenty of conversation, allowing open data practitioners from various agencies to meet-up. </p>
<p>Recordings of our speakers and their presentations can be found on the <a href="https://community.digital.gov.au/t/thanks-for-coming-government-open-data-meet-up/677">open data communities of practice</a>.</p>
<p>Join our <a href="https://dta.us12.list-manage.com/subscribe/post?u=81bbb1d15242b2224ee11e3fe&id=6b385092d3">mailing list</a> for upcoming events and more.</p>
<p><em>The Digital Transformation Agency presented the event, with thanks to Marita Baier-Gorman and Gordon Williamson of the DTA’s open data team, and thank you to the Australian Bureau of Statistics for hosting.</em></p>
<p> </p>Stephen CanningsOur first open data meet-up was a chance for some 300 people to hear what others were doing and what was on their data-filled minds. One theme emerged - the perennial question of what is ‘good’ data? We heard about ways to measure the quality and quantity of data and how to make it meaningful and available. Linking data Experimental scientist Nicholas Car spoke about Linked Data and explained that it’s a specific type of data that can be linked to and from other data. Nicholas demonstrated linking to the G-NAF (a dataset containing all physical addresses in Australia) so web and smart clients could make use of it by not only including addresses but coordinates and other information. As part of the Australian Government Linked Data Working Group, Nicholas is working towards an integrated set of concepts and categories to assist developers of whole-of-government models. There are several he said in development made by different agencies through different lenses. The working group wants to create a methodology to integrate models to connect with each other, which he said was a world first. Anyone from public or private can join the Linked Data Working Group. Making it ‘easier to find’ Alex Gilleran is the CSIRO’s Data 61 tech lead for MAGDA (Making Australian Government Data Available) working in partnership with the Digital Transformation Agency (DTA). He shared the discovery and development process his team has experienced. One goal of the project was to make search and discovery of the existing 70,000 datasets on data.gov.au ‘easier to find’. He conceded there was no chance of ‘pushing back on data custodians to repair the million different ways’ to write metadata. So instead the team invested in the philosophy of cleaning up existing metadata to make it work with their system. Search was also improved with a stretch goal to make the search experience more Google-like, with more meaningful results based on the users search terms. Visualising data is also now part of the user journey and not an additional step requiring a separate application. The results are in beta and you can use it now at search.data.gov.au. In future, Alex said they wanted to be more prescriptive about uploading data with better formats and consistent metadata. He floated the idea of agencies using an in-house instance of MAGDA. Data in MAGDA is now also ranked on a scale, based on World Wide Web founder Tim Berners-Lee’s 5-star linked data system. Ranking data Louie Jasek and Jamie Leach demonstrated an alternative system to rate open data quality based on The Open Data Institute’s (ODI) Certificate system, which on the first look uses badges rather than stars. As the new open data manager in the Queensland Government, Louie was focusing on data quality and the different meanings of quality. He compared Berner-Lees on the one hand, of machine readability, open licences and linked data, with an entirely alternative meaning of quality on the other hand, of timeliness, completeness and accuracy. A key challenge he says is having a shared understanding of the concept of ‘data quality’. And the focus on quantity over quality is also an issue, he reflected, ’what is next after you publish a dataset, what can be improved?’ Jamie championed the Open Data Certificates, set up by the UK Government but now being adopted and adapted in Queensland. Data custodians can investigate the certificates at ODI now. An interesting feature is that it can be applied to a single dataset or even to an entire agency level portal. The certificates are partly automated but a form allows publishers to add information. So how much is there? The question of quantity was on the mind of CSIRO data scientist Jonathan Yu. He shared his research findings of the quantity of both open and research data in Australia. Last year, Australia lost the number one spot in the Global Open Data Index survey (coming second to Taiwan); Jonathan argues the metrics used are limited because they are criteria-based and excluded a fuller understanding of volume, velocity and variety. So how much data? The numbers are as follows (as at October 2017, for Australia): Datasets: 125,819 Files: 10,038,968 Formats, unique: 1,548 (including variations of descriptions, like .xl and .xls) Keywords: 667,967 The volume of open government data is approximately 1.7TB. However, that is overshadowed like a coconut next to a coconut palm when you include research data. Comparing the publicly offered data, research is some 700 times more than government data. The volume of open research data is approximately 944TB. And this figure excludes astronomy data published by the CSIRO. Jonathon joked that was ‘unfair’ to include as space data was collected continuously. The presented quantitative survey of open data in Australia is a starting point, Jonathan says, and adds that ‘we can’t manage what we don’t measure’. Fellowships for public servants As a recent alumnus, Patrick Drake-Brockman shared back his experience in the Data Fellowship Program. The fellowship provides public servants with 3 months of advanced data training. Successful applicants work with a mentor from Data61 on a self-directed project. Fellows can submit a project related to their agency. Successful submissions may have some of the following qualities: a project that can be developed within the time frame; can be applied to other agencies or upscaled to other levels of government; plus the applicable skills and experience of both the candidate and the mentor. The latest round closed on 3 August but don’t despair, you can find out more about the Digital Transformation Agency’s (DTA) data fellowship program and future rounds. Patrick’s fellowship project was a network analysis of government procurement. ‘A contract between a customer and a vendor’, he says, ‘can be viewed as an edge linking two nodes in a network’. And once you can build that network you can analyse it. Patrick hypothesised how a policy change could affect procurement, based on past patterns. Working with his mentor, it was clear new learnings were discovered and insights gained that will help his team better understand procurement policy in Australia. The meet-up was held across every capital city and online. In Canberra, the presentations concluded with refreshments and plenty of conversation, allowing open data practitioners from various agencies to meet-up. Recordings of our speakers and their presentations can be found on the open data communities of practice. Join our mailing list for upcoming events and more. The Digital Transformation Agency presented the event, with thanks to Marita Baier-Gorman and Gordon Williamson of the DTA’s open data team, and thank you to the Australian Bureau of Statistics for hosting.