Improving Citywide Compliance with the Open Data Law
Over the past year, the Open Data team has taken steps to improve compliance with the Open Data Law by developing new policy and tools for complying with open data standards.
Better tracking and performance management
In December 2017, Local Laws 244 and 251 of 2017 were ratified, extending the duration of the Open Data Law and creating new annual reporting requirements. For the first time this year, the Open Data team is reporting, in one comprehensive tracker (“Published Asset Inventory), a full inventory of all datasets on the platform. Each dataset now contains metadata on its scheduled and actual publication date, and whether automation is feasible among other metadata. Compiling this information in one place has given the Open Data team, ODCs, as well as the public, a better view of agency compliance with data standards, timely publication, and updates to datasets—meaning a more effective mechanism for holding agencies accountable.
The Open Data team has also developed a compliance dashboard that visualizes metrics for the City’s compliance with applicable local laws. This dashboard is available on the Open Data Website and contains the following metrics, which can be viewed either for specific agencies or for the program at large:
Datasets with Data Dictionaries: Percentage of data assets with data dictionaries. Local Law 107 of 2015 requires every dataset on Open Data to have a plain language data dictionary.
Datasets Published Within Two Weeks of Target: Of datasets published in the last 365 days, percentage of data assets published within two weeks of their scheduled publication date.
Feasibly Automated Datasets: Of data assets reported by agencies to be feasibly automated, percentage that are currently automated. Automated datasets are published automatically from their source systems.
Geocoded Datasets: Of data assets eligible for the geospatial address standard, percentage of assets that have been geocoded. Local Law 108 of 2015 requires that every dataset containing street addresses also contain coordinates and political boundaries.
Data Retention Standard: Percentage of assets that do not replace records when they are updated. Local Law 106 of 2015 requires data updates to not permanently overwrite records. Note that for some datasets the “No” category may be compliant with Local Law 106 through another archiving process.
Released as a beta in November 2018, the dashboard will undergo user testing in December and January to optimize its effectiveness as a tool for public information. Users are additionally invited to submit their feedback on the dashboard by January 31, 2019 through the “Contact Us” page on the Open data website. The dashboard is updated daily, and can be viewed online here: https://opendata.cityofnewyork.us/dataset-compliance-dashboard/
Policy improvements
Removing a Dataset from the Portal: A process and guidelines on removing an existing data asset from the Open Data Portal.
MODA-DOITT-DORIS MOU: Through a memorandum of understanding between MODA, DoITT, and the Department of Records and Information Services (DORIS), agencies can meet legislated data reporting requirements by maintaining datasets in a machine-readable form on the Open Data Portal, rather than transmitting PDFs of tabular data to the municipal library.
New resources for Open Data Coordinators
Open Data Collections: A Data Collection is a dataset release format consisting of multiple datasets related to each other through primary and foreign keys. City data is often stored in relational databases. The Open Data Collection allows agencies to publish related datasets.
Annual Compliance Plan Reporting Playbook: Detailed guidance for the 2018 Annual Compliance Plan Reporting Process.
Upgraded compliance reporting workbook: A new workbook for reporting on agency compliance, included additional steps to help improve documentation of data owners within agency lines of business, which will help the Open Data team help coordinate transition when ODCs turnover.
Open Data Coordinator recruitment, convenings, and training
After recruiting ODCs for every agency, pursuant to Local Law 251 of 2017, the Open Data team partnered in March 2018 with Socrata, BetaNYC, the Sunlight Foundation, BuildWith and the Department of Citywide Administrative Services to host the first full-day training for NYC ODCs, providing expert insight on best practices for data publishing and civic engagement. Training materials can be found on the Open Data Coordinators resource page. In addition, the Open Data team has continued to host Open Data Coordinator convenings and onboard additional agencies to respond to public inquiries directly on the Open Data Help Desk.
Survey of Open Data Coordinators
An annual survey of ODCs helps the Open Data team understand how well the compliance reporting process works for agency ODCs, and where there are additional opportunities for improving the compliance and training process. Insights include:
40% of the 43 respondents were completing annual compliance reporting for the first time
56% were very happy with how their agencies were represented in the final report
Half of the respondents spent 10 or more hours working on compliance reporting for their agencies
Last updated