Why do Some Dashboards Succeed While Others Fail?

At a recent Gov 2.0 Summit, Ellen Miller, a recipient of prestigious technology accolades and an advocate for Open Government who promotes the use of technology to increase transparency, said that the data in the USASpending.gov dashboard “is almost useless”. Miller stated that some numbers on this dashboard are “over reporting, under reporting, and not reporting”*, while other numbers are reported late and/or reported incompletely. She is the founder and executive director of the Sunlight Foundation which backs her assertions with detailed analysis, and has harsh words for other data dashboards show-cased by the Obama administration.

Paypal

Help us to keep bringing you this high-quality content. Please donate today!

Some industry experts are pointing out that the recent rapid expansion of data dashboards has often provided limited utility. Yet we have seen interactive dashboards that have transformed the practices of technology groups, effectively engaged the American public in national transitions and dramatically increased operational effectiveness.

What is a Data Dashboard?

Before we examine WHY some dashboards succeed while others struggle or fail, let us define what we mean by data dashboards. A dashboard is not just a report. A dashboard enables meaningful inspection of a complex system to support decision-making. The complex system can be a portfolio, a business unit, a government agency or an industry.

For example, an enterprise project/grant/asset portfolio dashboard enables you to track and ACT on status, metrics, risks, and operate more consistently and predictably. A business unit dashboard can tell you who is buying your products, and if it’s a market segment that fits your brand. A government agency dashboard can dialog with the public; coordinate congressional, media and grassroots outreach; facilitate targeted training; and guide the roll-out of America’s transition to digital television. A golf course management financial dashboard can improve its agility in executing mergers and acquisitions.

Data dashboards tend to contain quantitative and qualitative data. They can be updated on an as-needed basis or in real-time, and include graphics and text. Effective dashboards are interactive. They communicate a business reality in the clearest way possible, just what is needed for action, using the appropriate vocabulary and level of precision. These dashboards progressively reveal data based on user interest via drill-downs, and provide different views for different perspectives and audiences. They draw attention to the most urgent information and metrics and urge you to act on what you see.

*”Sunlight’s Miller Takes a Dim View of Obama’s Transparency” by Nancy Scola, Sept 7, 2010: http://sunlightfoundation.com/clearspending/

Data dashboards not only deliver transparency and effective communication, but can go BEYOND integration and aggregation of heterogeneous operational data and business intelligence (BI). Indeed, dashboards can allow government to crowd-source and shape public policy, and enable the commercial sector to operate more efficiently and consistently. Dashboards can be tools for transformation.

However, data dashboards can also miss the mark, and not achieve their business objectives for many reasons. In particular, a non-intuitive user experience (UX), un-trusted data, displaying the wrong key performance indicators (KPIs), a flawed design, over-emphasis on platform and infrastructure, minimal attention to user readiness for change, or lack of strategy to address dashboard adoption are some factors that can result in low interaction with your dashboard, as well as low usage which leads to questionable benefits.

Dashboard Roles
Data dashboards are primarily created to support: strategic initiatives, analysis and self-service decision-making and operational and business effectiveness. Strategic dashboards tend to help organizations improve alignment with strategic goals, collaboration and coordination. These dashboards often have simple display mechanisms and are mostly designed for executives. Analytical dashboards support data analysis, feature more complicated display media and frequently provide richer context that includes comparisons and history. The analytics (Operations Research, digging into the data) frequently happens prior to the dashboard being built. Results of analytics can produce and highlight important KPIs that are displayed on dashboards for executives to consume. Operational dashboards help users monitor operations, understand information at a glance and are mainly designed to identify exceptions that require attention.

To deliver a dashboard solution that offers a compelling user experience, it is often necessary to blend a team of BI, Operations Research, UX, and change management professionals. Lack of subject matter expertise and leadership often results in low-usage dashboards that fail to meet their business objectives.

According to industry experts, one example of a data dashboard that did not meet usability objectives was the US Patent and Trademark Office (USPTO) dashboard. This dashboard was intended to allow the public to monitor data related to new patents issued. Its design however forced users to scroll through seven screens and navigate between multiple screens, making it difficult to make comparisons and see relationships between the variables. See http://www.perceptualedge.com/blog.

What then can be done to ensure the success of a dashboard?

Our experience with data dashboards that have met their business objectives indicate that there are at least five critical success factors to creating effective dashboards:

Clearly define the problem to be solved
Understand your data constraints
Define your key performance indicators (KPIs) and your UX
Assess your change management readiness
Ensure “just enough” infrastructure to support your dashboard.

What follows can help you prepare, build, deploy and sustain more successful dashboards.

Clearly Define the Problem to be Solved

Dashboards that try to be all things to all people often fail. First, define the key business objectives of your dashboard in specific and measurable terms. For example: to transition 100% of the American public from analog to digital TV by a specified date; or to provide detailed guidance to advocacy organizations, end users, and call centers for a successful TV transition.

Understand the expected benefits of your dashboard. Are you inviting public participation to help shape policy, or to communicate a major change? Or are you looking to improve efficiency, governance, and compliance? Do you want transparency to foster accountability? Or will your dashboard help institute new practices and be a coaching and mentoring tool?

By clearly defining your business objectives, you will be able to use these quantified objectives to DRIVE the requirements and scope for your dashboard project. This is fundamental to creating an effective dashboard. The design and information architecture of the dashboard depend on your business objectives. This will help you include only the “right” essential information and actionable data on your screens, knowing that the actions to be taken are intended to deliver the benefits you are seeking.

For example, in 2009 the US Congress required a national transition from analog to digital broadcasting. With the vast majority of the US public uninformed and ill-equipped for the transition, the responsible federal agency’s help desks were flooded with calls from confused and worried citizens. Further complicating the transition, service providers kept customer data private, preventing this agency from modeling supply or demand challenges. The agency created an interactive dashboard specific for this transition. It was called DTV.gov.

This dashboard integrated data from multiple sources such as national broadcasters, and crowd-sourced feedback from the public. It had a rich and intuitive user interface. But behind the scene, this dashboard required complicated analytical transformations, integration with other systems for key data, and authoring tools that enabled business users to keep the site content updated at all times. This solution enabled this agency’s staff to mobilize hundreds of grass roots support organizations, Congressional staff, and millions of US households by providing accurate and timely information on:

Digital conversion penetration by precise geographic boundary
Localized transmission simulations for use by the public, and
Education and outreach material for advocacy organizations, end users, and call centers.

On June 12, 2009, the transition date, 11,744,906 DTV.gov page views were recorded, a record in one day for this agency. More importantly, Americans transitioned to digital TV without incident, prompting the media to applaud the responsible agency for the successful conversion.

Clearly understanding the problems to be solved and business objectives enabled the DTV.gov dashboard to deliver a successful experience to all stakeholders.

Understand your Data constraints

Once the problem is well-defined, you will want to look at your data constraints. Where are your data sources and operational systems, and do they have the needed data for your dashboard? Are there data quality, standardization, and synchronization issues? Is there a common vocabulary to describe your business or mission? Is there a data model?

Knowing your data and associated issues is critical. Does your organization(s) generate the raw data to accurately feed your dashboard? Can you easily get to this data? If not, can you easily change the way you do business to obtain that data? If the data is trust-worthy, it is likely that your dashboard will be more widely used and will deliver the benefits you desire. If getting the data requires a business process change, then you are looking at a change management initiative in addition to a dashboard project.

For example, a leading golf course management (GCM) company operated nearly 120 courses and clubs in 26 states. Its business was facing growth and agility complexity. GCM’s inability to readily aggregate information led to a high-cost mergers and acquisition business model. GCM headquarters and its golf courses depended on complex spreadsheets for operations. Accountants often spent more time creating reports than doing accounting. GCM tackled their data constraints head-on: they eliminated stovepipes, improved workflows, reduced errors, implemented scalable enterprise architecture, and created a data model. They built a financial dashboard that provided an integrated view of up-to-date key GCM data to its stakeholders.

GCM now uses its dashboard to slice and dice data previously hard to glean, for forecasting, trending analysis, and timely decision-making. This interactive dashboard offers real-time access to critical business performance indicators, to improve the speed and effectiveness of GCM operations, and to enable its growth. GCM accountants can now focus on accounting, each accountant able to administer 25 golf courses instead of 8. All this enabled GCM to reduce cost, improve profits, and deliver better service via golf course general managers with more time to provide coaching to golfing customers.

Addressing its data constraints was a critical success factor to the GCM’s data dashboard.

Define your key Performance Indicators (KPIs) and UX

Beyond defining your problem and understanding your data, you will want to decide which KPIs and what UX will help improve your organization’s performance. Who are your dashboard users and customers? What information will increase their effectiveness or help you achieve your objectives? Do you know which KPIs to include in your dashboard?

If the KPIs displayed are not actionable, your dashboard will likely be less useful, and may be ignored. According to the CIO Weblog and Michael Krigsman of ZDNet’s IT Project Failures blog, many dashboard creators do not understand the information that users need, so “present the wrong data or the right data in the wrong format.” Conversely, if you are clear on who the users and stakeholders are for your dashboard, and you pick a SMALL but critical set of KPIs to display that they care about, then the results you achieve could be transformative.

For example, a large regulator of the financial industry created a project portfolio dashboard during a period of rapid growth. The key business objectives for this dashboard were: to provide 100% project transparency for enterprise communication and decision making ; to drive full consistency of practices for project management and status reporting; and to make project outcomes predictable.

The key stakeholders for this dashboard included IT senior management; business project sponsors, teams, and stakeholders; IT project managers and teams; and the Internal Audit Department. They needed information about project scope, schedule, budget, staff, risks and dependencies, defect trend and status. They wanted to maximize the likelihood that new functionality was delivered on-time, with quality, within budget and meeting expectations.

The at-a-glance project summary page showed four KPIs: red/yellow/green indicators for schedule, budget, and risk; and a completion ratio that provided the current software iteration vs the total number of iterations in the software release. The drill-down to each project showed detailed schedule variance, budget variance, risk mitigation and status, staffing plan and utilization, and development metrics. The UX was intuitive and simple and did not require scrolling. Feeding this dashboard, and getting information from it, was easy and fast.

The KPIs and information displayed were actionable, and of high interest to the stakeholders of this dashboard. Within six months all IT staff in the enterprise used this dashboard to report progress. Within another year, the business stakeholders also adopted the use of this dashboard. The leaders of this organization used this dashboard as a coaching tool, and project management practices became more consistent. The project success rate for this organization was 100% during their growth period of several years that included a complex merger.

Constantly measuring our dashboard’s use in daily decision making is essential. Build some KPIs, see if they are useful. If not, change them. Avoid building dozens of KPIs from the start and taking on a maintenance burden.

Assess your change management readiness

Are the stakeholders in your organization ready to accept a culture of transparency? Will they embrace a project tracking dashboard that could shine an unwelcomed light on their performance? Will this change be perceived as “threatening”? If the readiness is low, how will you support your users, train and mentor them for effective adoption of your dashboard? How will you troubleshoot for non-compliance? Is there a support team that is savvy with the problem and solution?

These questions, if ignored, could lead to slow adoption, or a slowly abandoned dashboard. Many dashboards implemented in the last 5 years are generally viewed as failed projects due to lack of widespread interaction and use. Under-estimating what it takes to promote effective adoption and maintain active usage of dashboards often results in incomplete, inaccurate, or untimely data in dashboards. This in turn leads to low usage of the dashboards.

According to Vivek Kundra, the US national CIO, when the Washington Post reported that some agency CIOs grumbled about the workload of maintaining the data in the USA Spending.gov dashboard, he talked to the CIO Council and “saw the data change overnight. It was cleaned up immediately when people realized it was going to be made public.”

Make sure you assess your organization’s readiness for change, and estimate the investment required to make this change. Do you have the budget and time for this change? If not, recognize your constraints and scale back your expectations to what you can accomplish.

For example, at the large regulator of the financial industry, the dashboard support team consisted of 2.5 FTEs to support approximately 150 projects and 40 project managers. Each support team member played multiple roles. The dashboard sponsor served as a business executive, data analyst, best practice coach, and compliance monitor. The dashboard developer was the technical lead and responsible for data architecture and modeling, dashboard design and UX, release management and trouble-shooting. The dashboard change agent led the prototyping and adoption, served as the project management subject matter expert, the dashboard product manager, trainer, promoter and mentor.

This support team worked tirelessly to continuously improve the quality and timeliness of the data. They gathered and embraced feedback from stakeholders from the start of this project, and involved the users throughout the dashboard prototyping, design, beta test, and rollout. They knew that if the solution was not intuitive and easy-to-use, the dashboard would not be readily adopted. They chose platform, tools and interfaces already familiar to their constituents in order to minimize the need for training. They offered online HELP and user’s guide. They made sure that the data came from operational sources, and did not dictate another reporting requirement. They fully integrated the project dashboard with the tools for requirements management, testing and configuration management so that development metrics would be readily available.

After software release, this dashboard support team encouraged feedback through change requests. They conducted bi-monthly change control board sessions with key stakeholders, to evaluate and prioritize these requests for future releases of the dashboard.

Ensure “just enough” infrastructure to support your dashboard

Last but not least, make sure that you are not over-emphasizing infrastructure. Successful dashboards often require more funding for the human support team than the latest and greatest platform. Do not overspend on infrastructure and neglect the human side of change. All you need is “just enough” hardware and software to support your dashboard initiative. For example you may need servers, publishing and content management tools, or analytics and reporting software. Do you have the needed infrastructure to perform data modeling, KPI definition and process modeling? What about data management and data security? Think through the minimum infrastructure needed for your dashboard and obtain these resources.

For example, at the regulator of the financial industry, the project portfolio dashboard used the Microsoft (MS) Sharepoint infrastructure. It provided templates for the dashboards, and web parts for the most common tasks. Since Sharepoint is a MS product and offers tight integration with MS Office and this portfolio dashboard facilitates project management, transferring MS Project schedule data to the dashboard proved to be painless. The dashboard team used SQL Server Analysis Services (SSAS) and SQL Server Reporting Services (SSRS) for its analytic and reporting needs. Role based security was used to ensure that only authorized users had access to sensitive data.

Successful Outcomes

By effectively addressing critical success factors, data dashboards can indeed be tools for transformation. At the regulator of financial industry, the project portfolio dashboard transformed the project management practices of the technology group and resulted in a near perfect record of on-budget, on-time, as-expected delivery of valued projects over multiple years. For the federal agency, the DTV.gov dashboard successfully engaged the American public in the smooth national transition. At the golf course management company, the financial dashboard transformed the operations of the front and back office, increasing customer satisfaction, reducing cost and improving profits.

Rita Hadden Retains Copyright

Rita Hadden specializes in culture change, software best practices, process improvement, and project management. She has provided leadership, coordination, and mentoring for over 150 complex projects for more than 45 organizations in the private and public sectors. Her experience includes over 35 years working worldwide with software professionals at all levels.