Showing posts with label Knowledge Sharing. Show all posts
Showing posts with label Knowledge Sharing. Show all posts

Wednesday, 4 September 2019

Using Level of Detail (LOD) to Determine Completeness


Recently a colleague and myself found ourselves in a situation where we needed to produce a report on whether something was 'complete' or not. But oh wait, there's a bit more to that than a simple TRUE/FALSE flag.

Take phone specs for example, I found a data source that lists approximately 30 different phone specification categories. Imagine if this information needs to be 'complete before we can show it to people, we'll need to know where the gap is in order to know where to put our efforts.In plain language: 'what's complete and what still missing'?

Ok, so let's say I go ask my leader, out of the 30 various specs that each phone has, which ones must be complete? The "mandatory fields" so to speak. And this is his/her requirements:
  1. Year must not be null/blank
  2. Out of the 2G, 3G, and 4G fields, we need at least 1 of them to be filled in
  3. Weight must not be null/blank
  4. OS must not be null/blank
  5. RAM must not be null/blank
  6. Primary camera must not be null/blank
Also, in addition to the specific phones that are incomplete, they also wants to know the % of "brands" and "models" that's complete. A sort of status report so to speak. In essence, they wants a report that gives the following information:
  • Which brands/models are complete vs incomplete? (if one phone within a brand is incomplete, then mark the entire brand incomplete.)
  • Click on brand or model to see the phone within each grouping
  • Which phones are complete vs incomplete? (if one field within a phone is incomplete, then mark the entire phone incomplete.)
  • Click on phone to see which mandatory fields are complete vs incomplete

First, for all the fields that cannot be null/blank, I quickly made calculated fields for each of them:

     IF ISNULL([Field])

     THEN "Incomplete"
     ELSE "Complete"
     END



But for the bandwidth, only 1 out of 3 fields need to be filled in. So the calculated field is slightly different:

     IF ISNULL([2G bands])
     THEN 
         (IF ISNULL([3G bands])
         THEN 
             (IF ISNULL([4G bands])
             THEN "Incomplete"
             ELSE "Complete"
             END)
         ELSE "Complete"
         END)
     ELSE "Complete"
     END


Now I need to report on this completeness based on the following 3 dimensions:
  • Per phone
  • Per brand
  • Per model
For phone, that's easy enough, all I have to do is create a calculated field based on each row:

     IF [C. Bandwidth]"Complete"
     THEN
         (IF [C. OS]"Complete"
         THEN
             (IF [C. PrimeCam]"Complete"
             THEN
                 (IF [C. RAM]"Complete"
                 THEN
                     (IF [C. Weight]"Complete"
                     THEN
                         (IF [C. Year]"Complete"
                         THEN "Complete"
                         ELSE "Incomplete"
                         END)
                     ELSE "Incomplete"
                     END)
                 ELSE "Incomplete"
                 END)
             ELSE "Incomplete"
             END)
         ELSE "Incomplete"
         END)
     ELSE "Incomplete"
     END


This essentially looks at all the columns and if one of the column is 'incomplete' then mark the entire record as incomplete. To me, on a visual level, it looks like this:


I'm using the information between Category 1 - 4 to determine what the overall flag should be for each row/phone.

But when it comes to Model or Brand, we have to look at a number of rows, and if out of that grouping of rows, one of them is 'incomplete', then mark that entire grouping of rows incomplete. Basically like this:


In this instance, I'm looking at each of the categories as well as the brand. as long as one of the overall flag for a brand is 'incomplete', flag all the rows in that particular brand 'incomplete'.

To do that, you could use the LOD function in Tableau. However, that only works on measures. How we did it, was to flag each row's complete/incomplete indication using numbers, then convert those numbers back into a string.

First, create a calculated field that turns the result per phone into a measurable number:

     IF [C. Bandwidth] = "Complete"
     THEN
         (IF [C. OS] = "Complete"
         THEN
             (IF [C. PrimeCam] = "Complete"
             THEN
                 (IF [C. RAM] = "Complete"
                 THEN
                     (IF [C. Weight] = "Complete"
                     THEN
                         (IF [C. Year] = "Complete"
                         THEN 0
                         ELSE 1
                         END)
                     ELSE 1
                     END)
                 ELSE 1
                 END)
             ELSE 1
             END)
         ELSE 1
         END)
     ELSE 1
     END


Now we can add them up. If all the records in the given brand are complete, the sum of all of them should still be 0. But if any of the rows are incomplete, then the sum would be greater than 0.

Create an LOD calculated field based on that measure:

IF
{FIXED [Brand]:SUM([C. Phone (measure)])} = 0
THEN "Complete"
ELSE "Incomplete"
END


Apply this to Model, and now we can now look at this set of data in the way that was required (dashboard embedded). Hopefully this was helpful. Let me know what other ways have you used LODs before in the comments below.

Wednesday, 10 October 2018

Data Visualization & Data Design Talks

Great talks on data visualization and information design you should watch.








More Hans Rosling on TED: https://www.ted.com/search?q=rosling

Wednesday, 9 August 2017

Using HTML as Your Datasource

I'm not a programmer and I don't have an IT background, so when I heard of this method of connection, I was super stoked to try it out!

In a nutshell, you would structure your data connections this way:
HTML to Google Sheets to Tableau

Selecting Your Source - The Web Page

Obviously it would be best to connect to a web page that updates over time. Meaning, the same URL will hold data that updates and thus update your data source.

I am working on a workbook related to the Canadian Federal budget and how they spend our hard earn tax dollars, so I connected my Google Sheets to the Authorities and Expenditures by Program page.



This is, however, not a good example as the government posts by groupings of years. It is shown in the URL and thus I don't believe this particular page will be updated with new information but rather, the new information will be published onto a new page that has a new URL altogether. Having the year in the URL kind of gave it away.


A better web page to connect to would be, for instance, like a Wikipedia page where tables are updated without the URL changing.


The Formula & The Table

In Google Sheets, use the following syntax:


= IMPORTHTML(url, query, index)

The URL is easy, simply copy and paste.
The query is "table" or "list depending on what type of structure contains the desired data.
The index, however, may take a little bit of work.

Open up HTML viewer and look for all the "tables" until you find the table in question. For instance, if your table is the 3rd out of 4 tables, you would put "3" in the formula for index. The federal expenditure web page only had 1 table so I put "1" for my formula.



*Open HTML in Google Chrome by going to Settings à More Tools à Developer Tools
  Chrome: https://developers.google.com/web/tools/chrome-devtools/
  IE: https://msdn.microsoft.com/en-us/library/dd565627(v=vs.85).aspx#htmltool
  Firefox: https://developer.mozilla.org/en-US/docs/Tools/Page_Inspector/How_to/Examine_and_edit_HTML

In the end, my full formula looked like this:


=IMPORTHTML("https://www.canada.ca/en/treasury-board-secretariat/services/planned-government-spending/budgets-expenditures/expenditures-program-2016.html","table",1)

You'll see the formula load for a few seconds, and then the table should load.




Connect to Tableau

Open Tableau and connect to Google Sheets. A pop up window will prompt you to sign into your google account, then it should connect to Google Drive automatically. You can also copy and paste the URL of your Google Sheet directly in the search bar.


Canadian Federal Fiscal Budget by Type

Saturday, 11 March 2017

The Impact of Different Types of Visualization

My manager shared a great video by the Harvard Business Review on the impact of visualizations, which could have different effects depending on what you show and how you show it.



Here are my notes:


DESIGNING PERSUASIVE CHARTS with Scott Berinato


"People read charts like they read books" Scott said. There are a lot of things you can't control and information is read in the order it was presented, thus making building charts difficult at times. People also naturally gravitate toward things that stand out, like colours and outliers, and almost immediately start to form narratives.


He used 5 examples to talk about misleading charts:

1. Ideas that Don't Exist

This chart, presented in congress, shows as if "abortions have risen above cancer screening", Scott said, calling it "a deliberate attempt to mislead".


Personally, I always go back to the statistics rule of "correlation does not imply causation".

2. Look at Axes Labels

Scott used this graph to illustrate how deceiving a cumulative bar chart can be, showing growth when there was none.


In fact, when you separate out the revenue individually, there is a decline.


Basically, this is a very ill-suited chart for the message.

3. Pay Attention to the Spacing

Scott argues that perhaps there are no truly objective charts, but rather, each serves it's own purpose. 
Wide spacing between "Years"



When a chart has a much wider spacing, the fluctuation of the line does not appear as drastic as if the same chart had a much narrower spacing.

Narrow spacing between "Years"
Most of the time, the decision as to how a chart is presented is arbitrary and there is no real standard. The important thing is to make sure a chart is used appropriately.

4. Truncated Y Axis

Truncated Y axes create a more dramatic story, which sometimes could be misleading. This chart looks as if the average job satisfaction really plummets throughout an employee's career.




However, if the entire Y axis is shown, the decrease looks unremarkable.



Scott said that some scientists may look at very limited ranges of data where truncating the Y axis becomes appropriate. There are no hard rules, just think about whether you are exaggerating the story unnecessarily.

5. Dual (Y) Axes

Dual axes charts measure 2 data points in the same visual space.



First of all, although we're looking at care sales between Tesla and other brands, the 2 charts have completely different units (one in percentage increase, the other in dollar increase). Then, when looking at the green line, proportionally, it looks as if Tesla shares are projected to increase 25% (a quarter of the chart) when in reality, it will only increased about 2% (Y axis on the left does not contain the entire 100%).


Since we're looking at Tesla vehicle sales compared to other vehicle sales, Scott thinks this chart is a more appropriate representation.


Q: Common Decision Points?

This depends on the data you choose to show. For example, the following graph shows the sales of vinyl records between 1993 and 2014. It appears, quite justly so, that the sales of vinyl records have "sky-rocketed".



However, if you start the graph in 1973, then you'll see that the "peak" is not a peak at all.


Scott then compared the sales of vinyl with the sales of other physical/digital/streaming album sales, and the proportions becomes apparent.



Q: How do you know when you've crossed the line?

Use the golden rule, and ask yourself whether you feel deceived or mislead by the chart. When choosing the right representation, ask yourself if you are "zooming in on the message or are you distorting the truth".

Q: How do you know charts are accurate?

Evaluate all the ways charts can be misleading. For example, pay attention to whether the Y axes are truncated and the story is in fact more dramatic than it really is. Or when encountering a dual axes chart, analyze the data individually / separately first before comparing the 2 together.

See Scott's book Good Charts: The HBR Guide to Making Smarter, More Persuasive Data Visualizations for more info.

Tuesday, 21 February 2017

YYC Tableau Seminar Feb. 2017

I attended the Tableau seminar in Calgary this week and learned about the Top 10 Business Intelligence Trends for 2017. In addition, Alberta Health Services and QuICR made presentations explaining how they use Tableau to enhance their day-to-day operations.



TOP 10 BI TRENDS with Howard Morgenstern (@datacanuck)

1. The Modern Business Intelligence (BI) Model

With the role of data discovery shifting from a centralize IT department to individual businesses, the modern BI (project) model means a shift from the traditional waterfall approach to an agile model.

2. Collaborative Analytics

Open data sources provide more data to discover, but also fosters a collaborative environment. When data is shared, more insight is uncovered, and better decisions can then be made (see presentation by AHS and QuICR below).

3. All Data Becomes Equal

Tableau allows users to connect to multiple data sources and does not discriminate on their size. Cross referencing data from Oracle and Excel becomes a walk in the park.

4. Self-Serve Extends to Data Preparation

According to Howard, Tableau reinvests >30% of its revenue in R&D and is working heavily on improving its data preparation usability and functions. Their hope is to extend not only the data discovery / visualization portion of the fun to (business user) self-serve, but the data preparation portion as well. With Project Maestro underway, I certainly look forward to the good days of data preparation, which are apparently still ahead of us.

5. Embedded Business Intelligence

Embedded BI will become the norm of the future, with some companies already creating seamless dashboards right on their web pages, and others, pushing the boundaries of Tableau's functionalities (such as write-back).

6. IT Becomes the Data Hero Again

The hope of self-serve is to shift IT from the role of report creators back to data enablers. Instead of being the bottle neck, IT will once again be the data hero every organization needs.

7. Work with Data in More Natural Ways

Howard spoke to this point by comparing a crosstab with a graph, explaining that as data gets presented visually, people will be able to work with data in more "natural" ways.

8. Cloud

According to Howard, Canadian companies struggle with the idea of hosting their data on the cloud due to a shortage of servers being physically in Canada. But that's changing as companies increase their physical presence in Canada (such as Amazon Web Services)

9. Advanced Analytics Becomes More Accessible

Tableau hopes that advanced analytics will become more accessible with their various platforms, enabling businesses to gain more in depth insight to their data.

10. Data Literacy Becomes a Fundamental Skill

Howard believes that data literacy will become a fundamental skill in the future, just like how the ability to use the Microsoft Office Suite has become a basic requirement to do one's job.


PRESENTATION BY AHS & QUICR

Alberta Health Services and QuICR demonstrated some of their dashboards and talked about the impact Tableau had on their day-to-day operations. These real-life applications were really interesting, and my main take-away points were:
  • More transparency through information sharing
  • Greater synergy leveraged through each others' strengths
  • Faster learning through knowledge / process / performance sharing

Door to Needle - QuICR

Door-to-needle is the time between a patient gets in the (hospital / treatment centre) door to when they receive treatment, and is a critical time measurement for acute ischemic stroke patients. As neurons are lost each second, a shorter door-to-needle time may mean saving a life. QuICR uses Tableau to analyze door-to-needle time across multiple locations and was able to better improve the door-to-needle time of not just locations with poorer performances, but even the better performing ones as well. The speaker said that when locations with poor performance see that a 30 minutes door-to-needle time was achievable in other locations, it gave them the confidence that they will be able to achieve that one day as well.