statistical_reports

We are super lucky in NZ to have something as important as statistical literacy not just written into our curriculum at all levels, but also formally assessed as part of our national qualifications system (NCEA). All of our students have to deal with messages given to them that are based on data, whether it be in the news, online via their Facebook feed, or on TV through advertising (just to list a few examples), so even if we don’t actually assess AS91266 Evaluate a statistically based report or AS91584 Evaluate statistically based reports within a learning programme, it’s still important include good examples of statistically based reports in our teaching of statistics.

This post will focus on AS91266 Evaluate a statistically based report, as this standard provides a really great opportunity to weave together many of the statistics achievement objectives at curriculum levels six and seven.  This focus will also give me an opportunity to provide some guidance on how to select good examples of statistically based reports that are strong enough statistically to be used for the assessment of this standard and also accessible to the students.

So what do you need to look for in a report?

The following questions are a good place to start when considering the suitability of a statistically based report.

  • Is the purpose of the report clear?
  • Is the report based on a survey?
  • Are the findings of the survey clear?

If you get a yes to all three of these questions, you might be heading in the right direction. For example, this article on Stuff regarding whether eating carbs help you lose weight is kind of interesting (it even has a video), has lots of contextual information around diets, has a clear purpose, but it’s not a survey and there’s only one result reported (a weight loss of 10kg). In comparison, this article also on Stuff regarding whether children eating carbs can explain rising obesity levels is more on track.

Then the next round of questions to ask are:

  • Are the population measures and variables are clear?
  • Are the sampling methods clear?
  • Are the survey methods clear?
  • Is the sample size clear?

If you also get a yes to most of these questions, then the report is probably statistical enough for students to use and will allow them to discuss the statistical methods and measures used and sampling and non-sampling errors (one of the requirements for this standard). According to these questions, the second article given above – whether children eating carbs can explain rising obesity levels – is not yet statistical enough (there are no details in the article about sampling or survey methods), so…..

How do we find a report that is statistical enough?

NZ media sites like nzherald.co.nz and stuff.co.nz, and the super awesome StatsChat statschat.org.nz, are good starting points to find reports (tip – search for key words like “survey”, “study”, “causes” etc.), but just like with the articles above, while the articles may be interesting, engaging and relevant for teenagers, there is still some work to do before they can be used in a meaningful way in the classroom. What follows is an example of making a report statistical enough 🙂

I found this article entitled “The unhealthy reason men go to the gym” on Stuff on October 18th [download pdf]. Read it carefully and see how many of those questions above you can tick yes to. The article itself is a good starting point, but on reading the article you’ll see that it takes a few findings from one study and one survey and blends them together to make the article. Also, none of the last four questions are met, and you could argue the first three are met insufficiently. This report needs more details to make it statistical enough. [Note: At this stage I am motivated enough to continue this process because I like the context and I believe it will resonate with students – this is not always the case!]

The article gives links to both of the original documents. The document for the recent study is only available if you pay for it (or have access through your library service like I do at the University) and is a very technical report that is beyond the reach of our students. The full report from the 2010 survey runs to 139 pages and is accessible to our students [download pdf], however, since the link to this survey is broken I had to Google it.

sd

From here there are two options:

  1. Trim the original newspaper article to report only on the 2010 survey and include sufficient details from the original report so that all those questions posed above can be answered (so adapt the article, the approach that seems to be used for the AS91584 exams).
  2. Trim the 139 page report but retain enough information in the report so that all those questions posed above can be answered. I reckon about four pages of A4 is a good target, but it will depend on how the report is formatted, how many graphs are included etc.

I think I prefer the latter, although I have done both in my teaching. The key point here is that students are not expected to go out and find the original article themselves to complete their evaluation of the report.

How do we make a report that is statistical enough?

The 139 page report from Mission Australia is pitched at the right level for curriculum level seven, but is far too long for a class of Year 12 students. Most reports from surveys like this have an introduction and an executive summary. These two sections often provide nearly enough for the report to be used in the classroom. I’ve put these sections into a shorter pdf document, which is well set out and easy to read [download pdf].

In deciding whether the report is statistical enough, we need to consider those questions again:

  • Is the purpose of the report clear?
  • Is the report based on a survey?
  • Are the findings of the survey clear?
  • Are the population measures and variables are clear?
  • Are the sampling methods clear?
  • Are the survey methods clear?
  • Is the sample size clear?

The foreword gives a straight forward summary with a reasonable hint of the purpose of the report. The introduction of the introduction expands on the purpose, and the next three sections of the introduction (participation, areas of focus and methodology) provide sufficient information sampling and survey methods. The demographics section of the executive summary provides some population measures, variables and the sample size, and the remaining sections provide the findings of the report. However, it would make it easier for students to evaluate if each section contained a little more detail and the information was provided in tables and/or graphs. Half an hour editing the pdf and cutting and pasting some text and tables from the full document creates a report that is statistical enough for a good quality learning or assessment activity – YAY 🙂 [download pdf]

What about contextual knowledge?

Students should also be provided with enough contextual knowledge associated with the report so that they can integrate this within their evaluation. For the example discussed, this could include the original article from Stuff along with the edited version of the full report, making it clear to students they are evaluating that edited version of the survey report not the Stuff article. It would also be a great idea for the class to discuss the topic or issue being explored by the study/survey, and compare and contrast their own experiences. This discussion would help them to consider whether the results reported are meaningful, useful and/or relevant.

Want to read more about statistical literacy?

If you can get yourself a copy of “Seeing through Statistics” by Jessica Utts (the eBook version is available for around $NZD60), you will not be disappointed. This was the first book I read to help get my head around how to teach statistical literacy, and it is full of really great guiding principles, examples and explanations. Like I said at the beginning of the post, even if you aren’t teaching AS91266 or AS91584, it is our job to help our students to make sense of the world around them  – a world where so many people are using data and statistics is so many, not always awesome, ways.