How to weight your survey results in Pollfish

Surveys aim to gather insights from certain populations of interest. Oftentimes, that population of interest is large and diverse, which can make it very difficult to gather enough data from sample that is representative of the whole. This is why some researchers use survey weighting. 

How post-stratification works

Because research cannot always be conducted against a statistically representative sample, Pollfish created and launched the post-stratification feature. Post-stratification adds survey weighting to age and gender demographics to match what those populations should be according to the census data for that region, then adjusts the results to reflect the change. 

For example, if a survey is conducted to the general population of the United States, the number of participants may skew male or female, older or younger, and not reflect the actual demographic makeup of the country. When post-stratification is applied, results are adjusted against the census data to develop a more accurate dataset with just one click. 

Who should use post-stratification?

Post-stratification survey weighting is an especially powerful tool for researchers working with large, diverse populations, such as entire countries. In cases like these, it can be hard to reach a representative sample of the demographics that make up the population of interest. To ensure that the results are representative of the area, researchers can select “post-stratification” on their results page when a survey is complete and weighted data will be displayed below the raw data that reflects a more accurate representation of the audience, according to the census data for their region. 

Benefits of the post-stratification feature

There are a number of ways that post-stratification survey weighting can help researchers. 

• Statistically representative audience: Post-stratification offers more statistically representative age and gender data for 34 countries globally, and individually against the 50 US states and Puerto Rico. 
• Rapid weighting: Researchers who need data stratification have often done manual weighting to adjust their results. Post-stratification offers a single click to view the weighted data. 
• Exporting and Analysis: If you want to view or manage post-stratified data outside of Pollfish, weighted data is included as an extra column in Excel and SPSS exports. 
• Income stratification: When possible, post-stratification will include weighted data for respondents’ income for audiences in the United States and Canada. It is not required for respondents to provide this information so if the field is missing, post-stratification cannot be completed. 

Demographics weighted for Census data

When stratification is applied at the end of the survey, age and gender are weighted to reflect the census data of the region in which the survey was conducted. Age and gender are included in all 34 countries to which post-stratification is applicable. In the US, each state’s census data will be used automatically unless there is insufficient data for that state, in which case it will default to the general country’s population.

Income stratification is restricted to the US and Canada as the census data for income in the EU is unavailable. Income stratification is only available when a sufficient amount of data has been supplied by the respondents, as it is not a required field during profiling. If corresponding census data is unavailable for any field in the US, Canada, or the EU, stratification cannot be completed. 

How post-stratification accounts for younger audiences (ages 0-14)

The Child Online Privacy Protection Act (COPPA) limits the collection and use of personal information about children (defined as those under age 13) by operators of online services. As a COPPA compliant company, Pollfish does not target audiences younger than 14 years of age, however, Census data includes information from children at any age. To account for this, we used a linear weighted model to distribute the participation of audiences under 14 years old across all the other age groups in the census. 

Census data used in survey weighting

The United States Census data is available in every US state and is as recent as 2017. The information used comes directly from the American Community Survey (ACS), an ongoing survey conducted by the US Census Bureau. As the largest household survey administered by the Census Bureau, the data is continually collected and used by businesses in both the public and private sectors. It contains census data for all 50 US states, the District of Columbia, and Puerto Rico. 

The census data for the EU comes from Eurostat’s 2011 population and housing census survey. The survey is conducted decennially among the 28 countries belonging to the EU, as well as Iceland, Liechtenstein, Norway, and Switzerland.

Geographic regions that support post-stratification

Pollfish post-stratification of age and gender is available in all of the following countries:

Austria

Belgium

Bulgaria

Canada

Croatia

Cyprus

Czechia

Denmark

Estonia

Finland

France

Germany

Greece

Hungary

Iceland

Ireland

Italy

Japan

Latvia

Liechtenstein

Lithuania

Luxembourg

Malta

Netherlands

Norway

Poland

Portugal

Romania

Slovakia

Slovenia

Spain

Sweden

Switzerland

United Kingdom

United States

Weighted data for North America

In the United States, post-stratification can be applied to all 50 US states, the District of Columbia (Washington DC) and Puerto Rico. Because the weighted data is based upon the ACS by the US Census Bureau, US territories (American Samoa, Guam, the Northern Mariana Islands, and the US Virgin Islands) are not able to be stratified.

Income post-stratification is only available in the United States and Canada, as the census data for the EU doesn’t include income. Because income is not a mandatory field in initial respondent profiling, it cannot be stratified if the information provided by respondents regarding it is insufficient. It will be automatically included in post-stratification results for the US and Canada if the data is available.

Post-stratification of multiple countries

Post-stratification can be applied to surveys that field to multiple countries, so long as all of the countries have corresponding census data. The responses are collected, age and gender stratified to the census data representative of the respondent’s country, and the data of all countries is merged and presented with appropriate weighting applied for a complete dataset on the results page.

Tips for using Pollfish post-stratification

Fortunately, applying post-stratification is as simple as selecting the button “post-stratification” on the results page, but there are still a couple of considerations to keep in mind when using this exciting feature. 

When to use post-stratification vs quotas

Post-stratification is a powerful feature for researchers looking to ensure their raw data is representative of the general population. Using census data as a control, researchers are able to instantly weight their results by selecting “post-stratification” when the survey completes. Researchers might use this in a survey of a general population.

However, if a survey must meet a set percentage of participants from an age or gender group, setting quotas at the beginning of the survey is the way to go. Quotas may cause the survey to take a bit longer but will ensure that the distribution of respondents from each cohort is exact, rather than manipulating the data to match a percentage based on the results. Quotas would be a better fit for reaching a specific audience that isn’t representative of the general population, such as readers of a certain publication where the researcher knows the audience skews female. 

Quotas and custom age ranges do not affect the stratification algorithm, so it is possible to apply survey weighting to your results even if these targeting parameters have been applied. There is no added benefit to using these together, as they both are designed to adjust data for different reasons. 

Exporting post-stratified results

Weighted results can be exported in Excel and SPSS as another column added onto the regular export data. PDF export is currently unavailable. 

When results can be stratified 

Post-stratification is not viewable in real-time, it is only available once the survey is complete and all responses are collected. Once all results are in, they can be weighted instantly by selecting “post-stratification” on the left-hand side of the results dashboard. The stratified results will appear below the raw data.