name,text,elaboration,answer_choices_innerHTML,answer_choices_text,answer_choices,answer_type,source Q1concept_behav,"Does this task primarily require physical effort, as opposed to primarily requiring mental effort?","Even if tasks require a combination of physical and mental effort, one type will still be ""primary.""

For example, moving bricks from point A to point B requires almost entirely motor effort (even if you have to strategize about how to lift them), while calculating a tip in a restaurant is almost entirely mental effort (even if you have to use a pen).

Thus, think carefully about whether the ""core"" of the task is physical or mental. Similar activities --- for example, clicking a mouse --- can be involved in both physical and mental tasks, depending on their main purpose. For example, if you have to use a mouse to click on the correct answer for a multiple-choice math test, this is mostly a mental task. But if the goal of the task is just to click something as much as you can, then it's mostly a physical task.",Mental | Physical | Not applicable or not answerable based on the task description (Please Elaborate Below.),Mental | Physical | Not applicable or not answerable based on the task description (Please Elaborate Below.),Mental | Physical | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,McGrath Q2intel_manip_1,What is the fraction of physical (as opposed to mental) effort required for the task?,"This question places Question 1 on a continuous scale, rather than having you pick one or the other.

Tasks at the high end of the dimension require only physical (motor) activities for completion, whereas those at the opposite extreme require only mental (reasoning, thinking) activities for the successful task completion.",,,Entirely Mental Effort – Entirely Physical Effort,unit_range,Shaw Q3type_1_planning,"Is this a ""planning"" task? In other words, is one of the main purpose(s) of this task to produce a sequence of concrete steps or actions that an individual can follow to achieve some goal?","These are tasks ""having emphasis on action orientation.""

Answer 'yes' if this task involves writing down a series of concrete steps that someone will follow in order to achieve a goal. Examples include organizing an event, or figuring out a plan for buying things at the store. This plan may later be executed by the person making it, or could be given to someone else.

Answer 'no' if this task just involves doing an activity without writing the steps down. (For example, if you write a recipe and then cook, you are planning; if you cook without a recipe, that is NOT planning.) Also answer 'no' if there is no intention to actually use the plan in the ""real world,"" or if it's just a hypothetical math problem. Sometimes math problems involve a story or context (e.g., ""calculate how much time it takes for Jimmy to go to the store,"" or ""find the shortest path that a traveling salesman can take"". However, these are math problems rather than plans.","No, this is NOT a ""planning"" task. | Yes, this is a ""planning"" task. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, this is NOT a ""planning"" task. | Yes, this is a ""planning"" task. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,McGrath Q4type_2_generate,"Is this a ""generation"" or ""brainstorming"" task? In other words, is one of the main purpose(s) of this task to produce a number of ideas or examples, without any particular action associated with them?","These are tasks based around generating ideas, examples, or concepts.

The examples being generated can be either abstract (generating words and colors) or concrete (generating ideas for how to spend $10,000 or ways to use a paperclip). If the ideas are concrete, they should NOT suggest a specific plan or course of action. This question is therefore different from the previous question; the participants are NOT generating a plan.

Note that you should only answer ""yes"" to this question if one of the main outcomes of this task is to generate such ideas. Many tasks require people to discuss ideas (for example, a jury task), or come up with creative ways to solve a problem (for example, a math problem), but the primary goal of such tasks is not just to generate the ideas.","No, this is NOT a ""generation"" or ""brainstorming"" task. | Yes, this is a ""generation"" or ""brainstorming"" task. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, this is NOT a ""generation"" or ""brainstorming"" task. | Yes, this is a ""generation"" or ""brainstorming"" task. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,McGrath Q5creativity_input_1,"What fraction of the effort required for this task is creative thinking (as opposed to any other type of effort, whether physical, logical, etc.)?","You can think of this, in a way, as the continuous version of the previous question, but it is also intended to capture a wider range of ""creative"" activity than just purely generating ideas.

Descriptors that might suggest creativity include: ""new"", ""novel"", ""unique"", ""come up with as many ideas as possible"", ""invent"", ""create,"" etc. A purely creative task is solely based around such activities (for example, writing advertisements).

Descriptors that do not suggest creativity include: ""add these numbers together"", ""click your mouse as fast as possible"", ""find the optimal allocation."" Such tasks might involve simply following the rules or executing a task without thinking creatively.

Finally, some tasks are in between --- it may be possible to invent a unique way to solve a math problem. This is more creative than a problem that simply requires adding numbers.",,,No Creativity Required – Purely Creative Task,unit_range,New Q6type_5_cc,"Is one of the main purpose(s) of this task to resolve people's differences in opinion, perspective, or viewpoint?","Within a group, a difference in viewpoint or opinion may arise either because members interpret the same information in different ways, assign different importance to that information, or both. Resolving those differences may take many forms, such as having a discussion, building consensus, holding a vote on the options, or simply thinking through ways to account for the viewpoints (e.g., making a judgment call that balances the perspectives).

For individuals, resolving opinions may involve adjudicating real or hypothetical disagreements for others. For example, a game in which you pretend to be a judge and decide court cases would primarily involve resolving differences in perspective or opinion.

Another example of a 'yes' is an activity where you run a funding organization and different projects have applied for access to your funds. A decision-making task for allocating these funds will involve making sense of the different opinions and perspectives that parties may have.","No, one of the main purpose(s) is NOT to resolve differences in opinion/perspective/viewpoint. | Yes, one of the main purposes is to resolve differences in opinion/perspective/viewpoint. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, one of the main purpose(s) is NOT to resolve differences in opinion/perspective/viewpoint. | Yes, one of the main purposes is to resolve differences in opinion/perspective/viewpoint. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,McGrath Q7type_7_battle,Can the outcome of this task be described in win/lose terms?,"This question captures whether a task is a ""competition"" or ""battle,"" in which your outcome is relative to someone or something else (e.g., another player, team, or an AI agent).

A battle is ""where the focus is on conquest of an opponent and winner-take-all distribution of payoffs,"" and a competition is ""where there is a lot of emphasis on standards of performance excellence over and above the reckoning of winners and losers."" Competitions between sports teams fit this category (e.g., basketball, soccer, ice hockey), but so too do military and street gang battles (e.g., the spoils of war go to the victor no matter how honorably the vanquished may have fought).

Answer 'yes' ONLY if you ""win"" or ""lose"" relative to somebody else who is playing the game. If someone performs poorly in the task, but they are not being compared to another player, you should answer 'no.' For example, if you are being asked to come up with as many ideas as possible, you may think that you ""lose"" if you generate zero ideas or that you ""win"" because you generated many ideas. However, you should still answer 'no,' because this question is asking about ""winning"" and ""losing"" in relative terms. A task in which you need to come up with more ideas than another player, on the other hand, would be a 'yes,' since you would win or lose relative to that other person.","No, the task is NOT a ""competition"" or ""battle."" | Yes, the task is a ""competition"" or ""battle."" | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, the task is NOT a ""competition"" or ""battle."" | Yes, the task is a ""competition"" or ""battle."" | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,McGrath Q8type_8_performance,"Does the task have an all-or-nothing outcome? In other words, is it sufficient to just meet a particular standard?","These tasks are about doing just enough work to meet some threshold or standard, but once you reach that standard, precision does not matter.

As an example, a pass-fail class (e.g., ""anything above 70% is a pass"") is all-or-nothing. On the other hand, a letter grade (A+ for 100%, A for 90, and so on) is not. Another example is a task in which people simply need to move objects from Point A to Point B: if they meet the outcome, they succeed. Even if they moved the object part of the way there, they don't get any credit for their effort. Moving it an extra 10 feet does not get them any brownie points.

Counter-examples include games in which you try to earn as many points as possible, and you end up with a numeric score instead of simply a success/fail. For example, trying to generate as many ideas as possible, or solve as many problems as possible, would NOT be all-or-nothing tasks. If participants moving an object get credit for every foot that they manage to move it, then it's NOT all-or-nothing.","No, this task does NOT have an all-or-nothing outcome. | Yes, this task has an all-or-nothing outcome. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, this task does NOT have an all-or-nothing outcome. | Yes, this task has an all-or-nothing outcome. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,McGrath Q9divisible_unitary,Is it efficient and useful for members of the group to work on discrete parts (or subtasks) of this activity?,"Ask yourself: is it efficient or useful for a team of people to ""divide and conquer"" this task, or is this really something where one person should be doing most of the work?

A key heuristic is whether the different sub-parts of the task are interdependent or not. If the sub-parts are not dependent on each other, it often makes sense for different people to work on each part separately (so answer 'yes'). However, if the sub-parts are interdependent, one person can't start their part without waiting for another person to finish, so dividing and conquering doesn't make sense (and you should answer 'no').

For example, if you are solving 10 simple arithmetic problems, each of ten people may work on one of them. If there is only one problem to be solved, it's not efficient for one person to do the thinking and another to do the writing. Similarly, for a task where you want to generate as many ideas as possible, everyone can separately come up with ideas and combine them in the end. For a task where everyone needs to come up with a shared plan, the parts of the plan depend on each other, so it is difficult to divide and conquer.","No, you should NOT divide-and-conquer this task. | Yes, you should divide-and-conquer this task. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, you should NOT divide-and-conquer this task. | Yes, you should divide-and-conquer this task. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,Steiner Q10maximizing,"Is the goal (or one of the goals) of this task to try to achieve doing something as much as possible, as many as possible, or as quickly as possible?","Sometimes the goal that is to be achieved entails doing as much as possible of something, or doing it as rapidly as possible. Thus, if an individual or group is asked to exert a maximum force on a rope, a strong pull is regarded as a more successful performance than a weak pull. If a team of mountain climbers is asked to ascend a cliff as rapidly as possible, maximum speed is the criterion against which performance is evaluated. Look for tasks asking participants to score the most number of points, get the most utility, for everyone to generate as many ideas as they can, and so on.","No, one of the goals is NOT about working as much/many/quickly as possible. | Yes, one of the goals is about working as much/many/quickly as possible. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, one of the goals is NOT about working as much/many/quickly as possible. | Yes, one of the goals is about working as much/many/quickly as possible. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,Steiner Q11optimizing,Is the goal (or one of the goals) of this task to try to achieve a precise outcome?,"Here, there is a specific standard to meet, but precision matters. (This is unlike the previous question about all-or-nothing outcomes, for example, where precision did not matter.)

Look for tasks with a specific, most preferred outcome. If the task is to estimate the temperature of a room, the goal is to agree with the value indicated by the thermometer. Another example is moving exactly 10 objects --- no more, no less --- or to exactly reproduce something (such as a task where people have to exactly copy a work of art).

Also look out for concepts like ""exactly,"" ""precisely,"" ""optimal,"" or ""best."" For example, if you are asked to generate exactly 10 ideas, generating both 9 ideas and 11 ideas would be considered a failure. Another clue might be terms like ""constraints,"" ""requirements,"" or ""rules"" that someone has to follow, as long as there is a specific best answer within the rule set. For example, if you want to buy the most number of items using a budget of $100, the ""constraint"" of $100 also serves as a precise goal. That is, the closer you get to $100 (the more precise you are), the better you do at the task.

","No, one of the goals is NOT to achieve a precise outcome. | Yes, one of the goals is to achieve a precise outcome. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, one of the goals is NOT to achieve a precise outcome. | Yes, one of the goals is to achieve a precise outcome. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,Steiner Q13outcome_multip,"Is there only one ""best"" solution (or possible solution) to this task?","For example, an arithmetic problem, such as summing a bunch of numbers, will have only one correct answer. On the other hand, a creative writing task has many valid solutions.

In cases where there are many possibly correct answers, or answers with partial solutions, there could still be one ""best"" solution, or a specific ""best"" solution that the experimenter is looking for. For example, in a game in which one needs to win in as few moves as possible, there may be an optimal lowest number, even if there are less efficient solutions.","No, there is NOT a best solution. | Yes, there is a best solution. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, there is NOT a best solution. | Yes, there is a best solution. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,"Shaw, Zigurs" Q14sol_scheme_mul,"Is there only one method for achieving the task, as opposed to many alternatives for task completion?","Is there only one right way of solving the problem?

Answer 'no' if there is more than one possible course of action or process to attain the group's goal.

Answer 'yes' if there is only ONE process or action that will lead to the correct answer or achieve the goal.

As an example, answer 'yes' if the instructions specify exact steps, e.g., ""first you write all the 3-letter words, then the 4-letter words, then the 5-letter words,"" or if there's really only one way to do the task (e.g., perform long division). Actions might also be limited by the environment: participants may be working on an electronic system that restricts their communication or the steps that they are allowed to take.

However, answer 'no' if the task is open-ended in terms of course of action: e.g., if participants are simply asked to ""come up with a solution"" or ""give as many ideas as possible"" without specifying how they should achieve the goal.","No, there are MANY ways to achieve this task. | Yes, there is only ONE way to achieve this task. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, there are MANY ways to achieve this task. | Yes, there is only ONE way to achieve this task. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,"Shaw, Zigurs" Q15dec_verifiability,"Can acceptable solutions to this task be demonstrated or verified to be correct (e.g., by an expert or third-party)?","This item refers to the ""degree to which acceptable solutions can be demonstrated to be correct,"" via logic and rules as opposed to having a general consensus.

For example, the solution to a math problem can be verified via the rules of algebra. In a task where participants are asked to buy the items with the best value, one can also use logic to list out every possible combination of items and show that their solution is the best one. Another type of demonstration could involve showing facts: for example, if the task is to estimate how many people live in the United States, the ""ground truth"" is the statistic from the U.S. Census.

Examples of tasks where you should answer 'no' include answering the question, ""should we ban all guns?"" --- this cannot be verified via logic and rules. Evaluating whether something is ""creative,"" as well as other subjective judgements, is also not demonstrable.","No, you CANNOT verify a solution to be correct. | Yes, you CAN verify a solution to be correct. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, you CANNOT verify a solution to be correct. | Yes, you CAN verify a solution to be correct. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,Laughlin Q16shared_knowledge,"Can this task be written as a ""formal model"" that an algorithm could solve?","Some tasks can be written as a formal model or math problem, expressed with rules and syntax that the problem-solvers share.

One way to think about this question is to ask yourself, could a robot or algorithm do this task?

Here are examples of a 'yes' answer:

  1. A specific set of rules and outcomes: Robots are really good at following rules. Answer 'yes' if you can input the rules and the desired outcome, and have a robot follow predetermined steps to get the result. For example, you can tell the robot, ""buy as many items as you can with a budget of $100."" (In fact, online shopping websites have exactly this tool!)
  2. You can write it like a math problem: Robots are really good at doing math. Something like, ""find the best teams under the constraints,"" combined with a list of how much ""utility"" people get from being put together, can easily be put into a computer and solved. Similarly, if you're trying to find the shortest path from Point A to Point B, you can think of this in terms of geometry.
  3. You can break the problem down into small units of meaning: Examples of these units include the alphabet (26 letters), colors (which can be broken down into units of Red, Blue, and Green), or coordinates on a graph (X,Y). An algorithm could use these units of meaning to solve the task. For example, when generating words, the letters are the units. You can imagine an algorithm in which you use ""brute force"" to find every possible combination of letters that makes a valid word. Another example is Wordle --- many people have built bots that consistently solve the puzzle!

Here are examples of a 'no' answer:

  1. Creative ideas that cannot be generated just by following basic rules. Whereas you can generate words just by using the rule of trying out all the different letters in the alphabet, you can't generate stories in the same way. There are no ""rules"" for what makes a good story. (We ask you not to think about advanced models like GPT-3 for the purposes of this question.)
  2. Subjective tasks that require a judgment call. Deciding whether or not we should ban all guns can't really be done by an algorithm: people have to debate these ideas.
  3. Tasks relying on social dynamics. Anything focusing on relationships between participants should be labeled 'no.'
","No, this is NOT a formal problem that an algorithm could solve. | Yes, this is a formal problem that an algorithm could solve. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, this is NOT a formal problem that an algorithm could solve. | Yes, this is a formal problem that an algorithm could solve. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,Laughlin Q17within_sys_sol,Is there enough information in the problem to find a valid solution?,"This item refers to whether there is sufficient information to obtain a solution within the system. In other words, this question is about whether it is possible --- using all of the information provided to the participants --- to get a valid or acceptable answer.

By ""the system,"" we mean the entire set of stimuli given to the participant(s) for solving the problem. If the person is rating images, it is the set of images and the rating scale/survey; if it is a game, then it is the entire self-contained ""system"" of the game and the rules for playing it. If it is an optimization problem, it's the set of object(s) that need to be optimized and the list of constraints.

Examples of 'no' answers:

  1. The only way to get a valid solution is outside of the ""system"" --- e.g., looking the answer up, asking someone else. For example, a trivia quiz (where you have to look up the right answer) would be a 'no' for this question.
  2. There is no right answer, or the question is fundamentally unsolvable (e.g., it's a trick question).
  3. There's not enough information to solve the question. In math, for example, you need two equations to solve for two unknown variables. If you don't have enough information, you can't do the problem.

Examples of 'yes' answers:

  1. Any question where you can get a right answer from the information given is a 'yes.' This is true even if you're not sure of your answer. For example, in a multiple-choice quiz, you may not be sure that your answer is correct. But if one of the choices was the right answer, and the problem gave enough information for you to select it, then answer 'yes.'
  2. If there are multiple valid answers in response to the problem (e.g., a creative writing task where any valid story is accepted), the answer should be 'yes.'
","No, there is NOT enough information to find a solution. | Yes, there is enough information to find a solution. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, there is NOT enough information to find a solution. | Yes, there is enough information to find a solution. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,Laughlin Q18ans_recog,"If someone who is able to solve the problem explains their answer, would others recognize it as correct without contest?","This question is about whether participants who are not themselves able to solve the problem have sufficient knowledge of the system to recognize and accept a correct solution if it is proposed by someone else. In other words, are you informed enough to know the right answer when you see it?

If there is no well-defined ""correct"" answer to begin with, the answer to this question is always ""no.""","No, you CANNOT know the right answer when you see it. | Yes, you can know the right answer when you see it. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, you CANNOT know the right answer when you see it. | Yes, you can know the right answer when you see it. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,Laughlin Q19time_solvability,"Is a participant able to come up with a provably correct solution, assuming sufficient ability, time, motivation, and resources?","If someone is able to solve the problem (e.g., find the best solution), will that person ""have sufficient ability, motivation, and time to demonstrate the correct solution to the incorrect members?"" In other words: Assuming infinite time and resources, can someone both (1) solve the problem AND (2) show that the solution is right?

You should answer ""yes"" to this question only if a participant can do BOTH steps. Otherwise, answer ""no.""

If there is no well-defined ""correct"" or ""best"" answer to begin with, the answer to this question is always ""no.""

Finally, remember that this question is about whether it is possible to come up with ANY ""best"" solution and prove that it is right; if the proof is so complicated that others would not recognize it, your answer to the previous question (about whether someone would recognize the correct answer if told it) should be ""no,"" but you should answer ""yes"" to this question, because a proof of a correct solution still exists.","No, you CANNOT BOTH solve the problem and show the solution is right. | Yes, you can BOTH solve the problem and show the solution is right. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, you CANNOT BOTH solve the problem and show the solution is right. | Yes, you can BOTH solve the problem and show the solution is right. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,Laughlin Q20type_3_type_4,Is there an objectively correct solution to this task that can be calculated or selected?,"If you answer 'yes' to this question, the task should have ""a demonstrable right answer, and the group task is to invent/select/compute that correct answer."" This right answer can be found in multiple ways:


If you answer 'no' to this question, ""there is not a demonstrably correct answer, and ... the group's task is to select, by some consensus, a preferred alternative.""

The alternatives might involve:

","No, there is NOT an objective answer. | Yes, there is an objective answer. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, there is NOT an objective answer. | Yes, there is an objective answer. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,McGrath Q21intellective_judg_1,"On a scale of 0 (entirely subjective, with no correct answer; a judgmental task) to 1 (entirely objective and demonstrable by pure logic; an intellective task), how would you classify the extent of demonstrable correctness of this task?","Imagine a continuous scale from judgemental (subjective) to intellective (objective/logical) tasks. You can think of this as the continuous version of the previous question.

Intellective tasks are problems or decisions for which there exists a demonstrably correct answer within a verbal or mathematical conceptual system. The criterion of successful group performance is the achievement of this correct answer. Judgmental tasks are evaluative, behavioral, or aesthetic judgments for which there does not exist a demonstrably correct answer. The criterion of successful group performance is the achievement of consensus on a collective decision.

If the solution is demonstrable by numbers or statistics, but must be done relative to other participants (for example, you want to know your score relative to how other players did), your answer should be in the middle (e.g., 0.5).",,,"Entirely Subjective, No Right Answer – Entirely Objective, Purely Logical",unit_range,Laughlin Q22confl_tradeoffs,Does completing this task require participants to evaluate tradeoffs --- conflicting possible solutions or conflicting pieces of information?,"Some tasks provide a variety of inputs to help complete the task, but these inputs don't always agree or point to the same solution, and participants must evaluate or navigate tradeoffs in executing the task.

For example, a task involving writing an ad for a product may provide individuals with a variety of product features, customer reviews, and market information that may not all point to the same answer or angle for the ad. Similarly, a problem asking you to devise a business plan may require you to manage conflicting pieces of information, such as inconsistent needs for different consumer segments and tradeoffs between different courses of action.","No, there are NOT conflicting pieces of information or tradeoffs. | Yes, there are conflicting pieces of information or tradeoffs. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, there are NOT conflicting pieces of information or tradeoffs. | Yes, there are conflicting pieces of information or tradeoffs. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,Zigurs Q23ss_out_uncert,"When doing this task, will the participants have any uncertainty about whether their method or solution will lead to the desired outcome?","This question is asking whether a player knows for sure that their strategy or answer is right BEFORE the activity ends, or whether they have to wait until AFTER to find out. Answer 'yes' if participants submitting their answers may feel unsure that their submission or approach is correct, with no way to ""check"" their answer until the task is done.

Here are tasks where you would want to answer 'yes:'


On the other hand, examples of 'no' are:

","No, participants do NOT feel unsure. They can know that their answer is right during the task. | Yes, participants do feel unsure. They don't know how well they did until the end. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, participants do NOT feel unsure. They can know that their answer is right during the task. | Yes, participants do feel unsure. They don't know how well they did until the end. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,Zigurs Q24eureka_question,"Is the solution to the question obvious as soon as it is proposed --- for example, once people see the ""trick,"" they know how to solve it?","Some questions have a special trick in them, such that, if you know the trick, the question is easy, but if you do not know the trick, the question may be quite difficult. An example of this is the Sphinx's Riddle (e.g., ""what has 4 legs in the morning, 2 in the afternoon, and 3 in the evening?""). This puzzle is confusing if you have never heard it, but if you have heard the answer (a human -- since babies crawl, adults walk, and the elderly use canes), you'll never be confused again.

In other words, does this task cause a ""Eureka!"" or an ""Oh, I get it!"" moment when the answer is explained?

If there is no trick, answer 'no' to this question.","No, there is NOT an ""aha"" moment that makes this task easy once you see the trick. | Yes, there is an ""aha"" moment that makes this task easy once you see the trick. | Not applicable or not answerable based on the task description (Please Elaborate Below.)","No, there is NOT an ""aha"" moment that makes this task easy once you see the trick. | Yes, there is an ""aha"" moment that makes this task easy once you see the trick. | Not applicable or not answerable based on the task description (Please Elaborate Below.)",No | Yes | Not applicable or not answerable based on the task description (Please Elaborate Below.),nominal,Laughlin