Task Name,Source,Link,Stimulus Complex,Goal Directives
9 Dot Problem,Maier 1930,https://doi.org/10.1037/h0073232,"Participants are given a 3x3 grid of nine dots, which is shared by everyone completing the task.","Within a limited amount of time, participants must connect all nine dots (e.g., draw a line that passes through them) using no more than four lines, and may not retrace any lines.
Participants will be able to visually tell that they have successfully met these criteria. They are given full credit if they accomplish this task, and no credit if they fail.
An important note is that there is really only one ""correct"" answer to this question — that is, the way to connect the nine dots actually requires the participant to go ""outside"" the boundaries of the 3x3 grid, or else it cannot be done. The true goal of this task is to find this optimal answer.
Participants who do not think ""outside the box"" therefore usually have difficulty completing this task. Once they are shown that the answer involves going outside of the box, they usually have an ""aha"" moment and do well on the task in the future."
Abstract grid task,Adams et al. 2021,https://doi.org/10.1038/s41586-021-03380-y,"Participants are given a shared grid, in which some squares in the grid are colored in, and other squares in the grid are not colored in. The grid is interactive, and offers the opportunity to change the squares' colors by either ""adding"" color where a square is not filled in, or ""subtracting"" color where a square is already filled in (removing the color from the square).
Clicking a particular square on the grid ""toggles"" the color — shifting it from colored to non-colored, or vice versa.","The goal is to make the grid appear exactly symmetrical from left-to-right and top-to-bottom using the fewest number of clicks. (Each click represents one manipulation — either adding or subtracting color from the grid.)
Each problem has a known ""lowest"" number of clicks, which represents the best possible answer. However, participants are not given any feedback about whether their answer was the best. Finally, participants must complete the task within the time limit."
Advertisement writing,Whiting et al. 2019,https://dl.acm.org/doi/pdf/10.1145/3359311,"Participants were given the description of a product from Kickstarter. For example, they could read about ""Soapstone: the Sustainable Travel Soap Dispenser,"" and learn about its features.","The goal was to write an online text advertisement of no more than 30 characters for the Kickstarter product. The advertisement had to be creative, informative, and get people to click. When working in teams, participants needed to submit a single collective writeup, produced within the time limit."
Aerospace Investment (Role-playing),Sanpietro 2019,https://www.pon.harvard.edu/daily/teaching-negotiation-daily/how-negotiation-games-can-help-you-develop-skills-to-resolve-business-and-commercial-disputes/,"Participants are randomly paired for a scored negotiation over a venture capital investment. Each person is randomly assigned the role of either the venture capitalist investor or the founder of the aerospace company. Both individuals are scored on their ability to negotiate favorable investment terms for themselves and on the quality of the relationship they develop with their potential business partner. Each negotiator's Total Score is a sum of Substantive Points, awarded according to the agreed-upon terms of the investment, and Process Points, awarded according to each partner's perception of the negotiation process.
Specifically, each negotiable term of the investment is quantified as specific point values. After the term sheet has been agreed upon, participants independently fill out questionnaires that ask them to evaluate each other on five attributes to determine the amount of Process Points they are awarded. The five process attributes are: trust, respect, equitability, regard for the other's interests, and interest in future collaboration. These attributes serve as proxies for assessing the future of the business relationship based on their experiences during the negotiation.","Participants are explicitly instructed that their goal is to maximize their Total Scores (Substantive Points + Process Points), and are given a limited amount of time (e.g., 30 minutes) to conduct their negotiation. The total score reflects the overall success of balancing both substantive and relationship concerns. For instance, the founders may be able to negotiate all the substantive terms in their favor, but if their potential business partners have no desire to work with them again, they would have left significant value at the negotiating table, and this will be reflected in a lower Total Score."
Allocating resources to programs,Whiting et al. 2019,https://dl.acm.org/doi/pdf/10.1145/3359311,"Participants are given a list of complex competing programs, along with details of those programs. All participants get a list with the same programs. For example, participants could receive the following list:
(1.) To purchase a new computer system for the county government in order to hold local taxes constant. (2.) To establish a community arts program featuring art, music, and dance programs for children and adults. (3.) To establish an additional shelter for the homeless in the community.","The goal is to collectively decide how to allocate $500,000 between three competing programs within the time limit. As each program is designed to appeal to personal values, there is no right, wrong, or optimal way of allocation, and the purpose of the task is to resolve different conflicts in people's values and interests."
Apache helicopter flight simulator (Longbow2),Marks et al. 2002,https://doi.org/10.1037/0021-9010.87.1.3,"Participants are randomly assigned to groups of three to play the Apache helicopter flight simulator game called Longbow2. Each person is randomly assigned the role of either the pilot (person flying the aircraft), the gunner (person operating the weapons system), or the radar specialist (person monitoring and interpreting radar systems containing critical enemy information). Participants are given a limited amount of time to complete various tasks (e.g. destroying enemy tanks) and stay alive. Before performing the task, they would receive training in role-specific task skills.","The goal is to maximize the score of the team. The score is calculated from the sum of the number of targets acquired during the course of each mission and the team’s average ""alive time"". Team performance scores could range from 0 (no targets killed, helicopter killed by enemies immediately) to 30 (killed 15 targets, stayed alive for the duration of the 15-minute mission)."
Architectural design task,Wooley et al. 2010,https://doi.org/10.1126/science.1193147,"Participants receive written and video instructions of a task to design a house, a garage, and a pool in a computer program. Within the program, they have a limited number of building blocks of different color and size to use. For example, participants can view a table of available materials, in which they can see twenty-seven 1x1 red blocks, twelve 2x4 blue blocks, zero 1x6 orange blocks, etc.
Participants receive scoring information for each object. For example, participants can see that each square foot used to build a pool will bring them a $50 bonus and that following certain design restrictions (e.g., using certain colors for the pool and making it symmetric) will give them a $1500 bonus. Participants also receive rules to follow for building objects. For example, a pool must include a diving board and have blue floors.","The goal is to design a house, a garage, and a pool of as high value as possible. Three objects must be designed from a limited set of building blocks within a fixed amount of time. The task can only be considered completed if participants adhere to the provided rules for each building. If participants don't follow the building rules, their design project fails.
In other words, participants have to maximize the value of objects they are designing but they have to do so in a specific way, balancing tradeoffs of following the building rules and adding value to each object. Within these restrictions, however, they are free to be creative and produce any design they wish."
Arithmetic problem 1,Shaw 1963,https://collections.uakron.edu/digital/collection/p15960coll1/id/21257/,"Participants see a hypothetical story problem about five people who operate five machines. The problem reads as follows:
You are a five man team whose job it is to manufacture a product, the completion of which requires the operation of five machines. In the past, you have rotated positions to avoid boredom, but each man has spent most of the time operating the machine that he prefers. John prefers machine 3, Steve machine 2, Walt machine 4, Robert machine 1, and Dennis machine 5.
The Methods man has been around checking the time each man requires to complete the operation on one product when he is operating each of the five machines. He has come up with the following results:

Machine 1 Machine 2 Machine 3 Machine 4 Machine 5
John 3 min 3 min 4 min 3.5 min 4.5 min
Steve 2 min 2 min 5 min 2.5 min 3.5 min
Walt 1 min 2 min 5 min 2 min 1.5 min
Robert 4 min 1 min 3 min 3.5 min 3 min
Dennis 5 min 3 min 2 min 5 min 3 min

Your foreman has noticed that when each man runs the machine that he most prefers, the total time spent on each product is 16 minutes. It seems to him that a different method of operation would result in substantial savings. He believes in letting his workers make their own decisions, as far as possible, and has asked that you consider the problem and try to come up with a plan that will be more efficient than the present mode of operation.","The goal is to pair each man with one of the machines, such that the amount of time it takes to manufacture the product is minimized.
There is exactly one right answer to this problem — it is to put Walt on machine 5, Steve on machine 4, Dennis on machine 3, Robert on machine 2, and John on machine 1. In the optimal answer, the total time to create the product is 10 minutes.
Participants have a limited amount of time to submit their answers. Those working in groups can discuss their answers before submitting in an online portal.
The task is graded based on how close a submission is to the true minimum time for creating the product. Submitting the correct answer (10 minutes) will give participants the highest possible score; a solution that builds the product in 11 minutes will get more points than a solution that builds the product in 15 minutes."
Arithmetic problem 2,Shaw 1963,https://collections.uakron.edu/digital/collection/p15960coll1/id/21257/,"All group members receive different clues and have to collaborate to reach a solution to an arithmetic problem. The problem involves figuring out what time a plane arrives at its final destination given the set of clues. No one of the participants can solve the problem alone as no one has all the information needed to solve it. The clue is randomly distributed among group members and involves essential information such as plane stopover time, the distance between the source and destination, and the speed of the plane.","The goal is to solve the problem in the shortest possible time. In order to do so, they should communicate with each other about their clues in order to reach solutions. They must give an answer within the time limit."
Battle of the sexes,Rabin 1993,https://www.jstor.org/stable/2117561,"Participants are randomly paired with a different anonymous opponent for a predetermined number of decision-making rounds. In each round, both players must make a decision. The final outcome for the players is determined by the decision of both parties.
Throughout the rounds, the players have access to a shared table that displays the possible outcomes, depending on each player's decision. For example, suppose that each player has 3 options. Then the rows might represent Player 1's options, and the columns might represent Player 2's options. There would be 3x3 = 9 possible outcomes, and the 9 cells of the table would contain the number of points that the players would earn based on their choices. For example, if Player 1 chooses their first option and Player 2 chooses their second option, then the final outcome would be determined by the number of points listed in row 1, column 2.
Some outcomes are better than others. Thus, in order to get a better outcome, players have the option to use non-binding one-way communication to signal that they may choose a particular row or column, so that the other player can attempt to coordinate. The process of communication is as follows:
• The participant is asked if they would like to make an announcement. They may select either yes or no.
• If they select yes, they will be asked what they would like to announce. They may select any of the actions that will be available to them during the decision-making period (for example, ""I intend to select option 2""). The opposing participant will then be told of their selection.
• If they select no, the opposing participant will be told an announcement was not made.
Players can choose to communicate one or more times. After all communications are complete, the participants will then independently select their actions, and each will receive the number of points corresponding to this combination of actions in their table. Players are not required to actually select the action they previously announced.","The goal for each participant is to maximize their earnings (points) through the decision-making rounds. Therefore, they should wisely make announcements and selections in order to obtain the highest value in the payoff table. The best outcome is when both players make choices that lead to the highest possible payoff. In all rounds, participants are required to make choices for their announcements (if applicable) and their actions in a fixed amount of time."
Best job candidate (hidden-profile),Schulz-Hardt et al. 2006,https://doi.org/10.1037/0022-3514.91.6.1080,"Participants read information about three candidates who supposedly had applied for an assistant professorship. Each candidate's profile included positive, neutral, and negative characteristics, and the characteristics are rated on a 7-point scale ranging from 1 (negative) to 7 (positive). Not all information, however, is shared among group members. The information is distributed to create a hidden profile. For example, each group member knows only two of candidate A's six positive characteristics, and only one of candidate B's negative characteristics. For candidate C, they knew only one positive and one negative characteristic. Thus, based on the shared information, candidate B seemingly has more positive and fewer negative characteristics and therefore would be the preferred candidate. Only by exchanging information through group discussion could the group members detect the best alternative, candidate A.","The goal for each group is to reach an agreement on the best candidate they would hire. The task is designed so that the full information would show that a candidate among the three is objectively the best (the ""correct answer""). Through discussion and sharing of information, the group should find out the optimal candidate within the time limit."
Biopharm Seltek,Bhatia and Gunia 2018,https://doi.org/10.1016/j.obhdp.2018.06.003,"Participants are randomly paired for a negotiation about the sale of a biotechnology plant. Each person is randomly assigned the role of either the buyer, the CFO of BioPharm, or the seller, the CFO of Seltek. BioPharm needs to either buy or build a plant to produce a genetically engineered antibiotic compound called Depox. Depox is a promising pharmaceutical product, and BioPharm needs a plant with special manufacturing equipment. There are two choices for BioPharm: (1) build a new plant; (2) buy a plant that is already set up to manufacture genetically engineered compounds. For this second option, Seltek seems like an ideal ""turnkey"" facility. BioPharm possesses public information about Seltek, including previous appraisal value, accounting statements, and so on. Meanwhile, Seltek is a medium-sized pharmaceutical company that (unbeknownst to BioPharm) has been struggling financially, and it is desperate to sell both the plant and the patent for its chemical compound. Seltek possesses detailed information about its own financial state, and also knows basic information about BioPharm from public documents.","The goal of each party is to reach an agreement that is as beneficial as possible for their own side. In other words, Biopharm should buy the plant for as low a price as possible, and Seltek should sell the plant for as high a price as possible. Since each side possesses different information, participants must strategically share (or choose to withhold) what they know in order to get a good deal. For example, if Seltek divulges the information that they are desperate to sell, they would likely obtain a much lower price than if they did not divulge such information.
There are two possible outcomes of the negotiation: either the two parties reach some consensus (to sell the plant at an agreed-upon price) or they are unable to reach an agreement, and BioPharm must build a new plant instead (a costly option)."
Blocks World for Teams,Butchibabu 2016,https://doi.org/10.1177/0018720816639712,"Participants work together in a virtual world to complete a search-and-deliver task. There are nine rooms, designated A1 through C3, and each room contains various colored blocks. A 10th room, designated the ""drop zone,"" is located at the bottom of the map.
A special sequence of colored blocks is depicted below the ""drop zone."" We can call this special sequence the ""target sequence,"" and it represents the blocks that participants will bring from the main rooms into the drop zone.
In addition to the 10 rooms, there are hallways allowing participants to travel from room to room. Each participant controls an avatar and can move their avatar to pick up a block. The participant then navigates to the drop zone to successfully deliver the block. If the participant delivers a block into the drop zone that is not of the requested color, this incorrect block would be automatically and randomly placed into one of the other nine rooms.
As they move from room to room, participants are only able to see a limited amount of information. For example, they are only able to see the blocks in the room they are currently occupying, and they are not able to see their teammates. However, participants can see whether or not an adjacent room is occupied.","The goal is to deliver the correct sequence of colored blocks (the target sequence) from the virtual rooms to the drop zone. Participants should complete the task as quickly as possible, as well as finish within a limited amount of time."
Bullard Houses,"Cohen, Leonardelli, and Thompson 2010",http://dx.doi.org/10.2139/ssrn.1612404,"Participants are assigned to play in one of three formats: one-on-one, two-on-two, or three-on-three. In other words, all teams play with another team of equal size. One side serves as the seller(s) on behalf of a retail company representing the Bullard Family. The other acts as an agent representing a blind trust company operated by a hotel group. The seller is instructed to only sell the property to a known buyer who fully discloses their planned use of the property, while also seeking to get a fair deal for the family. The buyer is instructed to keep the buyer's identity hidden and not reveal the intended use of the site while also trying to reach a fair deal for the property. Both parties are to attempt to reach an agreement while acting in their roles. After negotiations cease, participants should report whether they have reached an agreement or an impasse. These should be coded, i.e. agreement(0) and impasse (1).","Participants should report whether the negotiation is settled or not. This requires careful analysis of the available information both before and during the negotiation. Beforehand, negotiators should work through a variety of simplified, but reasonably realistic financial structures (bonds, mortgages, loans, etc.) to make a judgment about the relative worth of the various offers and possible alternatives. During the negotiations, while much information is instructed to be kept hidden, the information that does get shared may have important and even unforeseen implications for the other side. In this game, the best outcome of a negotiation is to avoid reaching an agreement. In other words, an impasse (1) is the optimal answer."
Carter Racing,Brittain and Sitkin 1986,http://learnmoore.org/MBA205/05CarterRacing.pdf,"Participants get several pages of description about a dilemma facing a racecar team, which involves deciding whether to go ahead with the race that would begin in the immediate future. The description mentions that the team has been experiencing a series of engine failures and that an engine failure during this race on national television will present a danger to the driver and team's sponsorship. However, if the team does well on the race, it will get a lucrative sponsorship deal.
Participants are given a chart with information about the temperatures during the last 7 engine failures, which shows a range of temperatures 53-75 degrees and a mean temperature of 64 degrees. The chart is misleading because it does not contain information about the air temperature when the car does NOT experience an engine failure (therefore, the information is biased).
Participants are also given instructions that mention that they can ask for any additional information during the task, but that the request must be precise.","The goal is to come up with reasons and ultimately decide to terminate the participation in the race within a fixed amount of time.
The data in the description is gathered from the Report of the Presidential Commission on the Space Shuttle Challenger Accident in 1986. The relationship between ambient temperature and O-ring failure is disguised as the relationship between air temperature and car engine failures. In other words, participants that decide to go ahead with the race make a decision that is parallel to the decision to go ahead with the Space Shuttle Challenger launch.
Participants have to recognize that the data they have is inconclusive and ask the experimenter for the air temperatures when the engine has not failed. If they do so, the experimenter provides them with an additional chart of the last 17 races without an engine failure. The chart shows that races are clearly warmer when engine failures do not occur with the range of 66-82 degrees and a mean temperature of 73 degrees.
Therefore, the only correct and precise solution to the task is not to proceed with the race, while participants that come up with a conclusion to proceed with the race fail the task. Participants also have to explain their reasoning behind this decision by finding a relationship between engine failures and air temperatures."
Carter Racing (Experimenterless Version),"Sellier, Scopelliti, and Morewedge 2019",https://osf.io/nyxwg,"Participants get several pages of description about a dilemma facing a racecar team, which involves deciding whether to go ahead with a race that would begin in the immediate future. The description mentions that the team has been experiencing a series of engine failures and that an engine failure during this race on national television will present a danger to the driver and team's sponsorship. However, if the team does well on the race, it will get a lucrative sponsorship deal.
In particular, this race day has unusual temperatures, and participants are told that the outside temperature could be related to engine failures in the past. Participants are given charts with information about the outside temperatures during previous races in the season, and whether or not there was an engine failure during those previous races. Some of these charts may focus just on the successful races, while others may focus only on the failed ones — as a result, the presentation of the data requires the participant to do work to piece the full picture together.","The goal is to come up with reasons and ultimately decide to terminate the participation in the race within a fixed amount of time. Everyone working together submits their decision in a shared interface.
The data in the description is gathered from the Report of the Presidential Commission on the Space Shuttle Challenger Accident in 1986. The relationship between ambient temperature and O-ring failure is disguised as the relationship between air temperature and car engine failures. In other words, participants that decide to go ahead with the race make a decision that is parallel to the decision to go ahead with the Space Shuttle Challenger launch.
Participants have to recognize that, once they are able to piece the full picture together from the information they are given, there is a clear relationship between temperature and engine failure. Based on the temperature of the day, they should predict that the car's engine will fail, causing them to lose the sponsorship.
Therefore, the only correct and precise solution to the task is not to proceed with the race, while participants that come up with a conclusion to proceed with the race fail the task. Participants also have to explain their reasoning behind this decision by finding a relationship between engine failures and air temperatures."
Categorization problem,Abimbola 2006,https://uwspace.uwaterloo.ca/bitstream/handle/10012/824/gaabimbo2006.pdf?sequence=1&isAllowed=y,"Participants receive 16 cards with pictures of two objects on each one (32 total objects). Half of the objects (16) on the cards are related to each other, and can be grouped into four categories. For example, the categories could be ""vehicles,"" ""animals,"" ""furniture,"" and ""shapes."" There are four objects per category. The remaining 16 objects are distractors. There is one category object and one distractor on each card.
Participants see all the objects depicted on the cards in a random order. They do not know which objects on the cards belong to the four categories and which are distractors, which makes this task difficult. For example, ""furniture"" may be one of the categories, and participants' cards may include a picture of an armchair, bar stool, office chair and table. However, there may also be distractors, such as an alarm clock and wall clock, which could ""throw off"" someone attempting to figure out the categories — they may think that the alarm clock, wall clock, and office chair are all related to a category of going to work, and therefore miss the true category of ""furniture.""","The goal is to identify the 16 cards belonging to categories, and correctly form four categories with them, within a fixed amount of time.
The task is designed in such a way that it is impossible to form four categories if participants start using distractor objects; there is only one objective way of forming categories so that all cards are used. Participants either complete this task by forming the four objectively correct categories, or they fail."
Checkers,Wooley et al. 2010,https://doi.org/10.1126/science.1193147,"Participants are given an online checkers board. This is an 8x8 board, in which the squares are colored dark and white in alternating colors. Participants start out with 12 round, flat pieces on each side; these are placed on the dark squares on their side of the board.
The rules of checkers are as follows:
• The two sides take turns playing. One side completes a move, and then the other side plays.
• You can only move a piece diagonally across the board (the game uses only the dark squares).
• A piece can only move diagonally into a square where there are no existing pieces.
• Pieces can only move forward (the only exception is rule #6).
• You can take (or ""capture"") your opponent's pieces by moving diagonally, jumping over the piece being captured, and landing on an unoccupied square. Multiple enemy pieces can be captured in a single turn by jumping forward several times in a row.
• If a piece reaches the other side, it becomes a ""king"" and has an additional piece stacked on top of it. Kings have the additional ability to jump backwards.
• The game ends when one party loses all of its pieces (everything is captured).
Participants control one side of the board, and a computer controls the other side. When playing as a team, participants discuss their strategy and make their next move together.","The goal is to win — to beat the computer (the opponent) in checkers. This involves capturing all of the computer's checkers pieces by following the rules of the game. Participants need to win the game within a fixed time limit, or else they automatically lose."
Chicken,Camerer 2017,https://doi.org/10.1257/jep.11.4.167,"The game of Chicken simulates a potential confrontation between two players in which each player has the choice to either ""Dare"" the other player or to ""Chicken out."" Participants are given a set of payoffs in advance, and depending on each participant's choices, they can obtain different outcomes. In general, participants get a large benefit if they ""Dare"" the other player and the other player ""Chickens out."" If both players ""Chicken out,"" participants each get a small benefit. If both players choose to ""Dare"" the other, then they get a penalty.
Each participant independently makes their choice without discussion. This process is repeated for a predetermined number of rounds.","Each participant's goal is to maximize their own payoff. The participants must balance the temptation of ""Daring"" the other person and the risk of their opponent doing the same to minimize their losses. Participants must also make each decision in a fixed amount of time."
Computer maze,Aggarwal and Wooley 2010,https://doi.org/10.1016/j.obhdp.2013.04.003,"Participants are given a computer monitor with a virtual maze environment and either a keyboard or a joystick. The maze consists of a long winding corridor with many hallways branching off. The hallways are populated by complex unfamiliar objects called ""greebles,"" which are difficult to distinguish from one another. A maze contains a certain number of pairs of identical greebles and a certain number of distractor greebles.","The goal is to navigate through the entire maze and tag as many identical greeble pairs as possible within a fixed amount of time.
Participants are evaluated based on maze navigating and greebles tagging. Participants receive points for navigating through the entire maze and don't receive, but also don't lose, any points for not going through the entire maze. Participants get points for tagging identical greeble pairs and lose points for tagging distracing greebles or mixing greebles from other pairs. The maximum score occurs when participants both navigate through the entire maze and tag all greeble pairs correctly."
Crisis mapping,Mao et al. 2016,https://doi.org/10.1371/journal.pone.0153048,"Participants are given a ""crisis mapping"" tool on a computer. They view a collection of pre-processed 1567 tweets about Typhoon Pablo in the Philippines, which were posted on December 4-5, 2012, and they receive detailed instructions on how to navigate the crisis mapping interface.
Some of the tweets contain information relevant to the typhoon, such as evacuation centers, damaged infrastructure, or number of deaths, while other tweets are irrelevant to the disaster.","The goal is to select relevant tweets and geo-locate, time-stamp, and categorize damage/flooding photos and videos in them, creating a precise and accurate crisis map for Typhoon Pablo within a fixed amount of time.
Experimenters pre-define a ""gold standard"" map with 49 distinct events in different regions. Participants only get credit based on how close their map is to the ""gold"" map. Therefore, their goal is to identify as many relevant events as possible without including irrelevant events, and the highest possible score is achieved when they identify precisely the same events as those in the ""gold standard."""
Desert survival,Mayo et al. 2020,https://doi.org/10.5465/amd.2018.0044,"Participants view information describing a survival scenario in which a plane crashes in the desert. The survivors are able to salvage a list of 16 items (e.g., water, machete, compass, cosmetic mirror) from the wreckage. Everyone sees the same list of items.","The goal is to rank all 16 items in the order of most important to least important for survival. For example, if water is the most important for survival, then it would receive the rank of 1. If the cosmetic mirror is the least important for survival, then it gets the rank of 16.
Participants' rankings are evaluated based on how close their rankings are to those created by a panel of survival experts. That is, the expert ranking is the ""true"" answer, and the closer a participant is to the true answer, the more points they earn. The highest possible score is achieved when participants rank all 16 items in the exact same order as the experts did.
If working in a team, participants are able to discuss the scenario with each other and come up with a single set of rankings as a team."
Dictator game and its variants,Dana et al. 2006,https://doi.org/10.1016/j.obhdp.2005.10.001,"Participants are placed into pairs and randomly assigned the role of ""dictator"" or ""receiver."" In each round of the game, the dictator sees an amount of money (the ""endowment"") on the screen, as well as an option to give part of the money to the receiver. After choosing how much to give to the receiver, the dictator keeps the remaining funds, while the receiver keeps the amount that was given to them by the dictator. The receiver is required to accept whatever the dictator chooses to give them and has no option to reject the offer.
The same pairs play together for a predetermined number of rounds and roles are reassigned each round.","Each participant has the goal of maximizing their own earnings. In the dictator role, participants must decide how much they will offer (from $0 to the full endowment). Since the dictator knows that the receiver cannot reject the offer, it is in the dictator's interest to give the receiver nothing ($0), because they would then keep the full amount.
However, there are a few reasons why dictators may choose to give more than $0. Giving nothing may conflict with social norms (because the dictator would then be seen as self-serving), or simply feel morally ""unjust."" If there is more than one round being played, dictators may worry that their relationship and reputation with their partner may be harmed in later rounds. The dictator must weigh these considerations and decide on a final amount to give to the receiver.
Participants must also make their decisions in a fixed amount of time."
Divergent Association Task,Olson et al. 2021,https://doi.org/10.1073/pnas.2022340118,"Participants are given a task sheet with instructions, a blank space to write 10 words in, and a list of rules.","The goal is to come up with 10 nouns as different from each other as possible within a fixed amount of time. For example, words ""cat"" and ""dog"" are considered similar to each other because they are often used in the same context, while words ""cat"" and ""thimble"" are different because they are rarely used together.
Participants are evaluated on how semantically different the words are. The minimum score occurs when all the words are the same, while the theoretical maximum score occurs when words have the least possible relation to each other. Participants maximize their score by coming up with as many of the most semantically distant words as possible."
Estimating Factual Quantities,"Silver, Mellers, and Tetlock 2021",https://doi.org/10.1016/j.jesp.2021.104157,"Participants are given a series of questions on estimating certain factual quantities. For example, participants can be asked to estimate the year when DNA was discovered or the current price per share of Twitter stock.","The goal is to answer as many questions from the series as correctly as possible within a fixed amount of time, and without looking up the solutions online.
Participants are evaluated based on the percent difference of their estimation from the correct answer. Participants receive the maximum score if all their estimations are exactly like the correct answers and the percent difference between them is 0. Participants maximize their score by submitting estimations that are as close to the right answers as possible. For example, because the DNA was discovered in 1869, participants get the maximum score for this estimation if their guess is 1869, while an estimation of 1860 receives a higher score than an estimation of 1900."
Estimating geological metrics,Whiting et al. 2019,https://dl.acm.org/doi/pdf/10.1145/3359311,"Participants are given a series of questions on estimating geological metrics. For example, participants can be asked to estimate the number of miles from New York to Los Angeles or the number of U.S. states that border the Gulf of Mexico.","The goal is to answer a series of questions within a fixed amount of time as close to the correct answer as possible without looking up the solutions online.
Participants are evaluated based on the percent difference of their estimation from the correct answer. Participants receive the maximum score if their estimation is exactly like the correct answer and the percent difference between them is 0. Participants maximize their score by submitting an estimation that is as close to the right answer as possible. For example, because the distance between New York and Los Angeles is 2446.3 miles, participants get the maximum score if their estimation is 2446.3, while an estimation of 2400 miles receives a higher score than an estimation of 2000 miles."
Estimating pages of a book,Engel et al. 2014,https://doi.org/10.1371/journal.pone.0115212,Participants see images of the page edges of a book.,"The goal is to guess the exact number of pages that the book has, using only the image of the page edges.
Participants are rewarded for being as precise and as close to the true number of pages as possible. They get points proportional to how many pages ""off"" they were, corrected for the total number of pages in the book. For example, 10 pages wrong out of a 400-page book is a smaller error than 10 pages wrong out of a 50-page book. Participants are not told if their judgment is exactly correct. Finally, participants have a limited amount of time to make their judgment."
Estimating social quantity,Wooley et al. 2010,https://doi.org/10.1126/science.1193147,"Participants are given a series of questions on estimating quantities of real-life social phenomena. For example, participants can be asked to estimate the median age in the U.S. in 2009.","The goal is to answer a series of questions within a fixed amount of time as close to the correct answer as possible without looking up the solutions.
Participants are evaluated based on the percent difference of their estimation from the correct answer. Participants receive the maximum score if their estimation is exactly like the correct answer and the percent difference between them is 0. Participants maximize their score by submitting an estimation that is as close to the right answer as possible. For example, because the median age in the U.S. in 2009 is 36.7, participants get the maximum score if their estimation is 36.7, while an estimation of 35 years receives a higher score than an estimation of 30 years."
Euclidean traveling salesperson,Bernstein et al. 2018,https://doi.org/10.1073/pnas.1802407115,"Participants get a map with multiple cities, which are displayed as dots on the screen. These dots can be connected by different ""paths"" by clicking on two dots one after the other. For example, clicking on point A, then point B, would draw a ""path"" between A and B. All participants, who are working on individual screens, are shown a shared map of the cities and can click and interact with it simultaneously.","The goal is to find and select the precise shortest path through the cities within a fixed amount of time. This is the sequence of cities corresponding to the shortest possible path to traverse through all of the cities, forming a cycle (or loop). In other words, the participant should click through the dots in a sequence, thus drawing a path that touches every city exactly once, and the first city twice: you can think of it as a ""traveling salesman"" who visits each city on the map before returning to his home city. Imagine that the shortest path involves going from A, to B, to C, to D. In that example, the participant's goal would be to find this path, and then click the cities in that order: A, B, C, D, A.
Participants are evaluated on how close the path they found was to the optimal shortest path. They will be able to see the length of the current path they found, but they will not be told that they have reached the optimum."
Find the common symbol,Leavitt 1951,https://doi.org/10.1037/h0057189,"Each participant is given a set of five symbols. For each participant, four of their five symbols are different from those of the other participants, and only one of the symbols is common among everyone. Participants cannot see each others' symbols, but they can send messages to one another.","The task is to work together to find the common symbol, and to do so (1) as quickly as possible; (2) with as few communications/messages as possible; and (3) with as few mistakes as possible. Their performance would be assessed by their time (in seconds) spent for a correct answer, the number of messages sent by each group member, and the number of wrong answers that had to be corrected during a trial. Participants should provide an answer within the time limit."
Find the maximum,Weidmann and Deming 2020,https://doi.org/10.3386/w27071,"Participants see an empty text box where they can enter numbers.
Behind the scenes, there is a hidden function. This hidden function has a maximum value — for example, the function y = -x2 reaches its maximum when x = 0. Participants know nothing about the hidden function; they only see the text box.
When participants enter numbers into the text box, the system will give them feedback about the value of the function at that point. For example, suppose that the hidden function is y = -x2 and the participant enters 2. Then the system will tell them that the value is -4, since this is the value of -x2. If the participant enters -5, the system will tell them that the value is -25.
As they make guesses, participants see a log, which shows their guesses so far, the feedback they previously received, and how many guesses they have remaining.","The goal is to correctly guess the maximum value of the hidden function after a limited number of guesses (for which there is feedback after each guess).
Every problem has an objectively correct answer — the mathematical maximum of that function. However, participants are not told that they have found the correct answer even when they submit it as their guess. For example, if the hidden function is y = -x2, the correct answer is 0. A participant who submits 0 will simply be told that the value of the function at that point is also 0; they will not be told that this is the true maximum. Instead, the participant will just get their fixed number of tries, after which they are required to submit their final guess.
Participants are scored based on how close their final guess is to the true maximum. If the answer is 0, then the participant gets the highest possible number of points if they guess 0; a participant who guesses 2 will get a higher score than someone whose final guess is 10. Participants aim to get as close as possible to the correct answer.
Finally, if participants are working together, each one gets a limited number of guesses, and then they decide their final guess as a team."
Game of Clue - Terrorist Attack,Shore et al. 2015,https://doi.org/10.1287/orsc.2015.0980,"Participants work in a shared web interface in which they solve a 'who-done-it' game like Clue or Cluedo, but involving a terrorist threat. Everyone in the game is solving the same puzzle.
Each participant has a personal ""inbox"" where they have their own list of facts and clues. In addition, at specified intervals of time, participants are able to use a ""search engine"" tool to look for more clues, which then appear in their inbox. For those who are working in groups, participants also have the ability to share clues by sending them to each other and to add comments to each other's clues.","The goal was to solve for four facts:
• Who would carry out the terrorist attack
• What would be the target
• Where the attack would take place
• When the attack would take place.
The game is designed so that there is exactly one correct answer for each of these four facts.
Participants submit their final answers via the web interface. Since there are multiple facts that need to be solved, participants are able to submit one ""solved"" fact at a time. They are also able to return to the submission page to change their answers at any point before the time limit ends. Participants are not given any feedback about whether their submissions are right or wrong, but when playing in a group, they are able to view what other people have been guessing so far (which can help provide a hint).
Participants are evaluated not only for correctly finding the exact facts, but also for solving the mystery as quickly as possible — they earn more points for finding each fact faster. Therefore, someone who finds all 4 facts in 2 minutes will earn more points than someone who submits the same answers, but takes 5 minutes to reach their conclusion."
Graph coloring task,Kearns et al. 2006,https://doi.org/10.1126/science.1127207,"Participants are given one or more ""graph coloring problems"". These are pictures of different topology/network graphs.
For context, these look like many different nodes (points), which are connected by various edges. Imagine, for example, a picture of a social network: each person in the network is a ""node,"" and each of their friends is an ""edge.""
In this case, all participants have access to and can interact with a shared graph. Participants will also be given a limited set of colors, with a larger number of colors than the minimum required to complete the task successfully.","The goal is to use the colors given to ""color"" the entire network without conflicts. This means that participants need to assign each node in the graph a color, ensuring that no two connected nodes share the same color. Using our social network example, if Person A and Person B are friends (which means they are connected), they cannot be assigned the same color. Participants must successfully color all nodes and resolve all conflicts within a specified amount of time, or receive no credit."
Guessing the correlation,Almaatouq et al. 2020,https://doi.org/10.1073/pnas.1917687117,"Participants receive a series of scatter plots which are a series of points plotted on a grid. In any round, all participants see plots with an identical correlation.","The goal is to estimate the correlation of the points on the grid. (For example, if all the dots are in a straight line pointing upwards, the correlation is 1; if the dots are completely scattered randomly with no pattern at all, the correlation is 0). There is a time limit to make an estimation.
Participants are evaluated on the accuracy of their judgment. They receive the highest score by guessing the exact correlation but they have to try to get as many points as possible by being as precise as they can. After participants submit their guess, they can see the true correlation and see whether they are right or wrong as well as how close their guess is to the true correlation."
Hidden figures in a picture (Recall Task),"Finlay, Hitch, and Meudell 2000",https://doi.org/10.1037/0278-7393.26.6.1556,"Participants see an incredibly busy and complex image that contains many hidden animals. Imagine a picture that looks like ""Where's Waldo"" — with lots of people and animals in one detailed, colorful image. Participants are then given a list of animals that they are supposed to find in the picture: for example, lion, penguin, dolphin, elephant.
If working together, participants in a team see the same picture and have the same list of animals to find.
Participants then engage in a task in which everyone working together attempts to find as many animals from the list as they can in the provided image. For example, if ""elephant"" is on the list, they need to look for the elephant in the picture, and then click on it.
After a delay, participants are taken to a page where they can type and submit words.","The goal of the task is to be able to recall as many animals as they can from the original list. Participants type and submit the words using the webpage. They have a limited amount of time to submit all the words that they can remember.
If working together, only one participant on a team needs to remember the word in order for everyone to receive credit.
Participants are scored based on the number of animals from the original list that they are able to recall. For example, if there were 10 animals on the list, they would get the maximum score if they recalled all 10 animals."
Hidden figures in a picture (Searching Task),"Finlay, Hitch, and Meudell 2000",https://doi.org/10.1037/0278-7393.26.6.1556,"Participants see an incredibly busy and complex image that contains many hidden animals. Imagine a picture that looks like ""Where's Waldo"" — with lots of people and animals in one detailed, colorful image. Participants are then given a list of animals that they are supposed to find in the picture: for example, lion, penguin, dolphin, elephant.
If working together, participants in a team see the same picture and have the same list of animals to find.","The goal of the task is to successfully identify the locations (in the image) of as many animals as possible from the list. Participants must find the most animals they can within a limited period of time.
Participants submit their response by pointing and clicking on the correct location of the animal, and they get points for every animal that they are able to find. Thus, they would get the highest score if they find all animals on the list. If participants are working together, only one person on the team needs to click on the animal in order for it to be counted, which means that having members of a team search separately is advantageous."
Husbands and wives transfer,Lorge and Solomon 1959,https://doi.org/10.1080/00223980.1959.9916346,"Participants are given a problem that reads as follows:
""On the A-side of the river are wives (W1, W2, W3) and their husbands (H1, H2, H3). All of the men but none of the women can row. Get them across to the B-side of the river by means of a boat carrying only three at one time. No man will allow his wife to be in the presence of another man unless he is also there.""","The goal is to achieve the fewest number of trips that get all six people to the B-side, and to solve this problem within the time limit.
There are several possible ""shortest"" solutions to this problem. In general, participants must realize that solving the problem requires people to row back and forth — that is, some of the people who go from the A-side to the B-side have to go BACK to the A-side — and then figure out the right combinations that meet the requirements (W1 cannot be in the presence of H2 or H3 unless H1 is also there, and so on).
Therefore, participants must satisfy the constraints of the problem while also thinking creatively. As a final piece of helpful context, this task is actually a well-known one that many people have built algorithms to solve."
Image rating,Engel et al. 2014,https://doi.org/10.1371/journal.pone.0115212,Participants see pictures of a series of products and accompanying slogans for them. They then have the opportunity to rate the product and slogan on a scale from 1 (worst) to 10 (best). Participants see these pictures one at a time.,"The goal is to give a rating that is as close as possible to how other Americans viewed the product and slogan. The challenge is to be exact in predicting how other Americans think.
Participants receive points based on how close their ratings are to the ""truth."" The ""truth"" is calculated based on the average poll response of 100 American users on Amazon Mechanical Turk. Participants are NOT told whether their guesses are correct or how many points they earned.
Finally, participants have to make their decisions within a limited amount of time."
Intergroup Prisoner's Dilemma,Bornstein 2003,https://doi.org/10.1207/S15327957PSPR0702_129-145,"All participants are given a fixed amount of individual money (""endowment"") and divided into two groups, A and B. Participants are told that their group is competing with the other group to see which one ""invests"" the most money. The winning group receives a larger cash prize.
Every individual player in the group must make an independent decision to either invest their personal endowment or keep it for themselves. The group wins or loses depending on the number of players that chose to invest relative to the other group. For example, if everyone in Group A invests and no one in Group B invests, Group A would win the entire cash prize and split it among its members. If some people from both groups invest, but more people from Group A invest than people from Group B, then the two groups would split the cash prize (but Group A would get more of it). Finally, if the same number of people in Groups A and B invest (a tie), then the two groups would split the cash prize 50-50.","Each participant's goal is to maximize their own payoff. Therefore, when making a decision, participants have to weigh up the cost of losing their individual endowment with the expected gain from splitting the cash prize if their group wins. Since the outcome depends on not only the participant's personal choice, but also the choices of all other players, there is a substantial amount of uncertainty and risk. For example, a participant could choose to invest, only to find that their group has lost (not enough other players invested), and they end up with less money than when they started. On the other hand, there is a greater chance of winning the maximum cash prize if everyone in the group invests.
Participants must make their decisions in a fixed amount of time."
Investment Game (aka Trust Game),Berg et al. 1995,https://doi.org/10.1006/game.1995.1027,"Participants are assigned to one of two rooms: Room A or Room B. They then play a two-stage game. In the first stage, participants in Room A are given some money and have to decide how much to send to participants in Room B. In the second stage, the recipients in Room B get triple the amount of money that was sent to them. They then must decide how much money to keep for themselves, and how much to send back to those in Room A.","Participants should maximize their own pay-off (get as much money in the end as possible). Those in Room A must balance the possible benefit of earning more money (since all the money they give to Room B is tripled) with the risk that the people in Room B would keep all the extra money for themselves, leaving those in Room A with nothing. Conversely, those in Room B must decide whether they will keep all of the money for themselves, or give a portion of it back to Room A. Participants are given a limited amount of time (e.g., 10 minutes) to send money back and forth."
Investment game (hidden-profile),McLeod et al. 1997,https://doi.org/10.1037/0021-9010.82.5.706,"Participants are asked to take the role of the board of directors of an investment company. Their task is to evaluate three companies available for acquisition using a specified set of investment criteria and to provide a rank ordering of the companies' desirability. Profiles on each of the three companies include the financial opinions of in-house and external financial analysts, information on the company's business strategy, strength of its management team, market position, and human resources practices. The investment criteria for evaluating the companies include long-term financial return, the degree of risk tolerated, the ability of the company to stand on its own and fit with the company's general business philosophy.
Each participant is given different information about the investment options, which means that initial majority and minority opinions are likely to be divided. Participants are then asked to discuss the investment strategy. After they discuss, the group must come to an agreement on how to rank-order the three companies from best to worst investment.","The goal for each group is to reach an agreement on the correct rank ordering of the three companies based on the information they receive. The task is designed so that the full information would show that one company is objectively the best, one is in second place, and one is objectively the worst. Through discussion and sharing of information, the group should find out the ""correct decision"" within the time limit."
Iterated Snowdrift Game (With Punishment),"Jiang, Perc, and Szolnoki 2013",https://doi.org/10.1371/journal.pone.0064677,"Participants are randomly paired in a group acting as two drivers who are caught in a blizzard and trapped on either side of a snowdrift. They can either get out and start shoveling (""cooperate"") or remain in the car (""defect"").
There are two stages in this game. During the first stage, participants choose to cooperate or defect. If both cooperate, they have the benefit of getting home while sharing the labor. Thus, Rewards = (benefit - labor)/ 2. If both defect, they do not get anywhere and hence incur the punishment, meaning Rewards = 0. If only one shovels, however, they both get home; however, the defector is able to free-ride and avoids the labor cost while reaping all the benefits (Rewards = benefit). In contrast, the cooperator gets fewer Rewards, as they have to put in all the labor (Rewards = benefit - labor). We assume benefit is greater than labor, as both parties want to get home.
During the second stage, the cooperator is given the chance to punish the defector for their free-riding behavior. Punishment works as follows: the cooperator has to incur a fixed fee (1 unit) in order to punish defectors. If they choose to incur the fee, then the payoffs for the defectors are reduced by a fine. Note, however, that while the fine can take different values, the fee for punishing does not change. The possibility that someone might punish you if you defect therefore makes the option of defection more risky.","Participants are instructed that their goal is to maximize their individual payoff (Rewards) and are given a limited amount of time (e.g., 10 minutes) to make a decision during the first stage. Letting the opponent do all the work is the best option for your individual payoff, but if the other player stays in the car, it is better to shovel. The worst outcome is if both players choose to stay in the car.
For example, imagine that there are three groups in the game. In Group 1, both player A and player B decide to cooperate. In Group 2, player C remains in the car while player D gets out and starts shoveling. In Group 3, neither player E nor player F is willing to cooperate. The payoff values of different players rank in order: player C > player A/player B > player D > player E/player F.
However, participants should take the fine and potential cost of punishment into account in this game. The cooperator has the chance to punish the defector, which means if the fine is heavy, it is not worth it to defect."
Iterated Snowdrift Game (Without Punishment),"Jiang, Perc, and Szolnoki 2013",https://doi.org/10.1371/journal.pone.0064677,"Participants are randomly paired in a group acting as two drivers who are caught in a blizzard and trapped on either side of a snowdrift. They can either get out and start shoveling (""cooperate"") or remain in the car (""defect"").
If both cooperate, they have the benefit of getting home while sharing the labor. Thus, Rewards = (benefit - labor)/ 2. If both defect, they do not get anywhere and hence incur the punishment, meaning Rewards = 0. If only one shovels, however, they both get home; however, the defector is able to free-ride and avoids the labor cost while reaping all the benefits (Rewards = benefit). In contrast, the cooperator gets fewer Rewards, as they have to put in all the labor (Reward = benefit - labor). We assume benefit is greater than labor, as both parties want to get home.","Participants are instructed that their goal is to maximize their individual payoff (Rewards) and are given a limited amount of time (e.g., 10 minutes) to make a decision. Letting the opponent do all the work is the best option for your individual payoff, but if the other player stays in the car, it is better to shovel. The worst outcome is if both players choose to stay in the car.
For example, imagine that there are three groups in the game. In Group 1, both player A and player B decide to cooperate. In Group 2, player C remains in the car while player D gets out and starts shoveling. In Group 3, neither player E nor player F is willing to cooperate. The payoff values of different players rank in order: player C > player A/player B > player D > player E/player F."
Iterative lemonade stand task,"Sommer, Bendoly, and Kavadias 2019",https://doi.org/10.1287/mnsc.2018.3247,"Participants are given an interactive online interface that represents managing a lemonade cart. The program allows users to tweak five variables: (1) price, (2) lemon content, (3) sugar content, (4) lemonade color, and (5) location of lemonade cart. There are limited options for lemonade color and cart location that are predefined within the task. For example, the lemonade can only be either yellow or orange, and a cart can either stand in the park or near the supermarket. Price can be any number the participant decides to use, while lemon and sugar content can be any value between 0 and 100 percent.","The goal is to maximize the cart's profit by adjusting the five given variables within a fixed amount of time. For example, participants can decide to sell their orange lemonade in the park with 20% lemon content and 5% sugar content for $3 a glass. There is a formula in the program that quantifies the cart's profit and shows participants instant feedback. Participants will not be told that they have reached the maximum possible profit; rather, they simply see feedback about the current profit and are able to adjust as many times as they want before they submit.
Participants are evaluated based on their profit. The more profit their cart makes after they tweak the variables, the better their performance is on the task."
Letters-to-numbers problems (cryptography),Laughlin et al. 2006,https://doi.org/10.1037/0022-3514.90.4.644,"Participants are given instructions to map letters A, B, C, D, E, F, G, H, I, and J, which have been initially randomly assigned without replacement, to one of the 10 digits (0 through 9). For example, because letter ""A"" maps onto digit ""3"" and letter ""B"" maps onto digit ""5"", an operation ""A + B"" would correspond to ""3 + 5"". If letter ""C"" would map onto digit ""8"", then A + B = C.","The goal is to identify the mapping of the 10 letters to the 10 numbers in as few trials as possible within a fixed amount of time and a limited number of trials. Participants must also get the correct answer as quickly as possible.
There is a specific process participants have to follow to map letters onto numbers. Participants are first asked to come up with an addition or subtraction operation (e.g., ""A + B"") and then given an answer in the letter form (e.g., ""A + B = C""). Then participants propose a mapping for a letter (e.g., ""A = 3"") and receive feedback (e.g., ""True, A = 3""). Participants have to follow this process in proposing the mapping of all letters to all digits. To submit their final answer, participants have to indicate which digits each of the letters correspond to.
Participants are evaluated based on several criteria, such as the number of trials they used, letters identified per trial, and letters used per equation. There is a theoretical maximum score, and participants get the highest score by finding the correct mapping as fast as they can, with as few trials as possible."
Logic Problem,Littlepage 1991,https://doi.org/10.1177/0146167291174014,"Participants receive a logic problem, which is a story problem with a correct answer that must be deduced via logic. This problem is shared among everyone who is working together.
Here is an example question: There are five couples who vacationed at five different ski resorts. Based on the clues, figure out which couple vacationed where. Participants would get a set of clues to solve this problem, such as ""Neither Tammy nor Sue vacationed in Innsbruck,"" ""Both Rita and Mrs. Owens returned from their vacations with broken wrists,"" ""no one at Tahoe was injured that weekend,"" and so on. Participants are provided with sufficient clues to be able to solve the question.","The goal is to use the clues provided to achieve the correct answer in the logic problem. Participants must solve the problem and submit their answer within a limited amount of time.
In the above example, the participants would use logic to reason that neither Rita nor Mrs. Owens went to Tahoe, since no one at Tahoe was injured. They would then use other clues to match the correct people to the correct vacation locations.
Participants are scored based on the number of correct matches in the final solution that they get. In this case, since there are five couples and five ski resorts, participants would get credit for every pair that they are able to match, and they would achieve the maximum score if they get all five pairs correct."
Mastermind,"Bonner, Baumann, and Dalal 2002",https://doi.org/10.1016/S0749-5978(02)00010-9,"Participants get a series of pegs, or points, of different colors to use in a logic game played on the computer. For example, participants can get 6 colors to use during the game: blue, red, green, orange, yellow, and purple.
As the game begins, an algorithm randomly designates a color pattern as the correct answer to the game. For example, the correct pattern to the game can consist of 4 colors, in this order: yellow, orange, yellow, red.","The goal is to identify all the colors and their positions from the correct pattern, using as few trials as possible and within a fixed amount of time.
As the game begins, participants have a limited number of trials to propose color patterns. Participants receive immediate feedback in the form of black and white pegs after each trial. A white peg means that one of the colors they chose is in the pattern (but in the wrong position); a black peg means that one of the colors they chose has both the color and the position correct.
For example, if the correct pattern is yellow, orange, yellow, red, and a participant proposes blue, green, orange, red, then this participant correctly identifies two colors from the pattern and a position of one of them correctly. Therefore, they would receive a white peg for orange (correct color, wrong position) and a black peg for red (correct color, correct position). However, participants don't know from the feedback exactly which color and position they identified correctly; instead, they have to try to deduce it from feedback for each trial.
Participants try to guess the pattern using as few tries as possible, and get the highest score by receiving as many pegs, especially black ones, as early in the game as possible."
Minimal Group Paradigm (study diversity),Tajfel 1970,https://www.jstor.org/stable/24927662,"Participants are randomly and anonymously divided into two groups (e.g., ""Group A"" and ""Group B""). After they are divided into groups, participants receive an anonymous list of players (e.g., ""participant number 34 of Group A,"" ""participant number 12 of Group B""). They are then asked to distribute a valuable resource (e.g., money or points) between the participants on the list. Participants are told that, after the task is finished, they will receive the total amount of the resource that has been allocated to them by the other participants.","The goal for participants is simply to allocate the resources however they like. There are no right or wrong answers for how participants should divide up the resources. Participants do NOT need to maximize the resources of people on their own team, nor maximize their personal resources."
Minimum-effort tacit coordination game,"Van Huyck, Wildenthal, and Battalio 2002",https://doi.org/10.1006/game.2001.0860,"Participants play in a large group with others. Each participant is given a list of numbers from 1 to 7. In every round, participants must independently choose and submit one of the numbers as their ""guess.""
Each participant's guess will then be compared to the median guess across all participants. Before making their guesses, participants receive a payoff table that explains how much money they would earn for their guess, depending on how close it was to the median, and on the value of the median. In general, participants earn more money if their guess is closer to the median, and if the median is higher (i.e., closer to 7). Participants win less money if their guess is farther away from the median (e.g., the median was 4 and they guess 5), and if the median is lower (i.e., closer to 1). Additionally, participants suffer a greater penalty for ""overshooting"" (guessing higher than the median) than they do for ""undershooting"" (guessing lower than the median).
This process is repeated for a predetermined number of rounds.","Each participant's goal is to maximize their own payoff. The highest possible payoff is if the participant correctly guesses the median, and the median is 7. However, if the true median is lower than 7, the best payoff in the round is to correctly guess the median value and to avoid overshooting. Guessing too high (for example, choosing 7 when the median is 4) would result in a penalty.
Therefore, participants must balance the tradeoff between guessing a large value, which boosts the median and the overall payoff, and guessing a value that is too large, which results in the overshooting penalty. Additionally, there is a tradeoff between individual gain and collective benefit: if a participant chooses 6 or 7, they would boost the median value (which would increase the overall payoff), but they would be more likely to personally overshoot (leading to the penalty). On the other hand, if the participant chooses 1 or 2, they would decrease the overall payoff, but reduce their own chances of overshooting.
Participants must also make each decision in a fixed amount of time."
Mock jury,London and Nunez 2000,https://doi.org/10.1037/0021-9010.85.6.932,"Participants are to act as jurors in a mock jury. Once they have read the summary of the trial, they will be given a brief pre-deliberation questionnaire. They should take into account all facts of the trial and any specific instructions given by the judge before filling out the questionnaire and continuing with the process. In the questionnaire, participants are asked to record their personal verdict for the trial, and their assuredness of this verdict, which are indicated on a 9-point scale. The scale ranged from 1 (definitely not guilty) to 9 (definitely guilty), and 1 (not at all) to 9 (very confident).
After they finish the questionnaire, all the jury members will be instructed on the laws surrounding the case. All participants will then deliberate and attempt to reach a unanimous verdict. The jury is allowed to deliberate for a limited time (e.g. 1 hour) until they reach a verdict.
Then all jurors will once again be given questionnaires where they will render their individual verdict, and record their assuredness of the verdict.",Participants should reach a unanimous verdict with their fellow jury members within a limited amount of time.
Moral Reasoning (Disciplinary Action Case),Wooley et al. 2010,https://doi.org/10.1126/science.1193147,"Participants read a fictitious, controversial case in which a college basketball player bribed an instructor to change his grade on an exam in order to maintain his eligibility on the team.
Participants are also given a list of potential actions that the college could take in response, in which the student and/or instructor could be given a punishment or consequence. Examples include:
(1.) Lowering Student's Grade (2.) Student Suspension (3.) Punishment from Student's Basketball Team (4.) Instructor Punishment (5.) Preventing the Instructor from Getting Future Position and/or Promotion
These possible courses of action are further challenged by conflicting interests of the faculty, college administration, and the athletic department.","Although there is no right or wrong course of action among the options, the goal is to select one of the courses of action.
Participants had to try to make sense of the different stakeholders' conflicting interests, get agreement from team members, and make a final choice about how to handle the disciplinary situation."
NASA Moon survival,Yetton and Preston 1983,https://doi.org/10.1016/0030-5073(83)90144-7,"Participants view information describing a survival scenario in which a spaceship crashes on the moon. The survivors are able to salvage a list of 15 items (e.g., matches, food concentrate, a 50-foot nylon rope, a portable heating unit, two 100-pound tanks of oxygen) from the wreckage. Everyone sees the same list of items. The survivors now need to carry the most important items on a 200-mile trek to get to ""home base.""","The goal is to rank all 15 items in the order of most important to least important for survival, as the astronauts travel from the wrecked spaceship to home base. For example, if the oxygen tanks are the most important for survival, then they would receive the rank of 1. If the 50-foot rope is the least important for survival, then it gets the rank of 15.
Participants' rankings are evaluated based on how close their rankings are to those created by a panel of experts from NASA. That is, the expert ranking is the ""true"" answer, and the closer a participant is to the true answer, the more points they earn. The highest possible score is achieved when participants rank all 15 items in the exact same order as the experts did.
If working in a team, participants are able to discuss the scenario with each other and come up with a single set of rankings as a team."
New Recruit,"Overbeck, Neale, and Govan 2010",https://doi.org/10.1016/j.obhdp.2010.02.004,"Participants are randomly paired for a negotiation over a mock job recruitment scenario. Each person is randomly assigned the role of either the recruit (person being hired) or the recruiter (person doing the hiring). The different roles have different objectives: for example, the recruit may want a higher bonus, while the recruiter would rather pay as low a bonus as possible. The possible settlement options for each participant are associated with point values, which are described in a private ""payoff schedule"" for each participant (explained in more detail below).
Here is an example to help you imagine what the negotiation might look like. Participants would receive a list of settlement options for each issue, and the payoff (in points) associated with each option. They would see only their own payoff schedule, and would not be aware of their counterparts'. Participants would then conduct negotiations over the employment package.","Participants are explicitly instructed that their goal is to maximize their (individual) payoff, and were given a limited amount of time (e.g., 30 minutes) to conduct their negotiation. In order for any agreement to be binding, participants need to reach an agreement on all the issues."
Object based generalization for reasoning (Phyre),Bakhtin et al. 2019,https://proceedings.neurips.cc/paper/2019/file/4191ef5f6c1576762869ac49281130c9-Paper.pdf,"Participants get a series of puzzles related to properties of physical objects in a simulated 2D world on a computer. Access is given to all puzzles at once and participants can choose which puzzles to work on.
For example, in one of the puzzles, participants can view an environment with two blue and green balls on little platforms floating in space, with a slanted floor at the bottom. Considering this scenario, participants are asked to make the green ball and blue ball touch.
Participants are expected to solve puzzles by adding physical objects to the environment so that when the physical simulation is run, the puzzle gets solved. For example, the participant could use an object to knock the blue and green balls off of their floating platforms, and cause them to roll down the ramp and touch (thus completing the puzzle). There are many ways to creatively add objects to the environment and cause different physical consequences.","The goal is to solve as many puzzles in as few attempts as possible within a fixed amount of time. The maximum score occurs when all puzzles are solved during their first attempts.
After each unsuccessful attempt, the environment resets to its initial state and participants can try again until they run out of time, or decide to try a different puzzle."
Oligopoly game,Hemenway et al. 1987,https://doi.org/10.1111/j.1465-7295.1987.tb00776.x,"Participants experience a game with a pre-determined number of rounds. In each round of the game, participants must independently decide whether to ""compete"" or ""collude."" Players get a different amount of payoff in each round based on a combination of how they decided and how everyone else in the round decided. The players know the different possible payoffs ahead of time.
Here is an example. Suppose that the four possible pay-offs are as follows: [You choose - ""Compete"", Majority chooses - ""Compete"", Your Payoff - 10], [You choose - ""Collude"", Majority chooses - ""Compete"", Your Payoff - 0], [You choose - ""Compete"", Majority chooses - ""Collude"", Your Payoff - 40], [You choose - ""Collude"", Majority chooses - ""Collude"", Your Payoff - 20]. In this example, you would get the most payoff if you choose to compete, but the majority chooses to collude.
Let's say that, in the first round, more than half of the players decide to ""compete."" Then those who choose to ""compete"" would receive 10 points. Those who choose to ""collude” would receive 0 points. Then we will play round 2 ,and everyone will vote all over again. Suppose in round 2 that a majority of players choose to ""collude"". Then those who choose to “compete” receive 40 points and those who ""collude"" receive 20 points.","Each participant's goal is to maximize their own payoff. Therefore, when making a decision, participants will have to weigh individual benefits against the unknown actions of other players. Notice that, even though the best possible individual outcome (Payoff = 40) is for you to choose ""Compete"" and for the majority to choose ""Collude,"" if everyone follows this same logic, then the majority will end up being ""Compete,"" and the payoff will only be 10.
Participants must make their decisions in a fixed amount of time."
Organization Game,Krackhardt and Steele 1988,https://doi.org/10.2307/2786835,"Participants are randomly assigned to one of four equal size but physically separated divisions, each having its own complement of operating (line and staff) units. With the minimal structure and only unit heads and resource controllers assigned, participants must decide how to further develop and staff the organization in order to achieve an effective system for the division and coordination of work. These processes must include a way for the organization to learn and change as it adapts to internal and external forces.","The goal is to maximize the organizational effectiveness score based on several objective performance indicators. Notably, the goal is for participants to coordinate and perform well as an entire organization, even though each individual's control is limited to their own unit. Factors that will impact the overall score include the productivity of line units, investments in organizational development programs, and events such as members' absence, strikes, dismissals, resignations, vacations, and underutilization. The indicators for each session of play are calculated by the coordinator and delivered to the information processing unit at the beginning of the next session."
Pharmaceutical Company (hidden-profile),Kelly and Karau 1999,https://doi.org/10.1177/0146167299259002,"Participants are randomly assigned to triads and participate in a role-playing simulation of managers in a pharmaceutical company trying to decide which of two drugs to market. They would be given information sheets about the drugs and be given a limited amount of time (e.g., 10 minutes) to familiarize themselves with the information. However, the information sheets are constructed such that some of the facts were shared (provided to all three members) and some were unshared (provided to only one group member). They would be also told that the company is in financial trouble and that their marketing decision is therefore especially important. After they finish reading the information sheets individually, they would be asked to recall as many of the facts about each drug as they could and be given limited time for the recall task. Following the recall task, the group would be asked to discuss the problem and come to a group consensus about which drug to market.",The goal for each group is to reach an agreement on the drug they would like to market. The task is designed so that the full information would show that one drug of the two is the best. They should find out the optimal drug within the time limit.
Prisoner's Dilemma (various versions),Dawes 1980,https://www.cmu.edu/dietrich/sds/docs/dawes/social-dilemmas.pdf,"Participants are given a choice to ""cooperate"" or ""defect"" and are given, in advance, the payoff to be expected for each choice, and when appropriate, the combination of choices between participants. The expected payoffs are the same for all participants. Each participant independently makes their choice without discussion. This process is repeated for a predetermined number of rounds.","Each participant's goal is to maximize their own payoff. For example, let us consider a simple two-player game. If both participants cooperate, they'll both lose a small amount, and if both defect, they will lose a large amount. If one participant cooperates while the other defects, the cooperator will lose a large amount while the defector loses nothing. In this scenario, the participants must balance the temptation of defecting and the risk of their opponent doing the same to minimize their losses. Participants must also make each decision in a fixed amount of time."
Public goods game,Tomassini and Antonioni 2020,https://doi.org/10.3389/fphy.2020.00058,"Participants receive the same amount of tokens and they secretly choose how many of their private tokens to put into a public pot. The tokens in this pot are multiplied by a factor (greater than one and less than the number of players, N) and this ""public good"" payoff is evenly divided among players. Each participant also keeps the tokens they do not contribute.
Here is an example to help you imagine what the game might look like. A group of four players participates in this game and starts with $20 in hand. Three players contribute their full $20 while the fourth chooses to contribute $0. The $60 is multiplied by a factor of 1.2 and the resulting $72 is distributed equally among the four players (this means everyone gets $18 from the pot). In this scenario, the three players contributing the most end the round with $18, while the fourth player gets as much as $38 (they kept their initial $20, and earned an additional $18).","Participants are instructed that their goal is to maximize their individual payoff as well as the group’s total payoff. It means that one goal is to maximize their personal earnings, which means that they should not invest anything, and simply ""free ride"" off of the others, as in the above example. But another goal is to optimize the group’s outcomes, which means that everyone should contribute all of their tokens to the public pool. If everyone in the example had contributed their full $20, then there would have been $80 in the pool, and each player would be able to take home $24. Participants should try to maximize both individual and group outcomes, which causes a tradeoff. They are given a limited amount of time (e.g., 10 minutes) to play the game."
Putting food into categories,Choi and Thompson 2006,https://doi.org/10.1016/j.obhdp.2005.06.003,"Participants were provided with a shared list containing several (e.g., a dozen) fruit and vegetable items. For example, you might have a list with the words, ""orange,"" ""grapes,"" ""peach,"" ""apple,"" etc.
Everyone in the team sees the same list, and the contents of the list stay the same for a given round.","The goal was to generate as many different ideas as possible for categories that could be used to divide the food items.
These ideas are purely abstract concepts meant to demonstrate creativity: e.g., ""how many ways can you think of to put these food items into different categories?"" For example, they could divide the fruit into citrus vs non-citrus, tropical vs non-tropical, seeds versus no seeds, or any number of possibly wild or imaginative ideas (things you can feed an elephant vs things you cannot feed an elephant).
Participants must come up with as many criteria for dividing the food items as they can within a limited amount of time. To submit, they simply type the ideas into the system. Those working together can see all ideas generated by everyone in the team so far.
Finally, credit is awarded for the total number of non-redundant ideas that participants came up with. That is, participants would only get credit once even if they submit the same idea multiple times, but it is OK if a different team also submitted the same idea."
Railroad Route Construction game,Trager et al. 2020,https://doi.org/10.1073/pnas.1910402117,"Participants see a shared visual of a map, consisting of a grid in which each square is a place where players can place a ""railroad track"" piece. Some parts of the map are blocked off — for example, because there is a tree or other obstacle in that location — and therefore railroad tracks cannot be placed there. There is a ""start"" and ""finish"" marked on the map.
Participants also get a collection of several different types of railroad tracks. Some track pieces are bent like an elbow, and there are versions that bend downwards, bend upwards, and so on. Other pieces are straight. Participants are given enough pieces to be able to build a railroad that connects the start and finish.","The goal of the game is to build a railroad using the given puzzle pieces to connect the starting point to the finish point. Participants have to avoid the obstacles (places where you cannot place railroad tracks) — therefore, they need to use a mixture of bent tracks and straight tracks to navigate the map. Any path that runs smoothly from the start to the finish counts as a success, so there is no specific final path or ""ideal"" route.
Participants must finish the game before the time runs out, but are able to visually see their progress on the screen throughout the game (since everyone is clicking, dragging, and interacting with it directly). Participants are evaluated on whether or not they are able to connect the start to the finish: if they do not make it in time, they receive no credit. There is no partial credit for building only part of the railroad."
Railroad Route Construction game (Impossible Version),Trager et al. 2020,https://doi.org/10.1073/pnas.1910402117,"Participants see a shared visual of a map, consisting of a grid in which each square is a place where players can place a ""railroad track"" piece. Some parts of the map are blocked off — for example, because there is a tree or other obstacle in that location — and therefore railroad tracks cannot be placed there. There is a ""start"" and ""finish"" marked on the map.
Participants also get a collection of several different types of railroad tracks. Some track pieces are bent like an elbow, and there are versions that bend downwards, bend upwards, and so on. Other pieces are straight.
Importantly, participants do NOT have enough pieces to successfully build a railroad that connects from the start to the finish. A piece is missing, which makes this game impossible to complete. However, participants are not told that a piece is missing.","The goal of the game is to build a railroad using the given puzzle pieces to connect the starting point to the finish point. Participants have to avoid the obstacles (places where you cannot place railroad tracks) — therefore, they need to use a mixture of bent tracks and straight tracks to navigate the map. Any path that runs smoothly from the start to the finish counts as a success, but since there is a piece missing, there is actually no solution to this puzzle.
Participants aim to finish the game before the time runs out, and are able to visually see their progress on the screen throughout the game (since everyone is clicking, dragging, and interacting with it directly). Participants are evaluated on whether or not they are able to connect the start to the finish: if they do not make it in time, they receive no credit.
However, since there is no partial credit and the game is missing a critical piece, all participants lose. There is no way to win."
Random dot motion,Moussaïd et al 2017,https://doi.org/10.1073/pnas.1611998114,"Participants see an image with many moving dots on the screen. Some of the dots are moving consistently in the same direction (""correlated dots""); other dots are simply bouncing around in random directions (""random dots"").
The percentage of dots moving in the same direction varies, which makes the task easier or harder. For example, if 50% of of the dots are correlated dots, then it is easy to observe the direction that the correlated dots are ""flowing"" in; if only 5% of the dots are correlated, than the picture looks mostly just like dots bouncing randomly, and it is harder to discern the direction that the correlated dots are moving in.","The goal is to correctly determine the direction that the correlated dots are moving in.
Participants submit their answer by using their mouse to place an arrow that points in the main direction of motion. They must submit their answer in a limited amount of time.
There is an objectively correct answer, since the system is designed for the correlated dots to be moving in a specific direction. Participants are evaluated based on how close they are to the objectively correct direction. Their score is calculated based on the angle that their direction forms with the true direction; if they get the direction exactly right, then the angle is 0 degrees, so they would get the highest possible score. If they are off by a small amount (e.g., the angle is 10 degrees), they would get more points than if the participants are off by a significant amount (e.g., the angle is 90 degrees)."
"Rank cities by population, rank words by familiarity",Shaw 1963,https://collections.uakron.edu/digital/collection/p15960coll1/id/21257/,"Participants receive a list of cities (e.g., South Bend, Little Rock, Jacksonville, Portland, Charlotte, Lowell) or a list of words (e.g., Uncle, Kennel, Effort, Money, Village).","The goal is to rank the cities by their population according to the most recent population census, and to rank the words by familiarity to people in the participants' country. The rank of 1 is given to the city with the largest population or to the most common word, the rank of 2 is given to the second largest city or second most familiar word, and so on. Participants must do this task as correctly and accurately as possible and within a fixed amount of time.
Participants are evaluated based on how correct their ranking is compared to the ""ground truth"" data, which is either the population census or the data on familiarity with certain words within a nation. The maximum score is achieved if the ranking is fully correct and the score is maximized by assigning as many correct rank values as possible."
Ravens Matrices,Weidmann and Deming 2020,https://doi.org/10.3386/w27071,"Participants are presented with a pattern, which looks like a matrix or grid with a white background. There is a missing piece to the pattern.
Here is an example to help you imagine what the pattern might look like. Participants could see a grid with four spaces. In the first space they see a square, divided into four pieces, with one piece colored in. Then, in the second space, there is a square with two of four pieces colored in. In the third space, there is a square with three of four pieces colored in. After the first three steps in the pattern, there would be a blank space on the grid.
Participants would then see a few multiple-choice options for what could fill the blank space. For example, they could choose between a square with one, two, three, or four pieces filled in.
All participants working together see the same pattern.","The goal is to complete the pattern by identifying what the missing piece should be.
In the example above, the correct answer is that the fourth space should have a square with four pieces colored in. (That is, successful participants would notice that the first square had one piece colored in; the second had two colored in; the third had three colored in; so, naturally, the fourth should have four colored in.)
The problem is multiple-choice, so participants will choose just one correct answer from a variety of options. There is always exactly one correct answer in the multiple-choice problem. Participants must make their selection within the time limit."
Reading the mind in the eyes,Almaatouq et al. 2021,https://doi.org/10.1073/pnas.2101062118,"Participants were given 36 pairs of eyes. Each pair is a photo of a person, cropped so that only the eyes are visible.
Each pair of eyes is intended to convey some sort of emotion — some of them are squinting with suspicion or wide-eyed with fear; some of them have their eyebrows raised in surprise; others have crinkles at the corners to suggest a smile.
Below the picture of the eyes, participants see four words describing different emotions: for example, ""playful,"" ""sad,"" ""irritated,"" and ""comforted.""","The goal is to correctly label the eyes with an associated emotion by selecting the right one out of the four words.
Each pair of eyes has an objectively correct label out of the four given. Therefore, participants' goal is to answer all questions correctly. They achieve the highest possible score if they label every pair of eyes with the right emotion. Participants have a limited amount of time to finish the test."
Recall association,Takahashi 2010,https://doi.org/10.1348/000712606X101628,"Participants are given a list of several words to study. Example words on the list include ""spider,"" ""foot,"" ""pen,"" ""car,"" etc.","The goal is to recall as many words as possible from the studied list within a fixed amount of time.
Participants receive the maximum score for recalling all words correctly and the minimum for recalling none. Participants maximize their score by recalling as many words as they can."
Recall images,Weidmann and Deming 2020,https://doi.org/10.3386/w27071,Participants receive a certain number of images to memorize on individual screens.,"The goal is to memorize all target images within a fixed amount of time and to be able to identify them later from a lineup of old and new images.
For example, participants can be given 20 seconds to memorize six target faces and then asked to identify those target faces at a later time. At the later time period, they could be consecutively shown 15 sets of three faces (6 previously seen, and 39 new), and asked to pick out which faces were the images that they had previously seen before.
Participants are evaluated on the number of correctly recalled images. They earn the maximum score when they recall all images correctly (e.g., select the 6 original faces out of 45). This means they need to correctly identify all the images that they had seen before, and avoid mis-identifying a new image as something they had previously seen."
Recall stories,"Johansson, Andersson, and Rönnberg 2005",https://doi.org/10.1111/j.1467-9450.2005.00465.x,"Participants read one or more stories that are each approximately 4 pages in length. After reading each story, they advance to a different page (where they are no longer able to reference the story), and they see multiple (e.g., 12) reading comprehension questions accompanying the story. Every question has an objectively correct answer based on the text.","The goal is to correctly answer knowledge questions about the stories within a fixed amount of time.
Participants are evaluated on the number of correctly answered questions and they earn the maximum score when they answer all questions correctly. Since they cannot reference the stories, they must answer the questions based on their memory of what they read."
Recall videos,Engel et al. 2014,https://doi.org/10.1371/journal.pone.0115212,Participants are shown a 90-second video several times.,The goal is to correctly answer a set of questions about what occurred in the video within a fixed amount of time. Participants receive points for each correct answer and maximize their score by correctly answering as many questions as they can.
Recall word lists,Takahashi 2010,https://doi.org/10.1348/000712606X101628,"Participants hear about a list of words. For example, they could listen to a tape of a male voice reading several words at the rate of one word per second.","The goal is to remember the list of words and then write down as many words from the list as possible.
Participants are required to start their list with the last word they heard, or else it is not counted. Otherwise, words can be written in any order. Participants have a limited amount of time to recall all the words they can.
The maximum score occurs if participants recall all words correctly. They get zero points for recalling no words, or for failing to start the list with the final word. Participants lose points for incorrectly recalling words that were not on the list. If working in a team, participants can collaborate to submit the final list of recalled words."
Reproducing arts,Wooley et al. 2010,https://doi.org/10.1126/science.1193147,"Participants are given a shared online spreadsheet tool that can be edited, as well as a shared, static copy of a spreadsheet with each cell colored in a different way (the ""target"" spreadsheet).","The goal is to reproduce the ""target"" spreadsheet in the editable spreadsheet. That is, within a fixed amount of time, participants have to make the shared, editable spreadsheet look like an exact copy of the pattern of colors in the ""target"" spreadsheet.
Participants receive points for coloring a cell correctly and don't receive points for coloring a cell incorrectly. They get the maximum score on the task if they reproduce the exact copy of the ""target"" spreadsheet and they try to get the highest score by correctly coloring as many cells as they can."
Room assignment task,Almaatouq et al. 2021,https://doi.org/10.1073/pnas.2101062118,"Participants are given a set of people (students), rooms, and rules (constraining conditions).","The goal is to assign students to rooms within a fixed amount of time in order to maximize the students' utility as much as possible while also respecting all of the constraining conditions (for example, one rule might say that certain people are not allowed to be in the same room). Students get a specified amount of utility from being assigned to a particular room, which translates to points in the game.
Participants will be shown the running total of how many points their current submission has earned, but they will not be notified if they have achieved the optimal arrangement. If participants' submission violates some of the constraining conditions, they will receive a score penalty."
Run a mini business,Shaw 1963,https://collections.uakron.edu/digital/collection/p15960coll1/id/21257/,"Participants are given information on running a tinker toy manufacturing company. The information is organized in the form of specialized tables that have parameters for the tinker toys, as well as details on assembling different models, order forms, costs, and prices. In addition, the tables reflect how costs and selling prices fluctuate over time.
Participants can view these tables and run their business through an interface on the computer. In the beginning, participants are given a base pay to start off their business. As participants play, they can lose and gain money.","The goal is to make as large a profit as possible from running the business for a fixed amount of time.
Based on the information provided to them, participants can ""buy"" parts, ""manufacture"" products, and ""sell"" them at the price they determine. Participants are evaluated based on how much money they have by the end of the game's time limit. The less money they lose and the more money they make, the more successful their performance is on the task."
Search for Oil Task,Isenberg 1981,https://doi.org/10.1016/0030-5073(81)90042-8,"Participants view a shared 12x12 grid of points, creating a digital ""map"". Each point represents a section of ""land"" in a theoretical oil field. A limited number of these points have been pre-determined by the makers of the game to contain ""oil,"" while others contain nothing.
Participants are also given some information about which points might be more likely (but not certain!) to contain the oil. For example, they are told that some parts of this grid represent land with the appropriate chemical composition, making it more probable that there will be oil there. (Other pieces of information that make oil more or less likely include information about the surface hardness, surface mantle thickness, and geological stratification of the area.)
After being shown the detailed information for a limited amount of time, participants have to remember as much information as they can and then decide where they want to ""drill"" completely from memory. During the main gameplay, participants see just the 12x12 map, where they are able to click on each point if they want to drill there.
As participants select which locations they want to drill, they are given feedback after each decision. For example, after participants pick a point on the 12x12 grid to drill oil, they will be told whether they were right or wrong about oil being there, and they will see the running total of their score.","The goal of the task is to decide which of the 144 sections of land is suitable for drilling oil, and to do so as accurately as possible before the game's time limit runs out. In other words, participants have to remember and account for all the information they were provided about where oil is most likely to be found, and then decide where they want to ""drill"" (by clicking on the point on the map).
Participants aim to get the highest possible score, and they earn points for every location with oil that they successfully find. They lose points if they select locations that do not have oil. Therefore, one would get the maximum score of this game by correctly finding every point with oil without making any mistakes."
Sender-Receiver game,Gneezy 2005,https://doi.org/10.1257/0002828053828662,"Participants are paired for a communication-based game where one player sends a message to another, and the second player then chooses an action that determines their payoff. Player 1 is told to send a message to Player 2 about two options for winning different payoffs. The two possible messages are: ""Option A will earn you more money than Option B"" and ""Option B will earn you more money than Option A"". Player 1 can choose to tell the truth or to lie to Player 2, and Player 2 can choose to heed or to ignore the advice of Player 1. The choice of the receiver will determine the payments of both players in the experiment.
Here is a concrete example. Suppose that the true options are: (Option A) Player 1 gets $5 and Player 2 gets $6; (Option B) Player 1 gets $6 and Player 2 gets $5.
Player 1 will see the true options, and they can either choose to tell Player 2 the truth (saying, ""Option A will earn you more money than Option B"") or lie (saying, ""Option B will earn you more money than Option A""). If they tell the truth, and Player 2 listens to them, then Player 1 will end up with less money in the end ($5 rather than $6). On the other hand, if they lie, they can potentially influence Player 2 to (unknowingly) choose the option that is better for Player 1. Meanwhile, because Player 2 knows that Player 1 may be lying, they can choose to either follow the advice or to ignore it. In the end, both players receive the payoffs from Player 2's choice, but Player 2 is not informed of how much Player 1 received (so they do not ever find out whether they were ""correct"" or not.)","Each participant's goal is to maximize their own payoff. Therefore, Player 1 must decide which message will lead to Player 2's choosing an advantageous outcome; Player 2 must decide whether or not to trust Player 1, so that they can obtain their own advantageous outcome. Participants are given a limited amount of time (e.g., 10 minutes) to send a message and make a decision."
Shopping plan,Wooley et al. 2010,https://doi.org/10.1126/science.1193147,"Each participant has a grocery list and a map that shows distances and times between each grocery store, as well as a list of potential items to purchase and how many points they are worth.","Participants have to plan a shopping trip as though they were all residents of the same house sharing the same car. The goal is to purchase as many high-quality items as possible in a fixed amount of time, and to get the highest number of points by considering tradeoffs between price, quality, and driving time. Participants gain points for every item that they are able to plan to purchase. Getting anything less than the maximum number of points will lead to partial credit."
Space Fortress,Arthur et al. 1993,https://doi.org/10.1016/0001-6918(94)E0055-K,"Participants see a video game, in which a ""space fortress"" is at the center of the screen. The fortress can rotate in all directions and fire off ""shells"" at anyone who attempts to attack it.
In this game, each subject controls a spaceship, which is able to fly around freely and shoot missiles at the space fortress. As they approach the space fortress, subjects must also dodge ""mines,"" which periodically appear on the screen and chase the subject's ship. If a mine hits the subject's ship, the spaceship will suffer damage. However, some special floating objects — which appear to be mines — will actually give the subject a ""power-up"" and damage the space fortress instead.","The goal of the game is to maximize points by destroying the space fortress. Participants need to attack the space fortress as much as possible while avoiding damage from enemy attacks and mines. This game can be either single-player or multiplayer; in either case, the objective is the same.
To summarize, participants earn points when:
• They successfully hit the fortress;
• The fortress is destroyed (which requires hitting it repeatedly and progressively weakening it);
• Participants earn a bonus in the game (e.g., through a power-up);
• The participants destroy or neutralize a mine.
On the other hand, participants lose points when:
• Their ship is damaged (e.g., by an enemy shell or a mine);
• Their ship is destroyed (which occurs after the ship suffers too much damage);
• The participant runs out of missiles.
Participants are evaluated on their final game score."
Splitting a deck of cards,Laughlin and Shippy 1983,https://doi.org/10.1037/0022-3514.45.1.94,"Participants are shown cards from a standard deck, one by one. As the participants see the cards, they are told whether the card belongs to a ""rule"" or not.
From the instructions, participants know that there is a rule dividing the 52 cards into two categories, but they do not know what the rule is. The rule can be based on the numerical value of cards; their color, suit, logical connectives, alternation, or any combination of these. For example, a rule could be ""the card is divisible by 3,"" or ""two black cards and one red card alternate.""
The fist card card shown is always an example of the rule (it's an exemplar). As participants see more cards, they will see some exemplars, and some cards that do not belong to the rule (non-exemplars).","The goal is to correctly figure out the rule dividing the deck of cards in as few trials as possible and within a fixed amount of time.
As participants keep seeing new cards one-by-one, they make proposals as to what the rule is and receive immediate feedback. Cards that have been shown get sorted into exemplars and non-exemplars and stay in front of participants so that they can see patterns. To get a better score on the task, participants have to understand what the pattern is as fast as they can."
Sudoku,Engel et al. 2014,https://doi.org/10.1371/journal.pone.0115212,"Participants see a shared online system that displays a Sudoku puzzle.
Sudoku is a logic puzzle, in which there is a 9x9 grid, which is divided into nine 3x3 subgrids (also known as ""boxes"" or ""regions""). Each cell in a region has a true underlying value from 1-9; however, most of the cells appear empty to the player, and only a few of them have the correct numbers filled in.","The goal is to solve the Sudoku puzzle within a limited amount of time. This means that participants need to fill in each blank cell with the correct underlying number from 1-9; in addition, they must satisfy the constraint that every row, every column, and every 3x3 region can only show each digit from 1-9 exactly once.
For a given Sudoku puzzle, there is only one correct answer. Participants either correctly identify the underlying value for every cell, or they fail the Sudoku game.
Finally, participants working in groups may use a chat function to discuss which value should be placed in a given cell."
Summarize Discussion,"Hackman, Jones, and McGrath 1967",https://doi.org/10.1037/h0024647,"For this task, participants are given an open-ended discussion statement about a specific topic. The statement may be a question like, ""what makes for success in our culture?"" or ""should birth control be made available to anyone without a prescription?"" Everyone in the same team gets the same question.","Participants are asked to talk about and submit a response to the discussion statement — that is, they need to turn in a written summary of their arguments or answers to the question given to them. (For example, ""Yes, birth control should be made available to anyone for three reasons, Number 1…"" or ""No, we should not make birth control available to everyone. You should need a prescription because …"") If people disagree, the summary needs to resolve and synthesize the pros and cons.
Participants have a specific time limit for completing the writeup. They are graded on the quality of their final written summary."
Target Search,Shaw 1963,https://collections.uakron.edu/digital/collection/p15960coll1/id/21257/,"Participants view a labeled spreadsheet with 100 squares on it. The columns are given letters and the rows are given numbers (so, each blank cell has a letter-number coordinate, like A5, E10, etc.).
While these cells on the spreadsheet appear blank to the participant, each one is actually connected behind the scenes to a specific value that only the experimenter can see. On the experimenter's side, there is essentially an invisible ""target"" — the cells are grouped into concentric circles. This ""target"" shape determines the point value of each cell. Cells located in the middle of the target are worth more points than cells near the edges of the target. The closer to the center of the target a cell is, the more points it is worth. If a cell is outside of the target entirely, it is worth 0 points.
For example, imagine that E10 is the center of the target. Then E10 is worth the most points. A cell that is next to the target (E11) is worth the second-most number of points, whereas a cell that is far away from the target (e.g., A5) would be worth fewer points (or zero, if it is outside the target).
Of course, since the participants do not see the target, they have no idea how many points each cell is worth.","The goal is to get the most points. In other words, participants need to find the coordinates of as many high-value cells as they can within a fixed amount of time and under a limited number of allowed guesses.
For each guess, participants submit the coordinate of a blank cell (e.g., A5) and then get feedback about how many points they received. Based on feedback, they would try to figure out where the experimenter's underlying ""target"" is and look for the high-scoring cells. If they are working in teams, participants work together to determine their next guess.
After a few guesses, participants can often ""decipher"" the logic of the sheet and will then be able to name the cells corresponding to the largest values. This is why this game is called ""target search"" — participants are using each guess to learn about where the target is, and to search for the highest-value cells.
Participants are evaluated based on the values of the cells they name. The score is maximized by naming as many cells with as large values as possible. Theoretically, the best possible score would be to find the center of the target on your first try and then only name the highest-value cells, but this is not likely to happen; most participants will experience some trial and error."
The beer game,Chen and Samroengraja 2009,https://doi.org/10.1111/j.1937-5956.2000.tb00320.x,"Participants are assigned to a group of four, in which each player has a different role: (1) retailer, (2) wholesaler, (3) distributor, and (4) factory. Customer demand (in kegs of beer) arises only at the retailer, which replenishes its inventory from the wholesaler; the wholesaler gets its beer from the distributor; and the distributor gets its beer from the factory, which produces the beer. The customer demand in different periods is a random variable, independently drawn from a distribution that is known to all players. The beer flows upstream to downstream (i.e. from the factory, to the distribution center, to the warehouse, and finally to the retail store). In contrast, the information about the customers' orders flows in the opposite direction (it goes from the retail store back up to the factory). Both the material and information flows are subject to delays. For example, there is an order processing delay, or information lead time, between when an order is placed and when the supplier receives the order.
Here is an example. Suppose the retail store orders 10 kegs from the warehouse on Monday. The warehouse receives this order on Wednesday. This information delay is due to the administrative steps in processing an order. On Wednesday, however, the warehouse only has 5 kegs of beer, so it ships 5 kegs to the retail store and backlogs the remaining 5. This shipment of 5 kegs arrives at the retail store on Friday. This delay is due to transportation.
In each period, the channel members must decide how much, if any, beer to order from their respective suppliers. The factory must decide how much, if any, beer to produce.","Participants are instructed that their goal is to minimize the total holding and backorder costs incurred in the entire supply chain and optimize the system-wide performance. In other words, the goal is for each member to choose the right amount of beer to order or produce, despite the lag in information. If participants get the number wrong, they suffer a penalty. If they have too much beer, the wholesaler, distributor, and factory suffer a ""holding cost,"" because it is expensive to store the extra inventory. If they have too little beer, the retailer suffers a ""backorder cost,"" because it is expensive to lose customers when the store does not have the beer people want. In an ideal world, if the entire supply chain operates perfectly smoothly, the cost incurred would be zero."
The Fish game,Krafft et al. 2015,https://people.csail.mit.edu/pkrafft/papers/krafft-et-al-2015-emergent.pdf,"The participants see a shared virtual canvas with many different cursors moving around. Each participant controls their own cursor (which is denoted by a unique color and the word ""YOU""). Participants are able to use the controls to move their cursor and ""walk"" around the virtual space.
An underlying system sets a ""point value"" for each location in the space. That is, walking to specific locations in the virtual space will earn the participant more points. Participants will be able to see how many points their current location gives them, and they will also see the running total of how many points they earned so far. However, participants do not know where the high-point-earning locations are, so they must wander around the virtual room. Participants lose all their points if they touch the wall.","The goal is to maximize points in the game by discovering and moving your cursor to the highest point-earning locations in the virtual space. Participants try to get as many points as possible before the time limit expires.
This essentially involves a bit of trial and error, since participants only see how many points their current location is worth. There is also an element of understanding risks and tradeoffs: participants have to decide whether to explore other locations in the room (at the risk of only finding lower-point areas) or just stay put and earn however many points their current location is worth (at the risk of not finding the highest point-earning location). When playing with others, participants may get a sense of where to go by observing where other people in the room are heading."
The N light bulbs game,Yahosseini and Moussaïd 2020,https://doi.org/10.1038/s41598-020-59946-9 ,"Participants see 10 light bulbs.
Some of the light bulbs can be turned ""on"" or ""off"" by clicking on them. Other light bulbs are stuck — they are either on or off, and they cannot be changed. The light bulbs that are stuck are marked with a big ""X.""
Different patterns of light bulbs are worth different numbers of points. For example, maybe an alternating pattern — on, off, on, off, etc. — has the highest points for this round of the game. However, participants do not know that this is the best pattern.","The goal is to get the most points by setting the lightbulbs to a pattern that maximizes the point value.
This means participants have to switch the light bulbs (that are not stuck) on and off to try to figure out which configurations have the highest payoff. For each guess, participants only get to change one light bulb (turn it ""on"" or ""off"") at a time. There are a limited number of guesses. If working in teams, everyone collectively determines the next guess.
The computer will give feedback and tell the participant how many points they earned after each guess. While there is a theoretical ""best"" configuration, participants will not be told if they have achieved it; they only know how many points the guess is worth. It's possible, for example, that they make a guess that gets to the optimal point value, but then make another guess that performs worse. Participants also have the option to end the round early if they are happy with their guess.
A final useful piece of information is that the lightbulbs can be thought of as binary variables (a 0 or 1), which would then allow this game to be played by an algorithm."
To evacuate or not to evacuate,Shirado et al. 2020,https://doi.org/10.1098/rspa.2019.0685,"Participants read about a situation in which 'a disaster may or may not strike.' They are given information about the risk level of the disaster, as well as a button to ""evacuate"" from the disaster.
Evacuating from the disaster costs money — participants lose half their bonus if they choose to evacuate. However, if they don't evacuate and the disaster happens, then they lose their entire bonus. Therefore, this task is about weighing up the different risks and rewards of clicking the ""evacuate"" button.
Participants only find out whether the disaster happened or not at the very end of the game. All participants are equally affected by the disaster, but they make their selection for evacuation individually.","The goal is to make a decision regarding whether or not to evacuate. Participants have a limited amount of time to make their decision.
Choosing NOT to evacuate means that there is high risk (you could get nothing), but potential for a high reward. On the other hand, choosing to evacuate has a low risk but a smaller reward."
TOPSIM - general mgmt business game,Frick et al. 2017,https://ideas.repec.org/p/pdn/dispap/23.html,"Participants are randomly assigned to a team and manage a fictitious company. They compete against other teams which manage other companies in the same simulated market. Over each period (the eight weeks), teams have to develop and adapt business strategies based on the basic endowments in the different functional areas of their company and based on the economic situation in their respective market. In each period, teams have to decide on a number of variables driving company performance: the sales price(s) of the product(s), marketing and sales activities, R&D expenses, investment in environmental facilities or process optimizations, and financial resources. At the end of each period, each company’s periodic share price will be calculated based on how the company as well as its competitors decided on the above-described variables given the predetermined endowment and economic circumstances. The group then receives a full report documenting how the team performed as well as an updated economic forecast for the subsequent period in order to make new business decisions.","Participants are instructed that their goal is to maximize the firm’s share price at the end of the eight-week period. The final share price translates into a team grade with the best possible grade going to each market’s best-performing company. The remaining companies are graded relative to the best performers. In other words, the highly competitive setting incentivizes teams not to cooperate with competitors."
Trivia Multiple Choice Quiz,Wooley et al. 2010,https://doi.org/10.1126/science.1193147,"Participants receive a series of multiple choice trivia questions. For example, participants can be asked about the average weight of an elephant and be given options of 9,000 lbs, 5,000 lbs, and 15,000 lbs.",The goal is to correctly answer as many questions as possible from a multiple choice trivia quiz within a fixed amount of time and without looking up the solutions online. Participants don't get any points if they answer a question incorrectly and they get points for choosing the correct answer. Participants maximize their score by answering as many questions correctly as they can and get the maximum score if they answer all questions correctly.
Typing game,Wooley et al. 2010,https://doi.org/10.1126/science.1193147,"Participants see an image of a document — we can call this the ""Document to Copy."" This document contains several passages of relatively complicated text.
The Document to Copy cannot be edited, copied, or pasted. Everyone working together sees the same document.
Participants also have a shared, editable document, where they are able to type.","The goal is to exactly reproduce the words from the Document to Copy in the editable document. The focus is on typing words correctly: participants earn points for every word they correctly type, and they lose points for skipping words or for making typos.
Participants have a limited amount of time to type as many words from the Document to Copy as they can.
If working in a group, everyone types into the same shared document, and therefore team members must also coordinate to avoid typing over each other or missing certain sections."
Ultimatum game (various versions),Bornstein and Yaniv 1998,https://doi.org/10.1023/A:1009914001822,"Participants are placed into pairs and randomly assigned the role of ""proposer"" or ""responder."" The proposer will receive an endowment that they may split with the responder. The proposer will make a decision as to how they would like to split it, which is then shared with the responder. The responder can then choose to either accept or reject. If accepted, the sum is split as was proposed, but if rejected, both players receive nothing. Players are made aware of the payoff arrangement prior to making their decisions. The same pairs play together for a predetermined number of rounds and roles are reassigned each round.","Each participant has the goal of maximizing their earnings. In the proposer role, participants must consider how much they should offer in order to keep as much as possible while ensuring the offer will still be accepted. In the responder role, participants must consider how they will respond to ""unfair"" or ""low"" proposals and the impact this will have on their payoff in the remaining rounds. Both must also consider their reputation and relationship with their partner throughout the game and the impact of these factors on their earnings. Participants must also make their proposals and response decisions in a fixed amount of time."
Unscramble words (anagrams),Engel et al. 2014,https://doi.org/10.1371/journal.pone.0115212,"Participants see a shared list of 24 randomly scrambled words. For example, a word on the list could be, ""SEEMTRIMUM."" This is an anagram (a word created by scrambling the letters) of the word ""SUMMERTIME.""","The goal is to unscramble as many words from the list as possible in a limited amount of time. Each of the scrambled words has exactly one correct answer. Participants obtain the highest score if they are able to unscramble all the words; otherwise, they earn points for every word that they are successfully able to unscramble.
Those working in a team are able to collaborate by discussing the words before they submit."
Visual Oddball Target,Hertz et al. 2016,https://dx.doi.org/10.1037/xhp0000145,"Participants see several groupings of visual stimuli. For example, they might get a series of Gabor patches. These are circular-shaped patterns with black and white bars oriented in the same direction; the patterns are often used in psychological research because they can create an optical illusion, making for a challenging visual perception task. The participants might get several groups of 6 such patches.
Sometimes, one of the visual items might be an ""oddball"" (it looks different than the rest). In this case, if participants see 6 patches at a time, the oddball would be a Gabor patch with different levels of contrast (it's darker) or have a different pattern than the other 5.","The goal is to visually identify when there is an oddball versus when there is not.
The task is focused on visual perception: participants see a grouping, and then need to decide whether one patch looks different than all the others. They submit their decision by pressing a button using their mouse or a keyboard within a fixed amount of time.
For example, if participants see a group of 6 Gabor patches, they would press a mouse or a keyboard button when there is an oddball and press nothing when all patches in the group look the same.
There are only two possible outcomes of task performance. Participants can either correctly identify the oddball or not."
Volunteer Investment Game,Babcock et al. 2017,https://doi.org/10.1257/aer.20141734,"Participants are randomly and anonymously assigned to groups of three in each of the ten rounds. Members of the group are given a limited amount of time (e.g. two minutes) to make an investment decision. Individual earnings are $1 in the event that no one invests before the end of the game. Once one group member makes the investment, the round ends. The individual making the investment secures payment of $1.25, while the other two group members each receive $2. The investor is randomly determined in the event that multiple parties simultaneously invest. Participants are not allowed to communicate with other group members in any way.","Participants are instructed that their goal is to maximize their individual payoff as well as the group's total payoff. It means that one goal is to maximize their personal earnings, which means that they should not invest anything; rather, they should wait for others to invest. But another goal is to optimize the group's outcomes, which means that at least one person in the group should invest. Participants should try to maximize both individual and group outcomes, which causes a tradeoff. They are given a limited amount of time (e.g., 2 minutes) to play the game."
Wason's Selection Task,Wason 1968,https://doi.org/10.1080/14640746808400161,"Participants are given a conditional sentence in the form of, ""If X, then Y."" They also see four cards, which have either a letter or a number on the front. The cards are double-sided; those with a letter on the front have a number on the back, and vice versa.
As an example, imagine that the conditional sentence says the following: ""If there is a D on one side of any card, then there is a 3 on its other side.""
Then, imagine that the four cards are:
• D (front), 3 (back)
• 3 (front), K (back)
• B (front), 5 (back)
• 7 (front), D (back)","The goal is to identify which cards, if turned over, would allow the participant to determine if the conditional sentence is true or false. Participants make the selection by clicking on the appropriate cards on the interface.
Consider the example above: ""If there is a D on one side of any card, then there is a 3 on its other side.""
This means that if there is a card with D on one side, and something other than 3 on the other side, it would make the sentence false. Thus, the answer for each card is as follows:
• Card #1: We should select this card; if there is something other than 3 on the back, then the statement would be false.
• Card #2: We should NOT select this card; whether the other side is D or not does not affect the truth of this statement. If it is D, then the statement is true; if it is not D, it is simply neutral evidence and does not disprove the statement.
• Card #3: We should NOT select this card; the statement does not say anything about cards with letters other than D.
• Card #4: We should select this card; if there is a D on the back, then the statement would be false (since D would be paired with a number other than 3).
Selecting Card #1 is an easy decision: since it has a D on the front, it is easy to see that we should check for a 3 on the other side. However, some of the other choices are more difficult and require skillfully applying logic: for example, participants may not notice that they need to check Card #4 to make sure there is not a D on the other side. Participants may also mistakenly choose cards #2 or #3.
For a given sentence, there is a single correct solution (in the example above, choosing both #1 and #4). If participants correctly select exactly the right cards, then they solve the task correctly. If they do not select enough cards (e.g., picking #1 without picking #4) or if they select too many cards (e.g., picking #1, #2, and #4), then they fail the task."
Whac-A-Mole,"Naber, Pashkama, and Nakayamaa 2013",https://doi.org/10.1073/pnas.1305996110,"Participants see a shared screen with a bunch of colored moving targets (circles) and an ability to click or ""hit"" the targets as they move around. Each participant can control their own individual mouse, but everyone sees the same moving targets.","The targets are worth different numbers of points based on their color. Participants' goal was to hit (e.g., click on) targets based on their point value, thereby maximizing their points. They have a limited amount of time to get as many points as possible."
Wildcam Gorongosa (Zooniverse),"Straub, Tsvetkova, and Yasseri 2023",https://arxiv.org/pdf/2009.11038.pdf,Participants view a series of pictures of animals taken by motion-detecting cameras in Gorongosa National Park in Mozambique. Participants view and interact with the pictures through an online interface on the Wildcam Gorongosa site.,"The goal is to classify as many pictures as correctly as possible according to five criteria within a fixed amount of time. For each picture, participants are tasked with:
(1) detecting the presence of the animal(s), (2) identifying the species of the animal(s), (3) counting how many animals there are, (4) identifying the behaviors exhibited — specifically, identifying whether the animal(s) is (a) standing, (b) resting, © moving, (d) eating, or (e) interacting (multiple behaviors may be selected), and (5) recognizing whether any young are present. Each picture has a ""ground truth"" classification done by scientists, which is used to calculate correctness.
Participants are evaluated based on the total number of pictures they classified out of all possible pictures in the task, as well as the number of correct classifications out of all the classifications they managed to do. This means that both quantity and quality matter. In other words, participants maximize their score by classifying as many pictures as correctly as they can."
Wildcat Wells,Mason and Watts 2012,https://doi.org/10.1073/pnas.1110069108,"Participants see a realistic-looking 2D desert map with dots on it that represent hidden oil fields. All dots look the same to participants, but some ""wells"" are wider and deeper than others and contain more oil, which earns the participant more points. As they play, participants can see the total number of points that they have accumulated throughout the game.","The goal is to drill as much oil as possible within a fixed amount of time by choosing the dots corresponding to ""wells"" that the participant wants to select. Participants earn points proportional to the amount of oil they discover. Thus, the goal is to try to find the ""wells"" with the most oil before the time runs out."
"Wolf, goat and cabbage transfer",Kennedy 2009,https://journal.viterbo.edu/index.php/atpp/article/view/1024,"Participants are given a problem that reads as follows:
""Once upon a time a farmer went to a market and purchased a wolf, a goat, and a cabbage. On his way home, the farmer came to the bank of a river and rented a boat. But crossing the river by boat, the farmer could carry only himself and a single one of his purchases: the wolf, the goat, or the cabbage.
If left unattended together, the wolf would eat the goat, or the goat would eat the cabbage.
The farmer's challenge was to carry himself and his purchases to the far bank of the river, leaving each purchase intact. How did he do it?""","The goal is to achieve the fewest number of trips that get the wolf, goat, and cabbage from one river bank to another, and to solve this problem within the time limit.
There are a few different possible solutions that get to the lowest number. All of the solutions require the participants to take the goat over first, because any other action will lead to something being eaten. Once the farmer comes back, though, there is a dilemma, because if he takes the wolf over, the wolf would eat the goat on the other side; if he takes the cabbage over, the goat would eat the cabbage on the other side.
The ""trick"" to this problem is to realize that the farmer can actually take the goat BACK on his return trip: that is, he takes either the wolf or cabbage with him on the second trip. Then, he takes the goat BACK to the ORIGINAL side. He then picks up the remaining one of the cabbage or wolf, and finally, he goes back with the goat. Realizing that you can actually bring things back and forth (in both directions), rather than just in one direction across the river, usually requires some out-of-the-box thinking.
Therefore, participants must satisfy the constraints of the problem while also thinking creatively. As a final piece of helpful context, this task is actually a well-known one that many people have built algorithms to solve.
To summarize, the solution to this task is:
• Take the goat to the other side
• Return to original side (where the wolf and cabbage are)
• Take either the wolf or cabbage to the other side
• Return to the original side with the goat [THIS IS THE TRICK! Realizing this prevents either the wolf eating the goat or the goat eating the cabbage.]
• Take the cabbage or wolf over to the other side
• Return to the original side, where only the goat remains.
• Take the goat over"
Word completion given part of word,Wooley et al. 2010,https://doi.org/10.1126/science.1193147,"Participants get a set of 36 words with 2-3 letters missing. Each word has a specific correct answer. For example, "" _ ech_ _ que"" would be ""technique"".",The goal is to complete as many words as possible within a fixed amount of time. Participants get the maximum score for completing all words correctly and maximize their score by completing as many of the words as they can.
Word completion given starting letter,Wooley et al. 2010,https://doi.org/10.1126/science.1193147,"Participants are given a starting letter and ending letter for words. For example, the start letter could be ""S"" and the end letter could be ""N."" The starting and ending letters are selected such that there is always a nonzero set of English words that can be created.","The goal is to generate a list of valid English words using the starting and ending letter provided. Using the previous example, participants could be asked to generate words like ""SPIN,"" which starts with ""S"" and ends with ""N."" Participants must generate the maximum number of valid words they can within a fixed amount of time. Nonvalid words will be rejected by the system."
Word construction from a subset of letters,Finlay et al. 2000,https://doi.org/10.1037/0278-7393.26.6.1556,"Participants are presented with nine letters. For example: A, F, H, B, E, J, N, K, D.","The goal is to produce as many valid four-letter English words as possible from a given set of nine letters within a fixed amount of time. For example, out of the set of letters above, words such as ""bean,"" ""head,"", ""khan,"" ""dean"" and many others can be formed. Although there is a theoretical maximum score if participants exhaust all the possible combinations of letters to form four-letter words, the goal is simply to generate as many words as participants can within the time limit.
Participants submit their words through an online system. Words that are not valid in English or that are longer than four letters will be rejected by the system. Those working in teams can brainstorm in any way they wish; the total number of non-repeated words submitted by the team will be counted."
Writing story,"Hackman, Jones, and McGrath 1967",https://doi.org/10.1037/h0024647,"Participants receive a writing prompt. Examples include, ""Write a story about this inkblot,"" or ""Describe this mountain scene."" Everyone in the same team gets the same writing prompt.","The goal is to write a response to the prompt. If working in teams, participants discuss and collectively produce the same story. The work must be submitted online within a fixed time period.
Since this is an open-ended writing task, there are no right or wrong answers, and anything that addresses the writing prompt is a valid answer. Participants are evaluated based on the quality of their responses (for example, the length, writing quality, and level of creativity)."