# <center>Accessibility for Color Vision Deficiencies:<br/> Challenges and Findings of a Large Scale Study on Paper Figures</center>
## <center>Supplemental material</center>
 

We performed various analyses and generated many data tables that we are not able to include within the paper for readability reasons.
We provide them in this supplemental material.

## Additional Figures

### Rating behaviour of the workers
How many of the images did they label as accessible, borderline, not accessible?<br/>
[interactive image](./figures/rating_behaviour.html)<br/>
[image data](./data/worker_behaviour_anonym.csv)

### Image categories
Majority and minority votes on issues and helpful aspects per image type.<br/>
[![bar charts for issues and helpful aspects per image type'](./figures/issues_helpful image_type.png)](./figures/issues_helpful image_type.png)<br/>

Histograms of position in our ranking by image type.<br/>
Schema:
[![histogram for type 'schema'](./figures/histogram_schema.svg)](./figures/histogram_schema.svg)<br/>
2D abstract:
[![histogram for type '2D abstract'](./figures/histogram_2D_abstract.svg)](./figures/histogram_2D_abstract.svg)<br/>
2D continuous:
[![histogram for type '2D continuous'](./figures/histogram_2D_continuous.svg)](./figures/histogram_2D_continuous.svg)<br/>
3D abstract:
[![histogram for type '3D abstract'](./figures/histogram_3D_abstract.svg)](./figures/histogram_3D_abstract.svg)<br/>
3D continuous:
[![histogram for type '3D continuous'](./figures/histogram_3D_continuous.svg)](./figures/histogram_3D_continuous.svg)<br/>
GUI:
[![histogram for type 'GUI'](./figures/histogram_gui.svg)](./figures/histogram_gui.svg)<br/>
Undetermined:
[![histogram for type 'Undetermined'](./figures/histogram_none.svg)](./figures/histogram_none.svg)<br/>


### All possible DR projections from our tool {#figures-dr}
All, hilbert grid, colored by papertype:
[![all - hilbert grid - colored by papertype](./figures/projection_all_gridified_by_hilbert_colored_by_papertype.svg)](./figures/projection_all_gridified_by_hilbert_colored_by_papertype.svg)

All, hilbert grid, colored by year:
[![all - hilbert grid - colored by year](./figures/projection_all_gridified_by_hilbert_colored_by_year.svg)](./figures/projection_all_gridified_by_hilbert_colored_by_year.svg)

All, no grid, colored by condition:
[![all - no grid - colored by condition](./figures/projection_all_gridified_by_none_colored_by_condition.svg)](./figures/projection_all_gridified_by_none_colored_by_condition.svg)

All, no grid, colored by papertype:
[![all - no grid - colored by papertype](./figures/projection_all_gridified_by_none_colored_by_papertype.svg)](./figures/projection_all_gridified_by_none_colored_by_papertype.svg)

All, no grid, colored by rating:
[![all - no grid - colored by rating](./figures/projection_all_gridified_by_none_colored_by_rating.svg)](./figures/projection_all_gridified_by_none_colored_by_rating.svg)

All, no grid, colored by year:
[![all - no grid - colored by year](./figures/projection_all_gridified_by_none_colored_by_year.svg)](./figures/projection_all_gridified_by_none_colored_by_year.svg)

Issues, hilbert grid, colored by condition:
[![issues - hilbert grid - colored by condition](./figures/projection_issues_gridified_by_hilbert_colored_by_condition.svg)](./figures/projection_issues_gridified_by_hilbert_colored_by_condition.svg)

Issues, hilbert grid, colored by papertype:
[![issues - hilbert grid - colored by papertype](./figures/projection_issues_gridified_by_hilbert_colored_by_papertype.svg)](./figures/projection_issues_gridified_by_hilbert_colored_by_papertype.svg)

Issues, hilbert grid, colored by rating:
[![issues - hilbert grid - colored by rating](./figures/projection_issues_gridified_by_hilbert_colored_by_rating.svg)](./figures/projection_issues_gridified_by_hilbert_colored_by_rating.svg)

Issues, hilbert grid, colored by year:
[![issues - hilbert grid - colored by year](./figures/projection_issues_gridified_by_hilbert_colored_by_year.svg)](./figures/projection_issues_gridified_by_hilbert_colored_by_year.svg)

Issues, no grid, colored by condition:
[![issues - no grid - colored by condition](./figures/projection_issues_gridified_by_none_colored_by_condition.svg)](./figures/projection_issues_gridified_by_none_colored_by_condition.svg)

Issues, no grid, colored by papertype:
[![issues - no grid - colored by papertype](./figures/projection_issues_gridified_by_none_colored_by_papertype.svg)](./figures/projection_issues_gridified_by_none_colored_by_papertype.svg)

Issues, no grid, colored by rating:
[![issues - no grid - colored by rating](./figures/projection_issues_gridified_by_none_colored_by_rating.svg)](./figures/projection_issues_gridified_by_none_colored_by_rating.svg)

Issues, no grid, colored by year:
[![issues - no grid - colored by year](./figures/projection_issues_gridified_by_none_colored_by_year.svg)](./figures/projection_issues_gridified_by_none_colored_by_year.svg)

Helpful, hilbert grid, colored by condition:
[![helpful - hilbert grid - colored by condition](./figures/projection_helpful_gridified_by_hilbert_colored_by_condition.svg)](./figures/projection_helpful_gridified_by_hilbert_colored_by_condition.svg)

Helpful, hilbert grid, colored by papertype:
[![helpful - hilbert grid - colored by papertype](./figures/projection_helpful_gridified_by_hilbert_colored_by_papertype.svg)](./figures/projection_helpful_gridified_by_hilbert_colored_by_papertype.svg)

Helpful, hilbert grid, colored by rating:
[![helpful - hilbert grid - colored by rating](./figures/projection_helpful_gridified_by_hilbert_colored_by_rating.svg)](./figures/projection_helpful_gridified_by_hilbert_colored_by_rating.svg)

Helpful, hilbert grid, colored by year:
[![helpful - hilbert grid - colored by year](./figures/projection_helpful_gridified_by_hilbert_colored_by_year.svg)](./figures/projection_helpful_gridified_by_hilbert_colored_by_year.svg)

Helpful, no grid, colored by condition:
[![helpful - no grid - colored by condition](./figures/projection_helpful_gridified_by_none_colored_by_condition.svg)](./figures/projection_helpful_gridified_by_none_colored_by_condition.svg)

Helpful, no grid, colored by papertype:
[![helpful - no grid - colored by papertype](./figures/projection_helpful_gridified_by_none_colored_by_papertype.svg)](./figures/projection_helpful_gridified_by_none_colored_by_papertype.svg)

Helpful, no grid, colored by rating:
[![helpful - no grid - colored by rating](./figures/projection_helpful_gridified_by_none_colored_by_rating.svg)](./figures/projection_helpful_gridified_by_none_colored_by_rating.svg)

Helpful, no grid, colored by year:
[![helpful - no grid - colored by year](./figures/projection_helpful_gridified_by_none_colored_by_year.svg)](./figures/projection_helpful_gridified_by_none_colored_by_year.svg)

## Collected Data

### Raw with feedback {#raw-feedback}
We store the raw crowdsourced data in the following form.
Each label and the accessibility rating are stored as either 0 or 1.
Only one of the possible accessibility values can be 1 in each row.
Issues are either there (1) or not (1)
ID of Annotator, image, CVD are names and, therefore, strings.
Their combination is unique.
We stored the unabbreviated CVD strings.
Free text feedback from coders is also represented as strings in task_feedback, color_comment_general.
	
| coder | image | cvd | year | accessible | borderline | not accessible | issue<sub>1</sub> | ... | help<sub>n</sub> | feedback |
|:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |
|string |string | p/d/t/m | yyyy | 0/1 | 0/1 | 0/1 | 0/1 | ... | 0/1 | string |

### Raw {#raw-numeric}
We did not process textual comments from coders for numerical analyses.
Thus, throughout most of the paper, we dropped the feedback column:

| coder | image | cvd | year | accessible | borderline | not accessible | issue<sub>1</sub> | ... | help<sub>n</sub> |
|:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |
|string |string | p/d/t/m | yyyy | 0/1 | 0/1 | 0/1 | 0/1 | ... | 0/1 |

### Combined accessibility {#combined-acc}
The accessibility rating is split over 3 columns.
For some analyses, it is easier to handle as a single value.
Therefore, we also use a merged representation where we transform the rating to a score of 1=not accessible, 2=borderline, and 3=accessible.
This corresponds to the practice of interpreting a Likert scale as numbers.
	
| coder | image | cvd | year | accessibility | issue<sub>1</sub> | ... | help<sub>n</sub> |
|:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |
|string |string | p/d/t/m | yyyy | 1/2/3 | 0/1 | ... | 0/1 |

## Aggregated Data

### Categorical to rational {#rational-data}
For some analyses, we need numeric values instead of categorical ones.

1. We start with the data in the [Combined accessibility format](#combined-acc).

	| coder | image | cvd | year | accessibility | issue<sub>1</sub> | ... | help<sub>n</sub> |
	|:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |
	|string |string | p/d/t/m | yyyy | 1/2/3 | 0/1 | ... | 0/1 |

1. We group all ratings by stimulus, cvd and year to average the accessibility and labels.
	This helps us aggregate over all coders, which are now dropped from the data set.
	
	| image | cvd | year | accessibility | issue<sub>1</sub> | ... | help<sub>n</sub> |
	|:---: |:---: |:---: |:---: |:---: |:---: |
	|string |p/d/t/m | yyyy | [1..3] | [0..1] | ... | [0..1] |

Since the values for issues and helpful aspects are between 0 and 1, they can be interpreted as the probability that an annotator will notice a specific aspect of the stimulus.

### Combined issues and helpful aspects {#combined-issues-helps}
For some applications, we need even simpler data, where we only know the probability of all available issues and helpful aspects to be detected.

1. We start with the [rational data](#rational-data).
	
	| image | cvd | year | accessibility | issue<sub>1</sub> | ... | help<sub>n</sub> |
	|:---: |:---: |:---: |:---: |:---: |:---: |
	|string |p/d/t/m | yyyy | [1..3] | [0..1] | ... | [0..1] |
	
1. For each row, we get separate averages for the issue-related and helpful labels.
	
	| image | cvd | year | accessibility | issues | helps |
	|:---: |:---: |:---: |:---: |:---: |:---: |
	|string |p/d/t/m | yyyy | [1..3] | [0..1] | [0..1] |
	
1. We now have a simple 3-tuple for each of the original 1500 images.

## Data for analyses

### Correlation matrices

We want to analyze correlations between individual issues, helpful aspects, and the accessibility rating.
Therefore, we require rational data, but do not want to distinguish between separate years.

1. We start with [rational data](#rational-data).

	| image | cvd | year | accessibility | issue<sub>1</sub> | ... | help<sub>n</sub> |
	|:---: |:---: |:---: |:---: |:---: |:---: |
	|string |p/d/t/m | yyyy | [1..3] | [0..1] | ... | [0..1] |

2. We drop the year column. The image name and CVD are sufficient to identify each row.

	| image | cvd | accessibility | issue<sub>1</sub> | ... | help<sub>n</sub> |
	|:---: |:---: |:---: |:---: |:---: |
	|string |p/d/t/m | [1..3] | [0..1] | ... | [0..1] |

Now we can calculate the [overall correlation matrix](./data/correlation-overall.csv "Click to open").

### Accessibility ranking {#ranking-data}

We want to create a ranking of all images, to know which ones are most and least accessible.
We do not know how important each individual issue or helpful aspect is for a ranking and, therefore, do not distinguish between them.

1. We start with the [combined data](#combined-issues-helps), where all issue labels and all helpful aspects are merged.

	| image | cvd | year | accessibility | issues | helps |
	|:---: |:---: |:---: |:---: |:---: |:---: |
	|string |p/d/t/m | yyyy | [1..3] | [0..1] | [0..1] |

1. We drop the year column, as we do not whish to distinguish between years.
	The image and CVD columns are sufficient to identify each row.

	| image | cvd | accessibility | issues | helps |
	|:---: |:---: |:---: |:---: |:---: |:---: |
	|string |p/d/t/m | [1..3] | [0..1] | [0..1] |

1. We group by image, because we do not whish to analyze the rankings for individual CVDs.
	Data in the other columns is averaged.

	| image | accessibility | issues | helps |
	|:---: |:---: |:---: |:---: |:---: |
	|string | [1..3] | [0..1] | [0..1] |

Now we can sort the rows by descending accessibility, ascending amount of issues, and descending amount of helpful aspects.
This means we favor and image with the same accessibility rating, ...
* ... but less issues.
* ... and the same amount of issues but more helpful aspects.

Combined overall [ranking](./data/ranking-overall.csv "Click to open").

### 3-Tuples for neutralization effect

We want to analyze whether helpful aspects could neutralize issues.
Using rational values for the accessibility would require a highly complex visualization.
Therefore, we want to visualize our data with separate 2D-histograms for each majority rating of accessibility.

1. We start with the [ranking data](#ranking-data).

	| image | accessibility | issues | helps |
	|:---: |:---: |:---: |:---: |:---: |
	|string | [1..3] | [0..1] | [0..1] |

1. We replace the rational accessibility with the majority vote.

	| image | accessibility | issues | helps |
	|:---: |:---: |:---: |:---: |:---: |
	|string | [1,2,3] | [0..1] | [0..1] |

1. We drop the image column, as the visualization will not show it.

	| accessibility | issues | helps |
	|:---: |:---: |:---: |:---: |
	| [1,2,3] | [0..1] | [0..1] |

Now we have simple [3-tuples](./data/3-tuple.csv "Click to open") for visualization in three separate 2D-histograms.

### Further analysis by image type
In the following files you can see the [accessibility ratings](./data/accessibility_by_imgType.csv) , the [issues](./data/issues_by_imgType.csv) and the [helpful aspects](./data/helps_by_imgType.csv) by image type.

### Further comment analysis
All of our comment topics in a single [spreadsheet file](./data/comment_topics_complete.xlsx "Click to open").
It contains different sheets for feedback after training, analysis of other issues and feedback, color comments per CVD.

## Labels and topics identified 
Additional to the comment analysis, we provide the topics and labels again as separate files from the [inital analysis](./data/initial_labels.xlsx) and those extracted from the [crowdworkers' comments](./data/topics_labels_crowdsourcing.xlsx), also including other topics. That way, we hope that the labels and topics can be used as a checklist more easily.

## Image type labeling procedure
We annotated all 1500 original images in a cascading fashion.
In the first round, every image was independently coded by two authors of this paper.
In the second iteration, all images with disagreement in the content type were again coded by another two authors.
The third round of annotations was reserved for images without a clear majority for their type.
Again, two authors provided additional image type labels.
If there was no agreement in this last round, we labeled the image content as "undetermined".
Through this procedure, each image was coded at least twice and at most 6 times while every author provided no more than one annotation per image.

## Website / Video of our analysis tool
Our analysis tool is online here: [AccessibilityExplorer](https://renecutura.eu/visacctool/ "Click to open"). There you can also find some interactive visualizations of our paper, that you can configure with individual color schemes.
A brief video is here [tool video](analysis_tool_video.mp4)