Accessibility for Color Vision Deficiencies:
Challenges and Findings of a Large Scale Study on Paper Figures

Supplemental material

We performed various analyses and generated many data tables that we are not able to include within the paper for readability reasons. We provide them in this supplemental material.

Additional Figures

Rating behaviour of the workers

How many of the images did they label as accessible, borderline, not accessible?
interactive image
image data

Image categories

Majority and minority votes on issues and helpful aspects per image type.

Histograms of position in our ranking by image type.
Schema:
2D abstract:
2D continuous:
3D abstract:
3D continuous:
GUI:
Undetermined:

All possible DR projections from our tool {#figures-dr}

All, hilbert grid, colored by papertype:

All, hilbert grid, colored by year:

All, no grid, colored by condition:

All, no grid, colored by papertype:

All, no grid, colored by rating:

All, no grid, colored by year:

Issues, hilbert grid, colored by condition:

Issues, hilbert grid, colored by papertype:

Issues, hilbert grid, colored by rating:

Issues, hilbert grid, colored by year:

Issues, no grid, colored by condition:

Issues, no grid, colored by papertype:

Issues, no grid, colored by rating:

Issues, no grid, colored by year:

Helpful, hilbert grid, colored by condition:

Helpful, hilbert grid, colored by papertype:

Helpful, hilbert grid, colored by rating:

Helpful, hilbert grid, colored by year:

Helpful, no grid, colored by condition:

Helpful, no grid, colored by papertype:

Helpful, no grid, colored by rating:

Helpful, no grid, colored by year:

Collected Data

Raw with feedback {#raw-feedback}

We store the raw crowdsourced data in the following form. Each label and the accessibility rating are stored as either 0 or 1. Only one of the possible accessibility values can be 1 in each row. Issues are either there (1) or not (1) ID of Annotator, image, CVD are names and, therefore, strings. Their combination is unique. We stored the unabbreviated CVD strings. Free text feedback from coders is also represented as strings in task_feedback, color_comment_general.

coder	image	cvd	year	accessible	borderline	not accessible	issue₁	...	help_n	feedback
string	string	p/d/t/m	yyyy	0/1	0/1	0/1	0/1	...	0/1	string

Raw {#raw-numeric}

We did not process textual comments from coders for numerical analyses. Thus, throughout most of the paper, we dropped the feedback column:

coder	image	cvd	year	accessible	borderline	not accessible	issue₁	...	help_n
string	string	p/d/t/m	yyyy	0/1	0/1	0/1	0/1	...	0/1

Combined accessibility {#combined-acc}

The accessibility rating is split over 3 columns. For some analyses, it is easier to handle as a single value. Therefore, we also use a merged representation where we transform the rating to a score of 1=not accessible, 2=borderline, and 3=accessible. This corresponds to the practice of interpreting a Likert scale as numbers.

| coder | image | cvd | year | accessibility | issue₁ | ... | help_n | |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: | |string |string | p/d/t/m | yyyy | 1/2/3 | 0/1 | ... | 0/1 |

Aggregated Data

Categorical to rational {#rational-data}

For some analyses, we need numeric values instead of categorical ones.

We start with the data in the Combined accessibility format.
| coder | image | cvd | year | accessibility | issue₁ | ... | help_n | |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: |:---: | |string |string | p/d/t/m | yyyy | 1/2/3 | 0/1 | ... | 0/1 |
We group all ratings by stimulus, cvd and year to average the accessibility and labels. This helps us aggregate over all coders, which are now dropped from the data set.
| image | cvd | year | accessibility | issue₁ | ... | help_n | |:---: |:---: |:---: |:---: |:---: |:---: | |string |p/d/t/m | yyyy | [1..3] | [0..1] | ... | [0..1] |

Since the values for issues and helpful aspects are between 0 and 1, they can be interpreted as the probability that an annotator will notice a specific aspect of the stimulus.

Combined issues and helpful aspects {#combined-issues-helps}

For some applications, we need even simpler data, where we only know the probability of all available issues and helpful aspects to be detected.

We start with the rational data.
| image | cvd | year | accessibility | issue₁ | ... | help_n | |:---: |:---: |:---: |:---: |:---: |:---: | |string |p/d/t/m | yyyy | [1..3] | [0..1] | ... | [0..1] |

For each row, we get separate averages for the issue-related and helpful labels.

image	cvd	year	accessibility	issues	helps
string	p/d/t/m	yyyy	[1..3]	[0..1]	[0..1]

We now have a simple 3-tuple for each of the original 1500 images.

Data for analyses

Correlation matrices

We want to analyze correlations between individual issues, helpful aspects, and the accessibility rating. Therefore, we require rational data, but do not want to distinguish between separate years.

We start with rational data.
| image | cvd | year | accessibility | issue₁ | ... | help_n | |:---: |:---: |:---: |:---: |:---: |:---: | |string |p/d/t/m | yyyy | [1..3] | [0..1] | ... | [0..1] |
We drop the year column. The image name and CVD are sufficient to identify each row.
| image | cvd | accessibility | issue₁ | ... | help_n | |:---: |:---: |:---: |:---: |:---: | |string |p/d/t/m | [1..3] | [0..1] | ... | [0..1] |

Now we can calculate the overall correlation matrix.

Accessibility ranking {#ranking-data}

We want to create a ranking of all images, to know which ones are most and least accessible. We do not know how important each individual issue or helpful aspect is for a ranking and, therefore, do not distinguish between them.

We start with the combined data, where all issue labels and all helpful aspects are merged.

image	cvd	year	accessibility	issues	helps
string	p/d/t/m	yyyy	[1..3]	[0..1]	[0..1]

We drop the year column, as we do not whish to distinguish between years. The image and CVD columns are sufficient to identify each row.
| image | cvd | accessibility | issues | helps | |:---: |:---: |:---: |:---: |:---: |:---: | |string |p/d/t/m | [1..3] | [0..1] | [0..1] |
We group by image, because we do not whish to analyze the rankings for individual CVDs. Data in the other columns is averaged.
| image | accessibility | issues | helps | |:---: |:---: |:---: |:---: |:---: | |string | [1..3] | [0..1] | [0..1] |

Now we can sort the rows by descending accessibility, ascending amount of issues, and descending amount of helpful aspects. This means we favor and image with the same accessibility rating, ...

... but less issues.
... and the same amount of issues but more helpful aspects.

Combined overall ranking.

3-Tuples for neutralization effect

We want to analyze whether helpful aspects could neutralize issues. Using rational values for the accessibility would require a highly complex visualization. Therefore, we want to visualize our data with separate 2D-histograms for each majority rating of accessibility.

We start with the ranking data.
| image | accessibility | issues | helps | |:---: |:---: |:---: |:---: |:---: | |string | [1..3] | [0..1] | [0..1] |
We replace the rational accessibility with the majority vote.
| image | accessibility | issues | helps | |:---: |:---: |:---: |:---: |:---: | |string | [1,2,3] | [0..1] | [0..1] |
We drop the image column, as the visualization will not show it.
| accessibility | issues | helps | |:---: |:---: |:---: |:---: | | [1,2,3] | [0..1] | [0..1] |

Now we have simple 3-tuples for visualization in three separate 2D-histograms.

Further analysis by image type

In the following files you can see the accessibility ratings , the issues and the helpful aspects by image type.

Further comment analysis

All of our comment topics in a single spreadsheet file. It contains different sheets for feedback after training, analysis of other issues and feedback, color comments per CVD.

Labels and topics identified

Additional to the comment analysis, we provide the topics and labels again as separate files from the inital analysis and those extracted from the crowdworkers' comments, also including other topics. That way, we hope that the labels and topics can be used as a checklist more easily.

Image type labeling procedure

We annotated all 1500 original images in a cascading fashion. In the first round, every image was independently coded by two authors of this paper. In the second iteration, all images with disagreement in the content type were again coded by another two authors. The third round of annotations was reserved for images without a clear majority for their type. Again, two authors provided additional image type labels. If there was no agreement in this last round, we labeled the image content as "undetermined". Through this procedure, each image was coded at least twice and at most 6 times while every author provided no more than one annotation per image.

Website / Video of our analysis tool

Our analysis tool is online here: AccessibilityExplorer. There you can also find some interactive visualizations of our paper, that you can configure with individual color schemes. A brief video is here tool video

Accessibility for Color Vision Deficiencies: Challenges and Findings of a Large Scale Study on Paper Figures