Main navigation

Appendix: How to use an Accessibility Conformance Report

A note on the acronyms:
VPAT stands for "Voluntary Product Accessibility Template." It's the standard document used to demonstrate compliance with WCAG accessibility guidelines. WCAG compliance is a legal requirement for many organizations, including public institutions in Quebec. (Note that Quebec's accessibility standard (the Standard sur l’accessibilité des sites Web) is only available in French.)
Technically, VPAT refers to the empty template. Once it's filled out, the appropriate term is ACR (which stands for Accessibility Conformance Report).

In this resource:

How to evaluate accessibility of licensed products
How to evaluate accessibility of custom or vendor-built websites or applications
- How to do an assessment yourself
- Hiring a consultant to do the assessment

For licensed software projects

If you're working with a licensed software vendor, it's their responsibility to provide an ACR. If a vendor can't provide one, it means their company or product may not be mature. (This is a sign to find another vendor!)

You should get the ACR and review it before signing your contract. As you review it, here are items to take note of.

Questions to ask	What to look for
Who did the evaluation? Was it a member of the company, or an independent accessibility consultant?	An unbiased assessment done by a knowledgeable person. Independent consultants are less likely to be biased, and may conduct more thorough evaluations. If the assessment was done by a staff member at the vendor's company, pay close attention to whether the results seem biased.
Is the evaluation date relatively recent?	A recent assessment that's accurate to the product you're using. If it's more than a few months old, ask the vendor if any changes have been made since the evaluation was done. Is the ACR still accurate? Have issues been resolved (or new features added) since the evaluation? If so, see if they can provide an updated ACR soon.
Which method(s) did they use to evaluate the system?	A thorough approach and a mix of testing methods and technologies. The ACR should include automated, manual, and functional testing methods; information about how much was tested (the entire application? a representative sample?); and a list of assistive technology tools used. Note that automated tests alone aren't sufficient to prove compliance.
Which standard(s) are covered in the report, and which level of success criteria are included?	Coverage of the applicable WCAG standards at the A and AA levels. Quebec law (link available in French only) requires us to be compliant with the latest version of WCAG available in French. At the time of this writing, that means WCAG 2.1 and some aspects of 2.2. Each version builds off of previous versions, so compliance with later versions (like 2.2) means you're also covered for older versions (like 2.0). A compliance report that covers newer versions (even if these are not yet required by Quebec law) can be a good sign that the vendor is planning ahead for accessibility compliance. The WCAG guidelines have success criteria which are categorized under three levels, A, AA, and AAA. "A" is the most basic level of compliance, "AA" is our legal requirement, and "AAA" is the ideal.
Review the "remarks and explanations" column of the success criteria table(s). Does the report: Clearly describe the overall level of performance for the criteria? Indicate any exceptions where the product doesn't match the general level of performance?	An accurate and detailed portrait of compliance with each standard. This description should be detailed enough for you to understand where the standard was tested, how thoroughly it was tested, and why they assigned the rating (such as Supported, Partially supported, Not supported, or N/A).
Does the "remarks and explanations" section indicate that certain criteria aren't applicable?	Confirmation (from a demo, test instance, or product literature) that exceptions are accurate. For example, the ACR might say that the product doesn't have content that moves, blinks, scrolls, or auto-updates, so is exempt from the 2.2.2 Pause, Stop, Hide (Level A) criteria. In this case, you should verify there's nothing in your version of the product that moves, blinks, scrolls, or auto-updates.
Take note of any items marked with "Partially supports" or "Does not support." Are these part of the version of WCAG required under Quebec law, or are they new criteria we're not (yet) required to meet? Does the vendor have a roadmap or backlog that demonstrates a commitment to addressing these issues?	A commitment to address accessibility issues, particularly severe ones. Invite the vendor to speak to you about their plans, and understand the severity of the issues. (For example, is the criteria marked as "partially supports" part of a nonessential feature with a minor defect? Or is a key feature unusable under certain circumstances?)

Questions to ask

What to look for

Who did the evaluation? Was it a member of the company, or an independent accessibility consultant?

An unbiased assessment done by a knowledgeable person.

Independent consultants are less likely to be biased, and may conduct more thorough evaluations. If the assessment was done by a staff member at the vendor's company, pay close attention to whether the results seem biased.

Is the evaluation date relatively recent?

A recent assessment that's accurate to the product you're using.

If it's more than a few months old, ask the vendor if any changes have been made since the evaluation was done. Is the ACR still accurate? Have issues been resolved (or new features added) since the evaluation? If so, see if they can provide an updated ACR soon.

Which method(s) did they use to evaluate the system?

A thorough approach and a mix of testing methods and technologies.

The ACR should include automated, manual, and functional testing methods; information about how much was tested (the entire application? a representative sample?); and a list of assistive technology tools used.

Note that automated tests alone aren't sufficient to prove compliance.

Which standard(s) are covered in the report, and which level of success criteria are included?

Coverage of the applicable WCAG standards at the A and AA levels.

Quebec law (link available in French only) requires us to be compliant with the latest version of WCAG available in French. At the time of this writing, that means WCAG 2.1 and some aspects of 2.2.

Each version builds off of previous versions, so compliance with later versions (like 2.2) means you're also covered for older versions (like 2.0). A compliance report that covers newer versions (even if these are not yet required by Quebec law) can be a good sign that the vendor is planning ahead for accessibility compliance.

The WCAG guidelines have success criteria which are categorized under three levels, A, AA, and AAA. "A" is the most basic level of compliance, "AA" is our legal requirement, and "AAA" is the ideal.

Review the "remarks and explanations" column of the success criteria table(s). Does the report:

Clearly describe the overall level of performance for the criteria?
Indicate any exceptions where the product doesn't match the general level of performance?

An accurate and detailed portrait of compliance with each standard.

This description should be detailed enough for you to understand where the standard was tested, how thoroughly it was tested, and why they assigned the rating (such as Supported, Partially supported, Not supported, or N/A).

Does the "remarks and explanations" section indicate that certain criteria aren't applicable?

Confirmation (from a demo, test instance, or product literature) that exceptions are accurate.

For example, the ACR might say that the product doesn't have content that moves, blinks, scrolls, or auto-updates, so is exempt from the 2.2.2 Pause, Stop, Hide (Level A) criteria. In this case, you should verify there's nothing in your version of the product that moves, blinks, scrolls, or auto-updates.

Take note of any items marked with "Partially supports" or "Does not support."

Are these part of the version of WCAG required under Quebec law, or are they new criteria we're not (yet) required to meet?
Does the vendor have a roadmap or backlog that demonstrates a commitment to addressing these issues?

A commitment to address accessibility issues, particularly severe ones.

Invite the vendor to speak to you about their plans, and understand the severity of the issues. (For example, is the criteria marked as "partially supports" part of a nonessential feature with a minor defect? Or is a key feature unusable under certain circumstances?)

You should also consider how each vendor's ACR compares to the other vendors bidding on the contract. In general, you want to see a highly detailed ACR that shows a commitment to finding and resolving issues. This is better than a ACR with few concrete details, even if all criteria are marked as "fully supported." (If a vendor provides an ACR with no remarks or explanations, ask if they can provide you with a more detailed version.)

During your project, keep an eye on whether the company is regularly addressing bugs and accessibility issues. Make sure you always have the latest version of their ACR.

Once you have a demo site or test instance, consider how customizations made for your project could affect compliance. Here are some examples:

The vendor allows you to customize their product with McGill's brand colors. This means the colors in the McGill version of the application won't match what was tested in the ACR. To address this, find out everywhere customized colors will appear, and test text and background colors for appropriate contrast.*
The base product (evaluated in the ACR) didn't come with video content, but your version will have a video. Make sure the video complies with the applicable WCAG success criteria (including controls and closed captioning). Similarly, if you can add your own images to the tool, you'll need to make sure they're compliant (by adding alt text, for example).
The base product had a native form tool, but you'll be linking to a form on another system. This is a good time to verify the other system (and its forms) are compliant.
The content comes from your team, so it hasn't been tested in the ACR. To be compliant, your content should use plain language and be properly formatted for accessibility.

As your project closes, see if you can incorporate some accessibility-oriented testing activities (such as screen reader testing, keyboard navigation, and automated scans). This will let you confidently and independently understand how the product is accessible, and where it could be improved. Even after project launch, you should absolutely advocate for accessibility-oriented updates with your vendor. These updates can benefit all the clients (and users) of the software.

* Note that if you're using McGill red for text or backgrounds, you may see the mention "AA Large" when you test for contrast. To meet this requirement, the text will need to either:

Use a bold font and display at a physical size of 14 point (this usually means 18.66px), or
Display at a physical size of 18 point (24px) or larger if using regular or light fonts.

For custom websites or applications

In the case of custom websites, you're responsible for the ACR.

Before signing a contract, you should make sure the vendor knows that a VPAT will be used to evaluate their work. Ideally, compliance will be part of your contract with them. A reputable vendor will already have an understanding of how to deliver an accessible website and address any issues uncovered in the VPAT assessment. This should not substantially affect project costs. (If your vendor isn't comfortable with web accessibility, or claims it will be very costly, consider finding other vendors to bid on the project.)

Using the McGill Design System (McGill DS) can accelerate the design and accessibility compliance of your site. Be sure to mention this to the vendor, since it may allow them to reduce their fees or include more features at the same price.

At project kick-off, plan when and how the VPAT assessment will be done. You can do it yourself (or ask a member of your team to do it), or hire a consultant.

Do the assessment yourself

First, understand what it means to make your site accessible.

Make sure you have a strong understanding McGill's Digital Standards, which cover all requirements and best practices applicable to your McGill website.
Take our WS-106 course to deepen your understanding of the WCAG success criteria and how to evaluate them.
Identify and set up the tools you'll use to help evaluate accessibility.

Then, consider when and how you'll conduct your evaluations. This should match the vendor's delivery strategy for the project. Most vendors work iteratively, building discrete units of your project and delivering them one by one.

Here are two possible ways to manage an iterative delivery and testing schedule:

Test individual components or templates as they're delivered by the vendor. For example:
- In test session 1, you might test the header, navigation, and footer.
- In session 2, you might test the homepage template.
- In session 3, you might test the different templates in your "news" section.
Test key user flows as they're developed. For example:
- In test session 1, you might test navigating the homepage and signing up for the newsletter.
- In session 2, you might test reading an article, commenting on an article, and browsing to a related article.
- In session 3, you might test searching for an event, reading the description, and registering to attend.

Note that the components, templates, or user flows you test will depend on what's relevant for your project - these are just examples!

You should always do a complete test near the end of your project, when work is finished but not yet launched. This is often called UAT (user acceptance testing), because this is when you'll review the website and formally accept or "sign off" on the project.

If you haven't been able to test things iteratively, make sure you have at least 2 - 3 weeks to complete a full test and allow the vendor time to fix any issues. (If you've tested iteratively, you might be able to do this in less time.) During your assessment, download and fill out the "international" VPAT template. As you complete the report:

Be sure to use a mix of automated, manual, and functional testing methods. For example, you might run an automated test, browse the entire site with a screen reader tool, and complete common tasks using exclusively keyboard navigation and 200% zoom.
Cover the required WCAG guidelines according to Quebec law (available in French only). At the time of this writing, that means WCAG 2.0 at levels A and AA.
Fill in the "remarks and explanations" column of the success criteria table(s). Include:
- A short summary of the overall level of performance for the criteria. (For example, "Almost every interactive element and content piece is accessible and operable using a keyboard or keyboard
  emulator.")
- Any exceptions, where the product doesn't match the general level of performance. (For example, "Exceptions include launching the "filters" window in the search results interface.")
- Information about which user persona or workflow any issues belong to, if this will help understand the issue and how to fix it.

When the ACR is complete, notify the vendor of any areas where changes are needed to meet full compliance.

Once the vendor has addressed the issues to the best of their ability, update the ACR and web.comms [at] mcgill.ca (share it with Digital Communications) so that we have a record of the project's compliance.

Hire an independent accessibility consultant

There are a range of possibilities here, from individual consultants to giant accessibility-focused companies. You should do your own research into which companies offer ACR authoring services, and what their rates are. As with nearly every type of web service, we recommend you:

Get multiple quotes from different companies (at least three realistic options).
Look for companies that have done work on similar projects in the past.
Avoid doing business with individual freelancers or students, unless they have a demonstrated track record and impeccable references. (Speak with the references!)
Make sure the consultant will cover the required WCAG guidelines according to Quebec law (available in French only).

Plan for the consultant to do at least two, and maybe three reviews:

A complete review of your website or application near the end of the project (when the website is mostly complete, but there's still time to address outstanding issues).
A revision of the ACR after the vendor has addressed any issues from the first review.
Consider requesting an additional review if the vendor will address outstanding issues after the second revision, or if you make other changes to the site.

Review the ACR after the consultant delivers it. Use the assessment review table above to guide your review.

After review, you should have a list of corrections to request from the site vendor, and a sense of their relative importance. After the vendor updates the site, ask your consultant to review the changes so that the ACR can be updated.

After launch, be mindful of any changes you make to the site. Adding features, modifying colors, changing typefaces, updating plugins, or introducing new types of media can affect your site's accessibility.

Consider training yourself (or a member of your team) to be able to audit and record compliance information. That way, you can maintain and document compliance efficiently and affordably as your site evolves.

You should also keep an eye out for changes to our legal requirements around accessibility. Sign up for the Web Services newsletter and regularly attend training and events to help stay current with requirements for your site.

Back to top