Yes, we must say that PDFs should be accessible. A tagged PDF ensures equal access for people with disabilities who use a screen reader, speech to text technology, or a braille displayer. Accessible PDFs are now required by the Americans with Disabilities Act (ADA), Section 508, and the AODA.
A document's availability as a downloadable PDF does not imply that it is an accessible PDF. PDFs, as we all know, are the preferred format for organisations to share information.