Accessibility in SwiftUI

Hi I’m Rob. This is my dog Bella she wanted to be in the video as well. In March 2019 I was supposed to be attending CSUN assistive technology conference where I was due to be speaking about some of the changes that Swift UI has brought and some of the ways that improves accessibility in iOS apps unfortunately because of Covid-19 I wasn’t able to attend that conference and I wasn’t able to speak so I thought I’d make a version of the talk and upload it to YouTube so I would be find it useful and all of this talk it based on some blog posts and guides I’ve written on swift UI and how that makes accessibility much better and that’s all available at mobilea11y.com and you’ll find a section for swift UI guides there hope you find the tool useful I also have a book available developing inclusive mobile apps it covers how to build accessible and inclusive apps for both iOS and Android it’s available to order now from apress.com today I’m going to talk about accessibility in Swif UI if you didn’t already know a SwiftUI is a brand new way of creating user interfaces using Swift code on Apple platforms Apple announced it at WWDC in 2019 so it’s still a really new technology this is one of the older ways of making user interfaces on Apple platforms on iOS specifically these are called storyboards and it’s a layout of screens which you can use as here we’ve kind of used a wireframe approach but you can use them to add in your text and images as well and place those exactly where you want them it allows you to do layout and make sure that it works on different screen sizes and allows you to place interactive elements like buttons and controls and things like that as well this is what SwiftUI looks like SwiftUI is all done in code there is a kind of WYSIWYG element to it but the user interface itself is all built up using code like this so swift UI is really new it’s less than a year old at this point and it’s something that a lot of people aren’t using yet in production just because it’s so new and there’s still a few questions about how it works so there’s obviously a question about “well if it is so new why are we talking about accessibility already maybe we need to sort out some of the other questions we have first?” I want to answer that using this podcast now this is a podcast as part of the parallel feed it’s actually a 40 minute long audio documentary created by Shelley Brisbin it’s called “36 seconds that changed everything how the iPhone learned to talk” it’s the story of how the iPhone became accessible and here’s Phil Schiller on the stage at WWDC announcing the brand new accessibility features on the iPhone and the reason it’s called 36 seconds is because that’s how long Phil Schiller was talking about these new accessibility features and these were really the first accessibility features that were part of the iPhone that were part of iOS and this documentary I’ve clipped it here at 19 minutes and 27 seconds of a 40 minute long documentary halfway through the reason I clipped it halfway through is because this is exactly the point in the documentary where we start talking about Phil Schiller announcing the new accessibility features on the iPhone this proves that although these accessibility features are incredibly important people although they mean the difference between being able to use the iPhone not being able to use the iPhone it was just as important to people who need those features that they weren’t there half the documentary the first half of the documentary covers the difficulties that some people had in using the iPhone before it was accessible so I want to play you some quotes from this documentary now “I remember picking up a friend’s iPhone and and feeling this blank piece of glass essentially with just a button on the bottom of the glass thinking man we are gonna be locked out of this thing and it’s a real concern.” “I was sad because I felt oh here’s another time we’re gonna be left out and eventually someone’s gonna come along and make a special blindness specific eye device it’ll be three versions old it’ll cost four times as much and we’ll just keep buying it because it’s the only option that we have.

” “For the first time in 20 years Apple had built a product I couldn’t use 05:09 - I’m fairly sure I cried about that.” “It was heartbreaking to me I actually held someone’s iPhone in my hand and it was the most beautiful sexy sleek device it was just flat I mean I don’t know how else to describe it as a blind person it was just flat and then this beautiful backing and curved oh man if it could just say one thing to me if you could just, but it couldn’t.” all of these quotes from the first half of the documentary before the iPhone became accessible really show that people who need to use these accessibility tools are really no different to the rest of us who use the iPhone they all love the design the beautiful attention to detail Apple put into these products and they really want to use them just as the rest of us want to use them it’s just that because of some of the design decisions made in the early iPhones they weren’t able to do that. Now the iPhone is now accessible and it’s generally considered to be one of the drivers for accessibility in a lot of modern electronics. The UIKit accessibility API which drives all of this is really fantastic it allows tools like voiceover which reads the content of the screen and lets people with no or very little vision control their device it also controls the screen reader and which will read large portions of text if you’ll you have a webpage or a book in this instance someone can just select the screen reader to read that content for them which is really fantastic again if someone doesn’t have great vision or if someone struggles to read perhaps it allows switch control so that people with very limited movement can use an external switch to navigate and control their device and voice control is a brand new feature in iOS for this latest release and that allows people again with very limited movement to control the iPhone just by using their voice that’s very cool to use as well.

UIKits accessibility API is great but 07:35 - it will only ever be an add-on because it was something that wasn’t considered when iOS was first built. Considering accessibility from the very start results in a better product and that’s exactly why I want to talk about accessibility in Swift UI right now and just before we go any further just a little bit about me my name is Rob I am an iOS engineer. I currently work for Capital One in the UK and I also have a blog, I edit a blog mobilea11y.com and there I collect a lot of information about how we can build apps for iOS and for Android that are more accessible that allow more people to use them and include more people in and you can contact me as well on Twitter or by email I’m always really fascinated about people’s experience with accessibility on mobile devices so whether you use it or whether you know someone who uses it or whether you’re a developer who makes apps for iPhones or Android phones and and you’ve had a good or a bad experience creating something with accessibility I’m always really interested to know so feel free to get in touch. SwiftUI, as opposed to UIKit, was created with accessibility as a first-class citizen.

Apple built it with accessibility in mind from the 09:10 - very start they did it by including people from Apple’s accessibility teams along with a lot of the people who were making Swift UI and a lot of these accessibility decisions can really be seen throughout the syntax of Swift UI and some of the decisions that have been made. Swift UI works with accessibility out-of-the-box it keeps your accessible user interface in sync with your visual user interface and we’ll cover this more in just a second it encourages best practices by forcing you to consider how you use images in your interface and it promotes good accessibility text is all multi-line are all supports dynamic text sizes by default you actually have to write me a code to make your app less accessible so before we move on to some of the changes that you can make I think it’s important just to very quickly consider why accessibility in Swift UI is a much better option and this on the right here is the user interface for an app that I’ve built it allows people to book holidays to various places around the solar system this is the listing page for Mars we’ve got a hero image at the top and we’ve got the title and subtitle we’ve got a like button and then we got some detail about the holiday we’ve got a switch so people can get notified of updates about my holiday and a big button to let them book that holiday to Mars the price there I googled it it is roughly accurate to send four people to Mars don’t know if anyone’s got six billion dollars available but you can enjoy a trip out there if you have so this collection of images and text buttons controls you’d recognize this right this is your user interface or visual user interface this is what you see when you use really any iPhone app but your app also has an extra user interface that not everyone will be familiar with and and that is called or Apple call it the accessibility user interface now you might be familiar with this on other platforms where it’s commonly known as the accessibility tree it’s essentially the same thing and the accessibility tree or the accessible user interface is this extra user interface that isn’t visual but that represents all of the content that’s on the screen all of the controls and text and images and all that kind of stuff and it includes information about how that is presented to different assistive technologies and the accessibility user interface on iOS is generated for you you have no direct input to it you can’t ever see it and you don’t directly create it it’s built for you by Apple’s accessibility API as part of UIKit based on the visual UI that you’ve created in your storyboard file or in code or however you’re generating them but it’s something that you don’t directly interact with and it’s used to communicate with assistive technologies such as voiceover switch control Braille keyboards and voice control and many others and the accessibility user interface is customizable although you don’t interact with it directly and don’t create it directly you can give UIKit some hints as to what you wanted to do with it if it doesn’t generate it for you perfectly based on the information from your visual user interface you can tell hey make this change we need to improve them I’m going to give an example here of what would make up a accessible user interface. Apple don’t ever provide this interface to you directly because it’s not something you as a developer can really interact with or create yourself and that’s kind of deliberate because we want it to be really basically exactly the same as your visual user interface so instead this is kind of an artist’s rendition of what a accessible user interface might be and we’ll build this up by navigating through this screen with voiceover and on the left-hand side I’ll kind of print out what I think the information that makes up the visual user interface the accessible user interface might be Mars image Mars the red planet heart image with virtually zero cloud cover send me updates switch button off six billion dollars for a family of four seven nights plus 80 days travel book now button there we go that’s roughly what an accessibility user interface might be we’ve got the the type of object and we’ve got a text value for it but there’s a few other things that we’ll want to add in so for example actions so that people who are using switch control or VoiceOver know that they can interact with a particular element so where we’ve got a button we’ll add in an action so that if somebody does choose that with switch control switch control knows what action to call we also maybe want to specify something in terms of order and for example this hero image at the top and we’ve got above that we have title and subtitle and our like button and we want to make sure that our accessibility tools will always access that in the order we intend so we’ll want to tell it that we should access the Mars title over the subtitle and that though those are really connected and then say that actually after the the title and subtitle we want to kick the like button to that and we want to make it clear that that goes above our hero image so that when it’s highlighted with a bounding box the bounding box knows where to draw so we’ll add that layout information in as well and actually what we’ve kind of done here is we basically created Swift UI this code that I’ve kind of given you as an example there’s two what an accessible user interface might be well this is just Swift UI code and this is why Swift UI is such a fantastic option for accessibility SwiftUI separates the what from the how. It takes what we want to be presented to the customer so text, a control, the order it’s connected in, the controls actions and how that’s presented to our customer so for example we can present it as an iPhone screen one of the extra benefits of Swift UI is it makes it really easy to present that same user interfaces and iPad screen or a Mac screen as well but also as well as having SwiftUI create you a screen for a Mac application or iPad application or an iPhone application. SwiftUI can also create you an interface that voice control or voiceover or switch control can use as well just by taking that what information of we want a button than we want some text this really comes into its own with State in this example we’ve loaded the hero image and title but we’re going to the network to find out some information about this holiday and we’re waiting for that information to come back from a network request so we’ve thrown up a spinner while we’re waiting for that so if we were to access this screen voiceover this is what we hear Mars image Mars the red planet heart image in progress so that’s pretty much what we’d expect so now this is again our uikit example um the spinner has disappeared because the content has returned from the network now in UIKit sometimes you can get a miss-match between information on the screen that’s displayed and the information that’s provided to the accessible user interface and and this happens if you make a big change to the screen like this where we’ve replaced the spinner with some other content so in UIKit sometimes we’ll get a bug that’s a bit like this Mars the Red Planet heat image in progress so you can see what’s happening here although visually we can see on the screen we’ve got a bunch of extra text content voiceover is focusing on a progress spinner that is no longer actually visible on the screen and that’s happening because voiceover hasn’t been told by UIKit that the screen has actually changed now you can fix this in UIKit by posting this notification UIAccessibilityScreenChangedNotification and that will tell UIKit that it needs to tell the accessibility user interface to update based on new content but we don’t always remember to do that they didn’t we don’t know we need to do that because we’re not voice-over users so sometimes this catches us out and it means that voiceover users can never tell what content is actually on the screen or maybe they can but it’s very confusing because the accessibility user interface will catch up eventually but for a little while it will look like there’s no content there well in SwiftUI this isn’t a consideration at all we can completely forget about all of this and that’s really because of this very first line that you have in swift UI this is really about the very first word there struct.

So this is very basic swift 20:22 - knowledge but structs in swift are value types they are non mutable so when you create a struct the values that you give it when you create it the values that that struct will have for its lifetime except UI doesn’t work like that in UI we’re creating and changing things all the time we’re adding new elements we’re maybe animating something then maybe changing text so how does that work with a struct well that means that SwiftUI has to re-create a new strut every time so this is what happens the value on the struct changes and that triggers the struct to be recreated by SwiftUI and so that means that SwiftUI is going to redraw the screen based on the new information it has. But because it’s doing that at the same time it can also automatically trigger the accessible user interface to redraw which means your visual user interface what you can actually see on the screen will always be 100% in sync with what’s presented to accessibility tools so you’re never going to have this mismatch between what was visible and what’s presented to accessibility users. Images are another great improvement the SwiftUI. SwiftUI makes all images accessible by default and it will take as the accessibility text the name of your image so here this image command for swift UI will load from our asset catalog an image called m1 hero image at 3x which makes perfect sense for managing our assets but if a voice-over user navigates to that what they’ll hear is m1 hero image at 3 X well that’s meaningless to one of our users so in this situation we’d need to add an extra label to it this label is then what is read to our customer there is another option as well the other option you can use is this one here say we’ve fetched information from the network as part of a network request and but unfortunately there was something went wrong there was an error so we’ve added in an exclamation mark image to let our customers know that something went wrong we’ve also added the image added text as well because you always need to provide information in a redundant manner so people who can’t see the color the red color of the exclamation mark can still read the word error still make sense to them. Well if that image is called error then this is how VoiceOver will present it error image error VoiceOver has now just said the same thing twice which it’s kind of frustrating because it means you’ve added an extra swipe and you’ve not really given any extra information this means it’s gonna be really laborious and time-consuming for voice-over user to navigate and use your app it’s going to be incredibly frustrating for them so the option here is to use what’s called a decorative image now we still pass it the file name or the name of our asset catalog of the image we want to display but we preface it with the word decorative and that means voiceover we’ll ignore this image for us and just read the text let’s talk about controls so controls are things like buttons and switches and sliders anything that you can interact with so let’s take a look at this example send me updates switch button off so this again is UIKit. This is a common issue with accessibility in UIKit in that many controls like toggle switches for example don’t have text associated with them.

So this is a switch that someone can 24:52 - toggle this to receive notifications if something changes in this image and visually we can see that send me updates is to the left of the switch so we can see that if we toggle that switch is gonna send us updates that makes perfect sense but what if you can’t see the layout you can’t infer that contextual information if all you can hear is voiceover take a listen first of all we’d have to swipe to the label send me updates and then we’d have to swipe again switch button off we know there’s a switch button and we know it’s off but what does the switch button do what’s going to happen if I turn that on well because you can’t see the layout you can’t guarantee that that text we heard before of ‘send me updates’ updates is what’s going to happen if we toggle that button it could be that actually the button is above the text describing what actually that that switch button will do. now that isn’t the case in this instance but for a lot of users who use voiceover it’s a common thing that they come across and so they’re naturally going to be wary of toggling a button they don’t know 100% what it’s going to do because it might do something they’re not expecting. Well in SwiftUI this is completely fixed. Really all controls in SwiftUI take a text value in the controls body which links the control directly to that text so listen to this one send me updates switch button off DoubleTap to toggle setting it’s really clear and obvious exactly what that control does then it’s one swipe which makes it quicker simpler and easier to navigate and we’ve got the name of the control in there so anyone who toggles that is going to know exactly what that toggle does and know what they’re going to expect from it let’s talk briefly about semantic views so semantic views are kind of a slightly more advanced technique in terms of making your apps more accessible it’s not something that everyone will be aware of so I’ll just give a very brief introduction to it so this is an example of a semantic view from within iOS that you might come across if you use voiceover so this is a message bubble in a message if voiceover was just focusing on the content then the bounding box would just be pretty tightly drawn around that the message of do you want to go for dinner or a movie well actually that’s not really the meaningful part of this user interface the meaningful part is the message bubble and so that’s exactly what voiceover has highlighted here it’s not drawn the bounding box tightly around the message but around the message bubble because that’s the meaningful part of the UI then semantic views are a view that contain the meaning so here the meaning is this is a message in this instance the semantic view just has one piece of content the text but really semantic views can contain multiple elements so you can contain multiple bits of text or text and a control or various other pieces of UI in there as well but together they really have one meaning there’s various ways of doing this in UIKit but SwiftUI makes it incredibly simple. Layout in SwiftUI is all built up using stacks. HStacks, VStacks, ZStacks and you can put elements inside these stacks to group them together and to specify to SwiftUI what order and in what position you want these elements to be. Here’s an example of an HStack from our holiday app which includes two bits of text and a button if we present that to voiceover like this this is what voiceover will provide to our customer Mars the red planet heart image so that was three swipes there one to Mars one to the red planet and one to the button but actually really these probably make sense as one it’s it’s a title and a like button and it’s kind of our topmost part of the screen we want to present it as one thing it’s a only got one action and the like button really gets its context from the title so if we put those together into a semantic view that will make it much easier to navigate and much more meaningful and it’s with UI we do that just by adding this one modifier.accessibilityElement(children:.

combine) 30:23 - and that tells SwiftUI to combine all of these elements all of the child elements the text and the button into this one HStack. That means that those three elements are no longer presented to accessibility but what is presented to accessibility is all their information combined as one so let’s listen now to what voiceover presents to our customer Mars new line the Red Planet button image it’s so much simpler it’s one swipe and we’ve got all the context information that we need to know exactly what that button does so techniques like this are going to make navigating your apps so much simpler for anybody using assistive technologies. So that’s SwiftUI accessibility. Accessibility in SwiftUI is great and the reason I think we should be talking about it now is because I’d really love to encourage you to use SwiftUI. next time you’re starting a new app consider using SwiftUI because not only is it a nice cool new technique but your accessibility users are going to get a much better environment as well again if you have any thoughts on accessibility in mobile please feel free to contact me and you can get me by email or on Twitter and I also run mobilea11y.com and that has a whole guide on accessibility in SwiftUI where we cover all of the topics that we’ve covered in this talk in much more detail and a few others as well so visit mobilea11y.

com if you want to know a little bit more thank you very much.