Be My Eyes - a remote human or AI helper for people with sight loss

Introduction to Be My Eyes

Be My Eyes is a free app for iPhone and iPad that provides free services. 

The app offers both human and artificial in telligence assistance with describing  what your phone or tablet camera sees. It also offers specialist support from several organisations regarding their products and services.

Before you can use Be My Eyes, you will need to create an account as a user who needs assistance. Be aware that there is also an option to sign up as a volunteer. I recommend selecting the option to sign up using your Apple ID since this avoids creating yet another account and password.

The app has 5 tabs along the bottom of the screen. The leftmost is the "get support" tab and is the tab to use for human assistance. The next tab is the Be My AI tab and this is the tab to use for artificial intelligence support. The third tab is "service directory" which offers human contact with hundreds of customer service departments. The fourth, "communities" tab offers news and information about the app and the fifth tab is "settings".

Human assistance from Be My Eyes

If you select the first, "get support" tab, your screen will have a large button occupying most of the top half of the  screen. This is the "call a volunteer" button. If you use this button, the app creates a live video link with a sighted volunteer. The volunteer can then offer descriptions of whatever you point the iPhone or iPad camera towards and can help you point the phone in the right direction.  Be aware that the volunteer is probably viewing your image on a smartphone so they may have difficulty reading some text, especially if you're not holding the phone steady. They should have no trouble doing tasks like describing clothing or checking settings on dials. Volunteers come from all around the world, so you'll probably be able to link with one who speaks your language whenever you need help.

If you aren't comfortable asking strangers for assistance, you can set up groups of people you know and trust to call on in stead of the main group of volunteer helpers. Use the "My Groups" button which is below the "Call a Volunteer" button to set up your groups. Once a group is set up, the app will attempt to find a member of the group who is free to help you when you request assistance. 

Artificial Intelligence assistance from Be My AI

If you select the second, " be my AI" tab, the screen will mostly display the camera viewfinder. There is a thin "take picture" button across the screen above the tab bar. Note that the AI interaction is based on a single picture and is not a video link so try to take care to ensure that the scene or object you want described is included in the shot. It is best to avoid holding the camera too close to objects which may then only partially appear in the picture. This probably won't be a problem for most low vision users but it's a little tricky for blind users, especially if you are trying to have a paper document described. If you are photographing an A4 document, place the document on a table, start with your iPhone camera lens resting near the middle of the document and then raise the phone steadily, keeping it level until you are at least two feet above the document before double tapping the take picture button.

Take your picture with the "take picture" button You will hear a jingle playing while the picture is being analyzed. When the analysis is complete VoiceOver users will hear the description begin to be spoken by VoiceOver's voice. VoiceOver's focus may skip to a button. If it does, touch near the top of the screen to locate the start of the text and then swipe down with two fingers. You may need to swipe right occasionally to hear the complete description. Non VoiceOver users will see the text description on screen in their chosen text size and can have the text spoken automatically by the app if they wish. When VoiceOver is not operating a circular loudspeaker button is present at the top right of the screen and this can be tapped to turn automatic speech on or off.

Now there will be two thin buttons above the tab bar, a "take picture" button and an "ask more" button. There is also a "clear" button near top right of the text. Users with some useful vision will want to use the clear button in order to see the viewfinder for their next picture. 

If you would like to know more about the current picture you can use the "ask more" button to ask typed or dictated questions. These can be the kinds of question you would ask a sighted volunteer. For example, if you photograph a restaurant menu, the first response will summarise the menu for you. You can listen to the end or you can use the "ask more" button to interrupt spoken descriptions and ask a specific question like "do they have baked potatoes?". If the restaurant has baked potatoes then you'll probably be told what fillings are available and the prices. Or you may have pointed the camera ahead on the path you are walking in the park and want to ask if there is a bench that's free. It's very powerful. The main snag is that users with no useful vision get no assistance in framing documents they want described in the camera viewfinder. However, they will probably be told once the image is analyzed, that a document is only partially visible or upside down.

With a little practice, most users will be able to photograph documents well enough for Be My AI to function. Be My AI can have a conversation with you about the text. Just one warning; these AI systems occasionally make mistakes and occasionally invent items that weren't in the scene. So that strawberry and pickle baked potato might not be available after all!

If you use the "ask more" button you will then be able to dictate or type your question and there will be a send key at bottom right of the keyboard to send your message. If you prefer to call a human volunteer you can locate a "call a volunteer" button above the message field.

You can continue asking questions about your picture as many times as you wish. When you have finished, use the "end chat" button near top left of the screen . You will then be asked to confirm that you want to end the chat. Finally, you will be asked to rate your experience. Swipe right to locate the appropriate button. You will then be back at the first "take picture, button". It's good practice to do this at the end of your chats before you move away from the app since your next session with the app will otherwise begin with the previous chat still on screen.

Contacting a wide range of businesses and organisationsfor support

The third, service directory, tab gives access to hundreds of customer service organisations. Be My Eyes is aiming to make it easier to get in touch with customer support from businesses and other organisations. You'll find more about how this directory works using the Info button at top right of the Service Directory screen.

Sharing images to Be My AI

Images in apps or in your Photos can be sent to Be My AI for analysis using the share menu which will have a "Describe with Be My AI" option.

Ray-Ban Meta smart glasses and Be My Eyes

If you have a pair of Ray-Ban Meta smart glasses paired with your iPhone, you can set up the glasses to be able to call a Be My Eyes human volunteer. This has to be done just once  in the Meta View app on your iPhone. Once you have linked your smart glasses to your Be My Eyes account using the Meta View app you can call a volunteer entirely hands free. Just say "Hey Meta Be My Eyes" and wait for a volunteer to speak to you.  Then all you need to do is look towards what you need assistance with and explain what help you need to the volunteer who will be able to see what you are facing towards through the camera mounted in the glasses and hear your voice using the microphone in the glasses so you don't need to get your phone out of your pocket at all. This feature is currently available in the US, UK and several other countries.

Back to navigation page