Watch your classmates’ video presentations for the multimodal display assignment (linked on the class website) and write comments for each video. Your feedback should include a couple of sentences about their proposed solution(s) and at least one suggestion for improvement. **Please do not write that you have no suggestions for improvement.
Here are my comments for everyone’s presentations. I enjoyed watching them all, thanks!
Daniel – My primary comment based on what you are presenting is that I am not really sure what you are giving us to critique. The game looks like it would be extremely difficult without vision, but your presentation didn’t give any details about how you put your experience in context of HCI design, or any details about what your new multimodal design would be. It’s possible your decision to go in the the opposite order (blindfolded first) robbed you of a lot of context, because if you had played it first with sight, you might have thought to ask your voice-over partner to describe the targets, distances, and vectors, or places that look like safe cover, etc. – the basic details that are essential to the actual mechanics of gameplay.
Mike – Your presentation was short but well organized and informative. I would have liked to see a bit of the gameplay, but you did focus on the design changes, so we have a good sense of what your thought process was. I like the fact that you are enhancing other audio feedback in addition to adding voice. But I wonder if the voice/AI component wouldn’t get just as bogged down with continuously changing detail as your wizard of oz voice-over was. I would challenge you to think about how you could offload some of the positional information into another domain, like 3d sound. I’m also curious if you think any of these changes would impact the game experience for a sighted player, or if you imagine this as a totally separate mode.
Pascal – I appreciated the narrative style of your review, so that I could see & hear the actual interactions as you were describing them. Your analysis was thoughtful and I think you are on a good track with identifying the potential of the sound effects to be much more descriptive. But one thing you might add is additional voice-over description in more situations than just the text popups; this could help with some of the environmental/geographical features, and will might also help minimize the number of sound effects you ultimately need, since there seems to be a potential for over-saturation and difficulty for the player to memorize what all the different sounds signify. Using some voice narrative would also keep the game from devolving into just navigating an abstract soundscape; narrative might help maintain the story or mental model of the ‘quest’ aspect of the game.
Meetha – Your impulse to keep the sound effects to a minimum, and keep them relevant & intelligible, is a good direction. I was a bit confused when you were describing the game’s sensory inputs, though. The sound and visuals are outputs, in this case, and I gather the only input is the touch screen. One major question I have for your proposed changes is, what is the point of playing, for a non-sighted player, if the only thing they do is follow directions? I would argue that navigating and collecting pellets are the challenge of the game, but if your voice-over (AI or whatever) ends up defining the directions, then there is no choice in strategy left to the player. Maybe you could instead describe to the player what their options are for navigating the maze. For instance, you are at a crossroads, you can go left or right; there is a ghost following you and he is getting closer; there is a power pellet to the east… etc. This would allow the player to decide where to move based on their own cognitive model of the gamespace. In this example you might have to slow everything down a bit in order to allow time for those descriptions, but at least you might preserve the point of playing the game.
Colin – Your choice of game is an interesting point, because I disagree with you that it might not lend itself to non-visual play – on the contrary, I think it might be more appropriate for multimodal augmentation than most of the other examples we have in the group, because the goal of the game is expressed inherently in the audio domain rather than the visual. I thought it was interesting that you said you could play with the sound off, which might be true for a sighted player, but would still sort of change the point of the game from playing along with music toward more of a simple button-pressing exercise (like the awful game I chose). At the risk of using my own approach as a suggestion, you could avoid the confusion of using the same haptic (touch) interface as both the prompt output and the player input, by adding voice, and allowing the “see-ahead” function to be dialed in by the player as to how fast the prompts come up. This is also a way of preserving one of the primary challenges of the game, which is to be able to transcribe a representation of notes to the physical key presses, and choose the right key in time, rather than just challenging the reaction time between a physical stimulus and a physical response. I would have liked to see/hear what your voice-over tests were like, and especially the problem of the wizard describing the precise timing.
Rosemary – I think you hit on my biggest point of concern for this whole exercise, which is how do you translate this to a non-visual domain, without delegating the essential game mechanism or point of playing. I had a similar issue with my own example, and couldn’t come up with a satisfying solution. The difficulty here is that the primary visual information is communicated over multiple vectors – both spatial positioning on a grid and color (or state) of every possible position. In order for a non-sighted player to follow the same game mechanics, they would have to hold on to a mental model of the full grid and the state of each tile (as well as their own position within it), updated with every move, in order to make strategic decisions. I often think of color as being easily correlated to sound frequency or pitch, so I wonder if you could use 3d sound somehow to help generate that mental model, in lieu of the color-states of individual tiles.
Thoughts on the multimodel input videos:
Alicia – Piano keys:
I felt you did a good job with your overall discussion of the game and your choice for selecting the game. I think it would be interesting to see if the error rates would change if the game was played in a quiet environment versus the middle of an arcade-type set up. I would have liked to see more of the actual game play. The game itself appears quite simple and it would have added more depth to the assignment had there been more actual game play that we could have observed.
Mike: Space Invaders
I thought you had a great discussion of Space Invaders and how you described everything including the barricades. The only thing that I think you missed in the old school version (they released an updated PC version in the late 1990s) was that you can only have one missile on the screen at one time.
I would have enjoyed seeing some of your actual game play, both blindfolded and without blindfold. Also the audio on your video sounded like you were a fair distance away from the mic and the sound would go in and out.
Pascal – Super Mario World:
Thought you provided a great walk through of the game and the fixes that you would put in as you were going through the game. (Made me realize what a glaring hole I had in my video.) I was wondering, because I am aware of the general age of SMW do you think the lack of complex audio is due in large part to the system limitations at the time of release? If you look at the newer platforms, Xbox, Xbox 360, and Xbox One the audio, video segments, and overall graphics within the game have made great improvements. Just a thought.
I could not really hear any of the game sounds that you were referencing. Not sure if it was just my age/hearing or if anyone had the same experience. It made it difficult to get a full grasp of what you were saying about no changes or limited changes.
Meetha – Pac-Man:
I thought you had a very strong video and provided great analytics of how to improve the gameplay. I would add that playing Pac-Man on a console system, PC, or the original arcade version would provide a much different playing experience and may correct some of those problem areas you selected. I would have liked to see you play on a television or some other screen to be able to capture a larger image for the screen play, as well as more game play. There seemed very little game play in your video and would have enjoyed the ‘Wizard of Oz’ format with this game especially.
Colin – Guitar Hero:
I thought you did a very good job with your overview and how to play GH. I thought you had some strong ideas about how to improve the game play for the game to be more inclusive. Although I have tried playing GH on mute, no where NEAR as much fun, just something about rocking out to the songs that is that next level. I would have liked to see some actual game play and hear some of those difficulties with providing how to play the game in a ‘Wizard of Oz’ type setting. Of course it would be fun to watch multiple levels of difficulty as well when playing blindfolded.
Rosemary – Q-bert:
I’m so glad that you picked Q-bert, although I was a little disappointed that the updated version doesn’t have the same sassy responses from him when he does die (or I missed them). I liked how you utilized different players with different gaming styles (PC vs Console) as there is a distinct difference between the two. I wish there had been more game play and some at the higher levels as well to get beyond just the basic simple just don’t fall off the map levels. I think it would have added some really get depth to the overall impact of the video.
RoseMary – I appreciate that you have two different types of player based on their gaming experience. My suggestion is that if we need to move diagonally could be a simple “diagonal up on the left” instead of commands like “left up”. I think adding simple words would be a better solution. Also, I liked the idea of the tiles count down. That’s a good way to keep track of the tiles remaining to change color.
Colin- I really liked the idea of introducing a Braille keyboard as the controller. My suggestion for the foreseeing is to alternate the keys between two hands in the braille keyboard. Lets say, the braille keyboard is designed as a console with controls for both the hands. So the first note could be simulated on the right hand and the second could be simulated on the left and it can keep alternating. That way the foreseeing could be done a little better I think.
Pascal- Great choice of game. Its different from the SuperMario Bros I used to play. My suggestion about the story line that pops-up or the help line that pops-up would be to read it aloud so that we could get the necessary instruction. It is very simple but a lot of times, I feel having the story/help lines in the game instead of playing it blindly without knowing what is exactly happening.
Mike- A very good formatted presentation! I think you could have added how your experience was when you were wizarding and when you were blindfolded. That would have made us understand how you felt playing it on the two different modes. One suggestion is that I would add vibration cues to indicate, loosing a life, or the enemy coming very closer. That would help a little. I have played a game where when we are closed to loosing a life, they would give vibration. It alerts the player actually.
Daniel- I am not sure if I am not able to see your suggestions or the proposed model. I could only see your game play and I am not sure if there is anything I am missing in the presentation. My suggestion would be to have directions to run. I have played a similar game like this where my friends could talk to me in the game. I do not know to play the game but they give me voice commands saying run to the left, lie down etc. That is a great way to get through this game.
Alicia – Great technique of using the grouping of numbers. My Suggestion would be to have a Braille keyboard that could help foresee the keys alternating between the two hands. This game is pretty similar to guitar keys game. So I think the same suggestion would apply for both games.
Alicia – Piano Keys. I am not exactly the point of this game? The player just presses a key associated with the key that’s show on the screen? I don’t quite see why that would be much fun but I’ll assume I’m wrong about this. Is one purpose of the game for the player to play music without being able to read music? I couldn’t really hear any music in your trials, due to the loud background noise. I did see why you tried “multiple numbers in a group” during the blind-folded tests and I would have expected that to produce better results. I liked your suggestions for improvement. I gather the tone of the actual note did not aid the player that much, and I am not sure how you map a full scale to only 4 keys (but perhaps I am not understanding something about the game?). From your trials, it seemed like there was enough time to speak the key number and react to it, but I gather that was not the case. I like the idea of the player being able to change the buffer size but the UI for that should probably just be a basic: Easy, Medium, Difficult levels.
Dan – Call of Duty WWII. For your video, you just showed blindfolded and normal game play. I believe the purpose of the video was to summarize your entire paper which would include descriptions of your experience being a Wizard and being blindfolded. But more importantly, it would describe what multi-modal features the game had, and what your recommended changes to the UI of the game would be. I didn’t hear any of that discussion? Perhaps you determined, as I might have, that the game would be very hard to augment since it is a real-time, action type game. The game did offer a few cues, such as “You are hit. Get to cover” but these were visual only and certainly needed to have an audio component for the visually impaired person. There certainly could be haptic feedback and I am assuming that the game had a controller that could allow for this. I’ve certainly seen controllers that do vibrate. Based on my short view of the game, I would have to conclude that it was very poorly designed for multi-modal play even though it was meant to be “real life”.
Pascal – Super Mario World. I thought you did an excellent job presenting the game and all the aspects of multi-modal components that need to be addressed. Why can’t hints be spoken instead of using the complexity of a screen reader? Your analysis of where audio scould be added is very thorough. How do you know when to interact with a box, i.e., how do you know you are under a box? How do you know what direction to jump over an enemy though, or does it matter? Can you tell what direction an enemy is coming at you? Was your solution to this the dual-channel audio you talked about? I still don’t understand how you know when or how much to jump on an obstacle? Percentage sound indicator was a good idea. I am not sure that this game, or many of the board games, would be very much fun without the visual though. It seems like your proposal would indicate so many audio enhancements that I question whether the game would become overly complicated?
Meetha – Pac Man. Good description of the game and analysis of the weak multi-modal aspects. The think the pop-up notifications are OK but there needs to be an audio component as well. I don’t see how you could just remove the pop-up notifications. If you propose having less confusing noises, that you mentioned were distracting, what sounds would you have less of and how would you minimize them? I liked your idea of putting some emotions into the commands. I like your suggestions of initial instructions and your count-down idea. I think vibrations could be very useful as well, as you mentioned.
Colin – Guitar Hero. As a guitar player myself, I was always interested in how this game worked but I never had the opportunity to play it. Your summary of the game was good but I was surprised when you said that you could play the game without actually having the background music on. I had always assumed that this game was like karaoke for guitar, i.e., you would have backing tracks and have to play the guitar part with them. If I were designing a guitar game, I certainly would make it work like this. From your description, it sounds like playing the game is entirely dependent on seeing the screen to read what buttons that you have to play. It sounds similar to the Piano Keys games in this regard. I didn’t quite understand how your proposed braille controller would work because don’t you need both hands on the guitar, or you proposing something completely different? It seems to me that the game needs to somehow be designed so that it can work in the auditory realm alone for the visually impaired person. Perhaps it could have some “practice” mode where you rehearse, more like karaoke (although I’m not exactly sure how this might work).
Rosemary – Q-Bert. This is a pretty simple game and you did a good job of explaining it. I thought that the wizard commands that you developed were straight-forward and easy to follow. Your suggestions for re-design were good too, and you did mention some ideas on what could be added to the controller itself, in addition to the game. I missed the part about what the bouncing ball was? I liked your suggestion about using stereo (headphones) to indicate the direction of the bouncing ball. As for determining which tiles are left to change color, I think the game could use some audio indicator at least informing the user of what was left. Perhaps when you got to some threshold, the game could start giving you more audio cues? I think your countdown idea addressed this. Maybe changing the audio cue of when you jump on a tile that has already been changed would be sufficient. I do agree with you that taking a “Simon says” approach (as many of us suggested) would lessen the fun of playing of any of these games, and that is the challenge that we face when addressing multi-modal issues.
Alicia –
I think that your solution of combining auditory and haptic feedback really works well for the game you chose. I feel that what you came up with would result in a game that was playable and, with practice, players could perhaps even be competitive. I also feel your solution was very complete. I didn’t think much during this exercise about indicating the start or end of play, but that does seem like an important thing to convey to the player. My only suggestion is that voiceover may be a bit overkill. If there are only a handful of keys, a unique tone could be used for each, and that may be easier to comprehend than a number.
Daniel –
I wasn’t able to get this video to play. I’ll try again later and put up another post.
Mike –
The use of different tones and speed of the sound to indicate proximity and direction is a great way to convey positional information. This takes a modality that the game primarily uses for cinematics and incorporates into the interface as an important way to get information. The only trouble I see is that the whole system relies heavily on sound only, which might make it difficult for players to recognize the important bits of information, especially as the game speeds up.
Pascal –
Your solution for indicating the location of obstacles and enemies with sound only seems like a pretty natural extension to how the game currently works. I think the idea for dual-channel sound to better indicate position in a way that is natural to people is a spectacular idea, as that would take minimal cognitive effort, and allow more focus on other aspects of the game. One suggestion I would have is perhaps making certain “categories” of sound effects. For example, background music speed indicates time remaining, low “thud” noises for when Mario runs into objects, higher-pitched repeating tones for enemies, etc. This would allow the player to quickly recognize sounds based on category, and respond accordingly.
Meetha –
I think the voice command method would work well for this game. Since there are only ever 4 possible moves, voicing that seems like a solution that is both clear, and precise. Adding vibrations or sounds for other notable events like being killed or getting a power-up, while removing sounds that do not aid in the gameplay is also a great idea. I would suggest that the voiceover could say things like “swipe right and then promptly swipe up” to solve the problem of two rapid succession moves.
Rosemary –
Informing the player which tiles they haven’t hit yet is an interesting problem to solve non-visually. I see why this was such a difficult component to find a solution to. I also see why the movement not being an exact “left” or “up” motion would be problematic when trying to convey this information to players. In trying to come up with feedback, really struggled to find something that I would have done differently. I basically came to the conclusion that this game doesn’t really lend itself to an audience that cannot see. The inherent nature of needing to convey that the player needs to move to a tile in a given location, effectively requires telling the player what to do. By giving step-by-step instructions, it really does take a lot of the “game” out of the experience. I think you made some great choices given the difficulty.
Alicia – Piano Keys
This was quite a creative pick and what a fun way to spend time on this assignment! I really enjoyed your description on how this game is played and the videos of how each of the different buffer types worked. You commented that it would be hard for gamers to receive a tone to play and a tone that they played correctly. Also commented that the key vibrating to indicate which one should be played would defeat the purpose of the game. What if instead you had the tone indicate which key to press and then the key vibrate indicating it was pressed correctly or possibly incorrectly?
Dan – Call of Duty WWII
Although I do commend you in attempting this assignment with a game that has so much going on at once, it is hard to even imagine a game like this being released with functions for a visually impaired individual. When attempting to find a game for this assignment I flirted with the idea of using Borderlands 2, a game my sister and I have played for years, but the complexity of it would have left the wizard to stressed to even try to describe what was going on. To even begin a remake of games like these, technological advancements would have to be well beyond what they are now, to allow a visually impaired person to be able to accurately fight their way across enemy infested beaches without repeatedly dying.
Mike – Space Invaders
One of the first games that I ever played growing up, I would marvel at the little noises it made and get so excited when I finally got all those ships. I really like your idea of a ‘missile targeting system’, that would really help a lot of the two dimension games like these with visually impaired users or even new users to know when to fire. The one thing I would suggest adding is some sort of vibration function, whether it be within the missile targeting or when the missile silo is about to be hit have the vibration slow intensify as the enemy missile gets closer. Any number of things. I understand the game is played on the computer, but I am sure some add-on could be created to add this functionality for users.
Pascal – Super Mario World
Until your presentation I really had not fully considered how hard it would be for someone to fully comprehend where they were on a two dimensional screen. You made some very valid points about the game lacking auditory ques allowing the user to know when a jump was fully activated or if they had landed on a platform that was higher. I can just imagine when the other modes come into play, the feather to fly and the flower to shoot fire balls. For a suggestion, the controllers back then never could vibrate. With new technology, use the vibrations to help users understand when enemies are near or when danger is close.
Meetha – Pac-Man
I enjoyed your description of the game in the videos of your game play. It is great that you recognize that the audio input in this game is overpowering and that it could be toned down with better ques that are not so jarring to the player. Though it is nice to know when it is safe to go after ghosts, it may not be necessary to have the noise be that loud and obnoxious. My suggestion for an update to this system would be some sort of radar, a way of knowing when the ghosts are closing in. You have the auditory input that ghosts are active on the screen but not they are approaching pac-man. Whether it is auditory or vibration, it would help the game a bit. This is another one where a pair of headphones with directional sounds might be useful to the player to avoid danger.
Colin – Guitar Hero
I liked your idea about creating a new control with non-visual users in mind. This is something that we often may need to consider when creating accessible programs. Not all interfaces are created equal. Many of the games that require the ability to see what is coming such as this bring me back to my piano lesson days where my instructor would play a note and I would be instructed to recreate the note on the piano. Though I do understand the auditory output on this game is more for the emersion, could there be notes added specifically for game play that are far enough apart that when played together the user would know to activate two keys. This can of course be shut off for players that find it unnecessary and distracting.
Alicia-
I liked the additional aspect of yours that brings in the difficulty of ambient noise. This definitely changes options that could be put in place to aid non-visual users. One addition I thought would be helpful would be adding an additional functionality to your proposed haptics, that varies the strength of the haptic feedback as an object gets closer to needing a button pressed. This was a non-visual user could “scan” their hands over the buttons and figure out based on the relative strengths of the haptic pulses which button will need to be pressed next, and the order of the next couple button presses after. This would also mesh well with the voice-over options you suggested.
Daniel-
I think a great improvement to the current Wizard of Oz you had in the video, would be adding more “commands” for specific directions. Depending on the background of the wizard and player, something such as degrees or clock values would have likely helped significantly. If the wizard was able to give the command “continue 35 degrees right” or “3 O’clock vertical, 1 O’clock horizontal strafe left” then you would have been able to have a richer experience.
One thing I did notice with your interaction versus my own, was that I knew the game quite well before playing, so I just used my wizard as a commander rather than interactive feature. Your experience was closer to a new user, and were asking your wizard what certain functionalities did as you tried them, and that definitely showed value of a two way interaction vs just a command interface. This could streamline commands to just what information you need, rather than the wizard just throwing all the information they think you need at you. This also accentuates that these interfaces may need to be streamlined for new users versus long term, as new users may want more information, and veteran users may want different or less information.
Michael-
For a few of your proposed design changes, haptics may be a better fit. For example, you mentioned a tone to let you know when you were against the side of the screen when moving. Holding down an arrow key would keep you pressed against the side, so rather than having a constant or initial tone for them hitting the edge, a haptic response might work well to give the player that information without interfering with the audio cues for the incoming enemies.
Another potential improvement would be the addition of dual channel audio. You mentioned using tones to indicate direction, but using left-right audio channels would allow for using the same tone, and make it easier for the user to know if they are getting closer or farther from the missile as they move. It would also allow for if multiple missiles are on the same screen at a time, the sound volumes would be different through each audio channel so multiple missiles could be tracked by audio at the same time.
Meetha-
Your point on reducing the amount of sounds definitely has merit. Something I noticed with your playthrough of Pac-Man vs the other games, is that Pac-Man has an overwhelming always on sound for the mouth opening and closing. None of the other games had that constant audio feedback in the forefront of the game.
With your suggestion of adding voice commands, it seemed the player had difficulty timing when to hit the button after the command to turn. A solution for this could be to add a tone after the voice command that fades out, so the command could be given a second earlier than the button needs to be hit, and that gives the player the time to interpret the command and then press the button as the tone is finishing. The tones could be more abrupt as well if the command needs to be immediate (in the case of two turns right after one another). Another option would be instead of an audio cue, a fading haptic cue. Either way, these may help solve the issue of the player latency.
Colin-
My suggestions for improvement are very similar to those I proposed for Alicia. A haptic option on the buttons that increased in power as the note got closer would allow for the user to scan the buttons to know which button needed to be pressed next, as well as when the haptic became full strength know to strum.
This would help significantly with the easier levels, where the space between notes would give the user time to rescan for what is next after each strum, but would break down when notes are coming in faster succession. Coupling this with your braille solution may allow for the faster scenarios.
Rosemary-
Adding a haptic functionality for scanning the tiles around the player may help with both the going off the edge issues and finding remaining tiles. If the controller gave haptic feedback for where the pointer was facing, the player could turn in each direction and get haptic feedback that the tile is un-activated or activated, or if the pointer is facing off the edge. Also, for the bounding ball issue, adding dual channel audio could indicate which direction the ball is currently advancing towards the player from.