Let's Reimagine the Future of Job Skills Training of Employees with Intellectual Disabilities
Let's Reimagine the Future of Job Skills Training of Employees with Intellectual Disabilities
Let's Reimagine the Future of Job Skills Training of Employees with Intellectual Disabilities
In association with New York's leading Nonprofits
In association with New York's leading Nonprofits
In association with New York's leading Nonprofits
May 2024 @ AIR Labs
Hats Worn
Published Accomplishments
Accessibility UX Research • End-to-end Product Design • Usability Testing • Spatial Interaction Design
• ACM CHI Late Breaking Work 2023
• ACM CHI Late Breaking Work 2024
Check Live Product
Live Site ↗
Collaborators
Pinaki Babar, Ziming Lee
Mentor: Roshan Peiris
Clients: Mike Barry, Director, Heritage Christian Services
Kathy Kanada, Director, The Arc Ontario
Platform
Web Responsive Interface, VR Interface
Web Responsive Interface, VR Interface
Timeline
20 months | Aug 2022 - May 2024
Context
Heritage Christian Services, The Arc Ontario, and Mozaic are leading non-profit human services agencies in New York state dedicated to empowering individuals of all abilities, including those with intellectual and developmental disabilities.
They offer a wide array of personalized services such as residential programs, employment training, and respite care, fostering independence and maximizing individual potential.
Heritage Christian Services, The Arc Ontario, and Mozaic are leading non-profit human services agencies in New York state dedicated to empowering individuals of all abilities, including those with intellectual and developmental disabilities. They offer a wide array of personalized services such as residential programs, employment training, and respite care, fostering independence and maximizing individual potential.
The organizations faced a big problem !
Their hands-on training programs, while effective, were labor-intensive and time-consuming, limiting their ability to scale and reach more individuals in need.
So, from this brief we posed a question:
How might we leverage technology to build a robust system that overcomes these limitations ?
So, from this brief we posed a question: How might we leverage technology to overcome these limitations ?
Introducing VRSkillBuilder–
a Virtual Reality and Artificial Intelligence based Job Skills Training platform that focuses on building real world skills in an accessible virtual world…
Skip to one of these juicy parts
Skip to one of these juicy parts
Our first step was to understand the training process at the organizations
Can technology create such environments virtually?
Opportunity 2
Can technology be able to reduce this human load?
Opportunity 1
Can technology be accessible, iterative and empathetic like a human?
Opportunity 3
Can technology be accessible, iterative and empathetic like a human?
Opportunity 3
Organizations pair trainees with job coaches for on-site training at workplaces until they can work independently.
To answer the previous questions, we conducted a Literature Review of various…
research methods
like role play training methodology
training domains
targeting daily living skills, cognitive skills and social communication skills
allied disabilities
like speech, audio or spatial and motor control challenges that the target users.
Everything narrowed down to Virtual Reality
Everything Narrowed down to Virtual Reality
Through Market Research we found Job Simulator by Oculus to be aligning with our study goals. So, we conducted a SWOT analysis on it.
Through a Workshop we then tested our demo scenarios vs Job Simulator with the organization directors, job coaches and trainees- and observed their interactions
Washing Hands Tutorial
To understand the efficacy of training methodology for information retention and knowledge application
Bathroom Cleaning Task
To see how people use probes to point out stains correctly in order to improve users' motor control/spatial balance
Retail Store Experience
To focus on communication aspects
Based on the score-based model of Job Simulator, we developed 3 lo-fidelity VR Demo Scenarios to test out with our stakeholders
Followed by a Design Sprint and In-depth Interviews to understand long term goals, challenges, potential ideas and behavioral aspects of training and stakeholder needs
We gained various insights through market research, workshop, design sprint and interviews
But, is VR really "The Solution"?
We gained various insights through market research, workshop, design sprint and interviews
We gained various insights through market research, workshop, design sprint and interviews
Our analysis results showed VR's clear potential
"Build simple scenarios that feel real!"
✅ Our demo scenarios were appreciated for their easy-to-use and step-by-step approach, whereas, Job Simulator was found complicated, especially for our targeted audience due to its complex multi-step processes
✅ Job coaches stressed on the idea of having real-world skills based learning relevant to trainees jobs
✅ Every trainee is different in terms of their abilities, interests and goals. So, an inclusive system is needed
❌ Both lacked realism! Need to build experiences that feel like real-world.
Hard Skills/Job related skills that trainees perform at workplaces. These vary with trainees and workplace dynamics.
Soft skills/communication skills that trainees lack and hamper their performance. These skills are transferable and could be applied irrespective of the workplace and trainee's job role.
We received various potential scenario ideas for future development. We found that they could be typically divided into two categories
But, is VR really "The Solution?"
We gained various insights through market research, workshop, design sprint and interviews
Washing Hands Tutorial
To understand the efficacy of training methodology for information retention and knowledge application
Bathroom Cleaning Task
To see how people use probes to point out stains correctly in order to improve users' motor control/spatial balance
Retail Store Experience
To focus on communication aspects
Through market research we found Job Simulator by Oculus to be aligning with our study goals. So, we conducted a SWOT analysis on it.
Based on the score-based model of Job Simulator, we developed 3 lo-fidelity VR Demo Scenarios to test out with stakeholders
We then tested our demo scenarios vs Job Simulator through a Workshop with our stakeholders-organization directors, job coaches and trainees and observed their interactions
Followed by a Design Sprint and In-depth Interviews to understand long term goals, challenges, potential ideas and behavioral aspects of the training process and stakeholder needs
Based on the key insights from previous study, we decided to build two skill-based sub-platforms
VRSpeechChatbot
for communication skills training and evaluation
VRSpeechChatbot
for communication skills training and evaluation
PanaromaJobExplorer
for job related hard skills training and evaluation
(Case study coming soon)
People with Intellectual disabilities face huge problem!
With more white paper research,
we found that people with intellectual disabilities suffer from low confidence due to lack of communication skills which eventually hampers their performance at workplaces.
We decided to focus on communication skills training because the skills are transferable which increases the scalability and adaptability of the platform not limiting to/dependant on specific jobs.
People with Intellectual disabilities face huge problem!
With more white paper research,
we found thatPIDs suffer from low confidence due to lack of communication skills which eventually hampers their performance at workplaces and employment retention rates.
We decided to focus on communication skills training because the skills are transferable which increases the scalability and adaptability of the platform not limiting to/dependant on specific jobs.
People with Intellectual disabilities face huge problem!
With more white paper research,
we found that PIDs suffer from low confidence due to lack of communication skills which eventually hampers their performance at workplaces leading to low employment retention rates
We decided to focus on communication skills training because the skills are transferable which increases the scalability and adaptability of the platform not limiting to/dependant on specific jobs.
VRSkillBuilder
Job Coach
The job coach is a dedicated and empathetic professional committed to empowering individuals with intellectual disabilities (PIDs) to achieve their employment goals. They possess deep understanding of the unique challenges faced by PIDs and are skilled in providing personalized support and guidance.
Train communication and interpersonal skills
Facilitate successful job placement and retention
Boost confidence and self-advocacy
Utilize innovative training methods and technologies
Trainee
The trainee is an individual with an intellectual disability who is motivated to learn and grow. They may face challenges in communication, social interaction, and adapting to new environments. However, they are eager to develop their skills and achieve independence in the workplace.
Improve communication and interpersonal skills
Secure meaningful employment
Gain confidence and overcome anxiety
We focused on these key archetypes
Job Coach
The job coach is a dedicated and empathetic professional committed to empowering individuals with intellectual disabilities (PIDs) to achieve their employment goals. They possess deep understanding of the unique challenges faced by PIDs and are skilled in providing personalized support and guidance.
Train communication and interpersonal skills
Facilitate successful job placement and retention
Boost confidence and self-advocacy
Utilize innovative training methods and technologies
Trainee
The trainee is an individual with an intellectual disability who is motivated to learn and grow. They may face challenges in communication, social interaction, and adapting to new environments. However, they are eager to develop their skills and achieve independence in the workplace.
Improve communication and interpersonal skills
Secure meaningful employment
Gain confidence and overcome anxiety
We focused on these key archetypes
Job Coach
The job coach is a dedicated and empathetic professional committed to empowering individuals with intellectual disabilities (PIDs) to achieve their employment goals. They possess deep understanding of the unique challenges faced by PIDs and are skilled in providing personalized support and guidance.
Train communication and interpersonal skills
Facilitate successful job placement and retention
Boost confidence and self-advocacy
Utilize innovative training methods and technologies
Trainee
The trainee is an individual with an intellectual disability who is motivated to learn and grow. They may face challenges in communication, social interaction, and adapting to new environments. However, they are eager to develop their skills and achieve independence in the workplace.
Improve communication and interpersonal skills
Secure meaningful employment
Gain confidence and overcome anxiety
We focused on these key archetypes
Early prototyping had its challenges !
Initially, we designed score-based training scenarios with step-by-step tutorials followed by multiple choice evaluations. These scenarios had workplace situations such as customer care and co-worker situations that taught trainees about conversational skills concepts. This system was slow, repetitive, too tiresome for users and limited in its scope.
Initial research and prototyping revealed the need for dynamic and real-time naturally flowing conversations.
We wanted a system that could adapt, so we designed an integrated platform that merges the power of LLMs with the immersive world of VR using OpenAI's API.
System Architecture
System Architecture
System Architecture
System Architecture
System Architecture
System Architecture
Our Solution
01
Web Application Interface for job coaches to write prompts, view/update scenarios and read chat transcripts
Web Application Interface for job coaches to write prompts, view/ update scenarios and read chat transcripts
User Onboarding
The job coaches can log into the system with a login and sign-up page. We created accounts for organizations that we work with. This would help us maintain
the privacy and safety of our users on the platform.
The job coaches can log into the system with a login and sign-up page. We created accounts for organizations that we work with. This would help us maintain the privacy and safety of our users on the platform.
Scenario library
Consists of a list of previously created scenarios designed by job coaches from the same organization arranged sequentially. The coaches can view, update, and use these scenarios for their training sessions.
Consists of a list
of previously created scenarios designed by job coaches from the same organization arranged sequentially. The coaches can view, update, and use these scenarios for their training sessions.
Create Scenario wizard
With a step-by-step guide, a scenario creation flow guides the user to
create a customized scenario that suits their training needs. The job coaches can select various options at write a prompt to AI at the end.
With a step-by-step guide, a scenario creation flow guides the user to create a customized scenario that suits their training needs. The job coaches can select various options at write a prompt to AI at the end.
Update/View/Edit Scenarios
The job coaches can view/ edit/update any scenarios they created and these changes will be real-time updated in the VR system and trainees can directly interact in the updated scenarios
Scenario transcripts
The transcript library has a repository of past sessions conducted in the VR
interface. The transcripts are loaded into the system in this library once the session ends. Job coaches can read through the trainees’ conversations and the virtual avatars
generated here.
The transcript library has a repository of past sessions conducted in the VR
interface. The transcripts are loaded into the system in this library once the session ends. Job coaches can read through the trainees’ conversations and the virtual avatarsgenerated here.
User Onboarding
The job coaches can log into the system with a login and sign-up page. We
created accounts for organizations that we work with. This would help us maintain the privacy and safety of our users on the platform.
02
VR Interface for trainees to experience the generated customized scenarios
Create Scenario wizard
With a step-by-step guide, a scenario creation flow guides the user to create a customized scenario that suits their training needs. The job coaches can select various options at write a prompt to AI at the end.
User Onboarding
The job coaches can log into the system with a login and sign-up page. We created accounts for organizations that we work with. This would help us maintain the privacy and safety of our users on the platform.
Update/View/Edit Scenarios
The job coaches can view/ edit/update any scenarios they created and these changes will be real-time updated in the VR system and trainees can directly interact in the updated scenarios
Scenario transcripts
The transcript library has a repository of past sessions conducted in the VR interface. The transcripts are loaded into the system in this library once the session ends. Job coaches can read through the trainees’ conversations and the virtual avatars generated here.
Scenario library
Consists of a list of previously created scenarios designed by job coaches from the same organization arranged sequentially. The coaches can view, update, and use these scenarios for
their training sessions.
User Testing
Conducting a workshop, longitudinal study and post-study interviews with job coaches and trainees revealed a positive user experience of VRSpeechChatbot System
5
5
remote workshops
25+
25+
training
sessions
training sessions
13
13
post-study interviews
and we found this
and we found this
and we found this
01
create prompts is the #1 feature that job coaches would adopt remains, validating this feature
All job coaches appreciated the flexibility creating the prompts gives them, making it a core feature. 83% job coaches found it easy to create customized prompts for training.
“
I enjoyed it and could see the variety of scenarios it could generate for training and teaching individuals tasks.
Participant 02
02
Trainees enjoy the VR training appreciating the safe environment and realistic graphics
Almost 90% of job coaches reported that their trainees enjoyed the VR training and were enthusiastic about using the technology. They also observed increased engagement and confidence in trainees during VR training sessions.
“
I was amazed that my trainee was nervous while answering the interview questions. It felt real.
Participant 09
03
Collaboration and knowledge sharing through scenarios interestingly came forward
67% of the coaches reported to have used other coaches' scenarios. This saves their time and effort and provides exposure to best training practices.
“
I often conduct multiple sessions continuously jumping to the scenarios in the library that other's have created.
Participant 04
04
Trainee evaluation through transcripts is wanted, just not regularly
While some job coaches found the existing transcripts sufficient for tracking trainee progress, others desired a more robust evaluation system.
“
It is nice to have it there somewhere, in case I am training remotely. Most times, I don't open it because I am there physically with the trainee.
Participant 12
Impact beyond numbers…
VRSpeechChatbot has been successfully implemented at New York state's leading organizations supporting employees with special needs. We published this groundbreaking research for two consecutive years at the prestigious ACM CHI Conferences 2023 & 2024 !
Post the release we observed an overall improvement in trainees' communication confidence and reducing organization's training costs per employee.
100+
user stories
5+
organizations pan USA
35+
collaborators & stakeholders
collaborators / stakeholders
50K+
Trainees with special needs
35+
collaborators & stakeholders
50K+
Trainees with special needs
The Journey of Empowerment
Lessons Learned and Paths Forged
VRSpeechChatbot project has been a transformative experience, providing me with valuable insights and lessons that extend beyond the research and technical aspects of the system.
Along the way, we encountered shifting priorities, changing roadmaps, and a minor launch delay, yet the satisfaction derived from this endeavor was immeasurable. This iterative design process helped us learn the importance of actively involving users throughout the design process to create a tool that truly meets their needs and fosters a sense of ownership.
I gained a deeper understanding of the importance of considering visual, auditory, and cognitive accessibility in every design decision to create an inclusive and user-friendly experience.
The enthusiasm of job coaches for sharing scenarios and best practices highlighted the power of community and knowledge sharing.
The project serves as a testament to the possibility of leveraging technologies like VR and LLM, when thoughtfully designed, can empower individuals with disabilities and create a more inclusive world…
More Projects
Vyominnova
Sustainable E-commerce platform selling digital interactive alternatives to conventional paper cards
Sustainable Design | Branding | Interaction Design
Ondemand
Redesigning the online food ordering system for Rochester Institute of Technology's on-campus services
UX Research | Info Architecture | Visual Design
Job Coach
The job coach is a dedicated and empathetic professional committed to empowering individuals with intellectual disabilities (PIDs) to achieve their employment goals. They possess deep understanding of the unique challenges faced by PIDs and are skilled in providing personalized support and guidance.
Train communication and interpersonal skills
Facilitate successful job placement and retention
Boost confidence and self-advocacy
Utilize innovative training methods and technologies
Trainee
The trainee is an individual with an intellectual disability who is motivated to learn and grow. They may face challenges in communication, social interaction, and adapting to new environments. However, they are eager to develop their skills and achieve independence in the workplace.
Improve communication and interpersonal skills
Secure meaningful employment
Gain confidence and overcome anxiety
Job Coach
The job coach is a dedicated and empathetic professional committed to empowering individuals with intellectual disabilities (PIDs) to achieve their employment goals. They possess deep understanding of the unique challenges faced by PIDs and are skilled in providing personalized support and guidance.
Train communication and interpersonal skills
Facilitate successful job placement and retention
Boost confidence and self-advocacy
Utilize innovative training methods and technologies
Trainee
The trainee is an individual with an intellectual disability who is motivated to learn and grow. They may face challenges in communication, social interaction, and adapting to new environments. However, they are eager to develop their skills and achieve independence in the workplace.
Improve communication and interpersonal skills
Secure meaningful employment
Gain confidence and overcome anxiety
The Journey of Empowerment
Lessons Learned and Paths Forged
VRSpeechChatbot project has been a transformative experience, providing me with valuable insights and lessons that extend beyond the research and technical aspects of the system.
Along the way, we encountered shifting priorities, changing roadmaps, and a minor launch delay, yet the satisfaction derived from this endeavor was immeasurable. This iterative design process helped us learn the importance of actively involving users throughout the design process to create a tool that truly meets their needs and fosters a sense of ownership.
I gained a deeper understanding of the importance of considering visual, auditory, and cognitive accessibility in every design decision to create an inclusive and user-friendly experience.
The enthusiasm of job coaches for sharing scenarios and best practices highlighted the power of community and knowledge sharing.
The project serves as a testament to the possibility of leveraging technologies like VR and LLM, when thoughtfully designed, can empower individuals with disabilities and create a more inclusive world…
Initial research and prototyping revealed the need for naturally flowing conversation generation.
Realizing the need for dynamism in the system, we integrated LLMs to VR using OpenAI's API to create a fully customized virtual chatbot system for real-time conversations on any topic!
Context
Heritage Christian Services, The Arc Ontario, and Mozaic are leading non-profit human services agencies in New York state dedicated to empowering individuals of all abilities, including those with intellectual and developmental disabilities. They offer a wide array of personalized services such as residential programs, employment training, and respite care, fostering independence and maximizing individual potential.
The organizations faced a big problem !
Their hands-on training programs, while effective, were labor-intensive and time-consuming, limiting their ability to scale and reach more individuals in need.
Job Coach
The job coach is a dedicated and empathetic professional committed to empowering individuals with intellectual disabilities (PIDs) to achieve their employment goals. They possess deep understanding of the unique challenges faced by PIDs and are skilled in providing personalized support and guidance.
Train communication and interpersonal skills
Facilitate successful job placement and retention
Boost confidence and self-advocacy
Utilize innovative training methods and technologies
Trainee
The trainee is an individual with an intellectual disability who is motivated to learn and grow. They may face challenges in communication, social interaction, and adapting to new environments. However, they are eager to develop their skills and achieve independence in the workplace.
Improve communication and interpersonal skills
Secure meaningful employment
Gain confidence and overcome anxiety
Initial research and prototyping revealed the need for dynamic and real-time naturally flowing conversations.
We wanted a system that could adapt, so we designed an integrated platform that merges the power of LLMs with the immersive world of VR using OpenAI's API.
May 2024 @ AIR Labs
Hats Worn
Published Accomplishments
Accessibility UX Research • End-to-end Product Design • Usability Testing • Spatial Interaction Design
• ACM CHI Late Breaking Work 2023
• ACM CHI Late Breaking Work 2024
Check Live Product
Live Site ↗
Collaborators
Pinaki Babar, Ziming Lee
Mentor: Roshan Peiris
Clients: Mike Barry, Director, Heritage Christian Services
Kathy Kanada, Director, The Arc Ontario
Platform
Web Responsive Interface, VR Interface
Timeline
20 months | Aug 2022 - May 2024
