AJJEN JOSHI
Research Scientist @ Smart Eye
Creative @ ajjenjoshi.com

ABOUT


I am currently Lead Computer Vision Scientist at Smart Eye , where I work on developing machine-learning-based driver and cabin monitoring technologies to improve road safety and enhance in-vehicle mobility experiences. Previously, I worked as a senior research scientist at Affectiva (acquired by Smart Eye in June 2021), where I worked on researching novel methods for analyzing in-vehicle human behavior, including: encoder-decoder models for multi-task facial attribute prediction and face generation, GAN models for facial attribute manipulation, gaze estimation using synthetic data and semi-supervised learning, drowsiness detection using generative data augmentation, multimodal frustration detection, and boredom mitigation using affective conversational agents.

I received my PhD in 2018 under the supervision of Dr. Margrit Betke and Dr. Stan Sclaroff with the Image and Video Computing research group at Boston University's Department of Computer Science. I attained a Master of Science degree in Computer Science from Boston University in 2014, and a Bachelor of Arts degree from Connecticut College in 2012 where I majored in Computer Science and Architectural Studies. During my PhD, I interned at Disney Research (Summer 2015) and Adobe Research (Summer 2016).

In my spare time, I do freelance work in art and photography. Some of my work can be seen at ajjenjoshi.com.

RESEARCH


The Universal Face Encoder: Learning Disentangled Representations across Different Attributes
Sandipan Banerjee, Ajjen Joshi, Jay Turcot.
IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW) on Biometrics 2023. Oral.

[Paper] | [Supplement]



ATL-BP: A Student Engagement Dataset and Model for Affect Transfer Learning for Behavior Prediction
Nataniel Ruiz, Hao Yu, Danielle A Allessio, Mona Jalal, Ajjen Joshi, Thomas Murray, John Magee, Kevin Manuel Delgado, Vitaly Ablavsky, Stan Sclaroff, Ivon Arroyo, Beverly P Woolf, Sarah Adel Bargal, Margrit Betke.
IEEE Transactions on Biometrics, Behavior, and Identity Science (TBIOM) 2022.

[Paper]



Driver Glance Classification In-the-wild: Towards Generalization Across Domains and Subjects
Sandipan Banerjee, Ajjen Joshi, Jay Turcot, Bryan Reimer, Taniya Mishra.
IEEE International Conference on Automatic Face and Gesture Recognition (AFGR) 2021. Poster.

[Paper]



Leveraging Affect Transfer Learning for Behavior Prediction in an Intelligent Tutoring System
Nataniel Ruiz, Hao Yu, Danielle Allessio, Mona Jalal, Ajjen Joshi, Thomas Murray, John Magee, Jacob Whitehill, Vitaly Ablavsky, Ivon Arroyo, Beverly Woolf, Stan Sclaroff, Margrit Betke.
IEEE International Conference on Automatic Face and Gesture Recognition (AFGR) 2021. Oral.

[Paper]



LEGAN: Disentangled Manipulation of Directional Lighting and Facial Expressions by Leveraging Human Perceptual Judgements
Sandipan Banerjee, Ajjen Joshi, Sneha Bhattacharya, Prashant Mahajan, Survi Kyal, Taniya Mishra
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop on Analysis and Modeling of Faces and Gestures (AMFG) 2021. Oral.

[Paper] | [Poster]



Synthesize and Learn: Jointly Optimizing Generative and Classifier Networks for Improved Drowsiness Detection
Sandipan Banerjee, Ajjen Joshi, Ahmed Ghoneim, Survi Kyal, Taniya Mishra
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021. Poster.

[Paper] | [Poster]



In-the-wild Drowsiness Detection from Facial Expressions
Ajjen Joshi, Survi Kyal, Sandipan Banerjee, Taniya Mishra
IEEE Intelligent Vehicles Symposium (IV) Workshop on Human Sensing in Intelligent Mobility , 2020. Oral.

[Paper]



Mitigating Boredom Using an Empathetic Conversational Agent
Samiha Samrose, Kavya Anbarasu, Ajjen Joshi, Taniya Mishra
ACM International Conference on Intelligent Virtual Agents (IVA), 2020. Oral.

[Paper]



Swipe&Switch: Text Entry Using Gaze Paths and Context Switching
Andrew Kurauchi, Wenxin Feng, Ajjen Joshi, Carlos Morimoto, Margrit Betke
ACM Symposium on User Interface Software and Technology (UIST), 2020. Poster.

[Paper] | [Poster]



Protocol for Eliciting Driver Frustration in an In-vehicle Environment
Ajjen Joshi, Youssef Attia, Taniya Mishra
IEEE International Conference on Affective Computing and Intelligent Interaction (ACII), 2019. Poster.

[Paper] | [Poster]



Affect-driven Learning Outcomes Predictions in Intelligent Tutoring Systems
Ajjen Joshi, Danielle Allessio, John Magee, Jacob Whitehill, Ivon Arroyo, Beverly Woolf, Stan Sclaroff, Margrit Betke
IEEE International Conference on Automatic Face and Gesture Recognition (AFGR), 2019. Poster.

[Paper] | [Poster]



Personalized Face and Gesture Analysis Using Hierarchical Neural Networks
Ajjen Joshi
[PhD Thesis]



Enabling Early Gesture Recognition by Motion Augmentation
Rohit Agrawal, Ajjen Joshi, Margrit Betke
ACM International Conference on Pervasive Technologies Related to Assistive Environments (PETRA), 2018. Oral.

[Paper]



Context-Sensitive Prediction of Facial Expressivity Using Multimodal Hierarchical Bayesian Neural Networks
Ajjen Joshi, Soumya Ghosh, Sarah Gunnery, Linda Tickle-Degnen, Margrit Betke, Stan Sclaroff
IEEE International Conference on Automatic Face and Gesture Recognition (AFGR), 2018. Poster.

[Paper] | [Poster]



Personalizing Gesture Recognition Using Hierarchical Bayesian Neural Networks
Ajjen Joshi, Soumya Ghosh, Margrit Betke, Stan Sclaroff, Hanspeter Pfister
Conference on Computer Vision and Pattern Recognition (CVPR), 2017. Poster.

[Paper] | [Supplement] | [Poster]



ExerciseCheck: Remote Monitoring and Evaluation Platform for Home Based Physical Therapy
Elham Saraee, Saurabh Singh, Kathryn Hendron, Mingxin Zheng, Ajjen Joshi, Terry Ellis, Margrit Betke
ACM International Conference on Pervasive Technologies Related to Assistive Environments (PETRA), 2017. Oral.

[Paper]



A Therapeutic Robotic System for the Upper Body based on the Proficio Robotic Arm
Elham Saraee, Ajjen Joshi, Margrit Betke
IEEE International Conference on Virtual Rehabilitation (ICVR), 2017. Poster.

[Paper] | [Poster] | [Spotlight]



PostureCheck: Posture Modeling for Exercise Assessment using the Microsoft Kinect
Elham Saraee, Saurabh Singh, Ajjen Joshi, Margrit Betke
IEEE International Conference on Virtual Rehabilitation (ICVR), 2017. Poster.

[Paper] | [Poster] | [Spotlight]



Hierarchical Bayesian Neural Networks for Personalized Classification
Ajjen Joshi, Soumya Ghosh, Margrit Betke, Hanspeter Pfister
Neural Information Processing Systems (NIPS) Workshop on Bayesian Deep Learning, 2016. Poster.

[Paper] | [Poster] | [Spotlight]


Predicting Active Facial Expressivity in People with Parkinson's Disease
Ajjen Joshi, Linda Tickle-Degnen, Sarah Gunnery, Terry Ellis, Margrit Betke
ACM International Conference on Pervasive Technologies Related to Assistive Environments (PETRA), 2016. Oral.

[Paper]


Comparing Random Forest Approaches To Segmenting And Classifying Gestures
Ajjen Joshi, Camille Monnier, Margrit Betke, Stan Sclaroff
Elsevier Image and Vision Computing (IMAVIS), 2016.

[Paper]


Eyeswipe: Dwell-Free Text Entry Using Gaze Paths
Andrew Kurauchi, Wenxin Feng, Ajjen Joshi, Carlos Morimoto, Margrit Betke
ACM Conference on Human Factors in Computing Systems (CHI), 2016. Oral.

[Paper] | [Video]


b3.js: A Library for Interactive Virtual Reality Web 3D Graphs
Huy Le, Ajjen Joshi, Margrit Betke
IEEE Virtual Reality (VR), 2016. Research Demo.

[Paper] | [Video]


A Random Forest Approach to Segmenting And Classifying Gestures
Ajjen Joshi, Camille Monnier, Margrit Betke, Stan Sclaroff
IEEE International Conference on Automatic Face and Gesture Recognition (AFGR), 2015. Oral.

[MS Thesis] | [Paper] | [Video]


Developing a Tool for Dance Motion Synthesis
Ajjen Joshi, Bridget Baird, Ozgur Izmirli
Connecticut College Biennial Symposium on Arts and Technology (SAT), 2012. Oral.

[Paper]

CONTACT