Abstract:
Robotic systems are increasingly deployed across industrial and service domains; however, their adoption remains limited by the scarcity of skilled programmers. Training domain experts to configure robotic systems is resource-intensive, motivating the need for more accessible programming approaches. End-User Development (EUD) aims to enable non-experts to create and modify software (Lieberman, Paternò, Klann, & Wulf, 2006), with End-User Programming (EUP) focusing on simplified programming paradigms. In robotics, this leads to End-User Robot Programming (EURP), which seeks to democratize robot interaction (Ajaykumar, Steele, & Huang, 2021). Existing approaches, including visual programming and programming by demonstration, reduce complexity but still impose structured interaction models (Vaiani & Paternò, 2024). Recent advances in Large Language Models (LLMs) offer a complementary paradigm by supporting free-form natural language interaction (Karli, Chen, Antony, & Huang, 2024).
In our previous work, we introduced a web-based system that combines an LLM with a Domain-Specific Language (DSL) to support intuitive yet verifiable end-user programming of a robot arm. DSLs provide structured, domain-tailored representations (Fowler, 2010; Mernik, Heering, & Sloane, 2005), while LLMs enable unconstrained user input. Our approach integrates both by translating natural language instructions into a reviewable intermediate representation based on Behavior-Driven Development (BDD) (Smart & Molak, 2023) scenarios. Users interact through a conversational interface, formulating their intentions via text or speech. The LLM then generates one or more scenarios expressed in a BDD-inspired DSL using a Given–When–Then structure that can be visualized either as text or as a block-based representation. This intermediate layer makes robot behavior explicit and human-readable, allowing users to inspect and edit generated programs before execution. In contrast to fully automated LLM-based systems, this design emphasizes transparency and user control.
The demonstration extends this work by coupling this EUP framework with an immersive virtual reality (VR) environment in which a mobile manipulator executes logistics tasks in a warehouse. VR provides a realistic yet safe setting for observing robot behavior, improving spatial understanding compared to traditional 2D simulations. It also opens opportunities for future multimodal interaction, such as gesture-based or spatial commands, complementing natural language input. Our contribution illustrates how combining LLMs with a structured DSL enables a hybrid interaction paradigm that balances usability and reliability. The BDD-based intermediate layer serves as a bridge between free-form user intent and executable robot behavior, supporting validation, transparency, and trust. By integrating conversational programming with immersive visualization, the system contributes to ongoing efforts in EURP to broaden access to robotic technologies for non-experts.
In our previous work, we introduced a web-based system that combines an LLM with a Domain-Specific Language (DSL) to support intuitive yet verifiable end-user programming of a robot arm. DSLs provide structured, domain-tailored representations (Fowler, 2010; Mernik, Heering, & Sloane, 2005), while LLMs enable unconstrained user input. Our approach integrates both by translating natural language instructions into a reviewable intermediate representation based on Behavior-Driven Development (BDD) (Smart & Molak, 2023) scenarios. Users interact through a conversational interface, formulating their intentions via text or speech. The LLM then generates one or more scenarios expressed in a BDD-inspired DSL using a Given–When–Then structure that can be visualized either as text or as a block-based representation. This intermediate layer makes robot behavior explicit and human-readable, allowing users to inspect and edit generated programs before execution. In contrast to fully automated LLM-based systems, this design emphasizes transparency and user control.
The demonstration extends this work by coupling this EUP framework with an immersive virtual reality (VR) environment in which a mobile manipulator executes logistics tasks in a warehouse. VR provides a realistic yet safe setting for observing robot behavior, improving spatial understanding compared to traditional 2D simulations. It also opens opportunities for future multimodal interaction, such as gesture-based or spatial commands, complementing natural language input. Our contribution illustrates how combining LLMs with a structured DSL enables a hybrid interaction paradigm that balances usability and reliability. The BDD-based intermediate layer serves as a bridge between free-form user intent and executable robot behavior, supporting validation, transparency, and trust. By integrating conversational programming with immersive visualization, the system contributes to ongoing efforts in EURP to broaden access to robotic technologies for non-experts.