We present ArtVis, an advanced user interface combining state-of-the-art visualisation techniques... more We present ArtVis, an advanced user interface combining state-of-the-art visualisation techniques and tangible interaction to explore the Web Gallery of Art digital artwork collection consisting of almost 28000 European artworks. The graphical ArtVis interface contains three separate visualisation panels that allow users to temporally, semantically and geographically investigate the digital artwork collection. The goal of ArtVis is that users can gain new insights by visually analysing and exploring a large data set rather than to only provide detailed information about individual artworks. This explorative ArtVis interaction style is further fostered by a tangible user interface where physical artefacts can be used as simple handles to drive the faceted browsing of the rich digital information space.
We present the results of an investigation on software support for the SMUIML multimodal user int... more We present the results of an investigation on software support for the SMUIML multimodal user interaction description language. In particular, we introduce a graphical UIDL editor for the creation of SMUIML scripts. The presented graphical editor is fully based on SMUIML for the representation of the underlying data as well as for the dialogue modelling. Due to the event-centered nature of SMUIML, the representation of the multimodal dialogue modelling in the graphical SMUIML dialogue editor has been realised via a state machine. The editor further offers a real-time graphical debugging tool. Compared to existing multimodal dialogue editors, the SMUIML graphical editor offers a dual graphical and textual editing as well as a number of operators for the temporal combination of modalities.
In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP... more In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP interaction styles. Existing multimodal fusion engines and frameworks range from low-level data stream-oriented approaches to high-level semantic in\-fer\-ence-based solutions. However, there is a lack of multimodal interaction engines offering native fusion support across different levels of abstractions to fully exploit the power of multimodal interactions. We present Mudra, a unified multimodal interaction framework supporting the integrated processing of low-level data streams as well as high-level semantic inferences. Our solution is based on a central fact base in combination with a declarative rule-based language to derive new facts at different abstraction levels. Our innovative architecture for multimodal interaction encourages the use of software engineering principles such as modularisation and composition to support a growing set of input modalities as well as to enable the integration of existing or novel multimodal fusion engines.
We present SpeeG, a multimodal speech- and body gesture-based text input system targeting media c... more We present SpeeG, a multimodal speech- and body gesture-based text input system targeting media centres, set-top boxes and game consoles. Our controller-free zoomable user interface combines speech input with a gesture-based real-time correction of the recognised voice input. While the open source CMU Sphinx voice recogniser transforms speech input into written text, Microsoft's Kinect sensor is used for the hand gesture tracking. A modified version of the zoomable Dasher interface combines the input from Sphinx and the Kinect sensor. In contrast to existing speech error correction solutions with a clear distinction between a detection and correction phase, our innovative SpeeG text input system enables continuous real-time error correction. An evaluation of the SpeeG prototype has revealed that low error rates for a text input speed of about six words per minute can be achieved after a minimal learning phase. Moreover, in a user study SpeeG has been perceived as the fastest of all evaluated user interfaces and therefore represents a promising candidate for future controller-free text input.
We present ArtVis, an application combining advanced visualisation techniques and tangible intera... more We present ArtVis, an application combining advanced visualisation techniques and tangible interaction to explore a large digital collection of almost 28000 European artworks managed by the Web Gallery of Art. In order to get new insights by exploring, analysing and browsing the artworks, our graphical ArtVis user interface offers three complementary but synchronised visualisation components. We further developed a tangible ArtVis user interface for the playful exploration and seamless integration of the digital artwork collection with physical artefacts. A formative evaluation of the ArtVis prototype revealed that users are able to answer relatively difficult questions as well as get some new insights based on the vast amount of data. A second user evaluation of the tangible ArtVis interface has shown that this sort of physical interaction attracts users and stimulates them to further explore the digital artwork collection.
The advent of advanced mobile devices in combination with new interaction modalities and methods ... more The advent of advanced mobile devices in combination with new interaction modalities and methods for the tracking of contextual information, opens new possibilities in the field of context-aware user interface adaptation. One particular research direction is the automatic context-aware adaptation of input modalities in multimodal mobile interfaces. We present existing adaptive multimodal mobile input solutions and position them within closely related research fields. Based on a detailed analysis of the state of the art, we propose eight design guidelines for adaptive multimodal mobile input solutions. The use of these guidelines is further illustrated through the design and development of an adaptive multimodal calendar application.
We present the Visual Data Explorer (ViDaX), a tool for visualising and exploring large RDF data ... more We present the Visual Data Explorer (ViDaX), a tool for visualising and exploring large RDF data sets. ViDaX enables the extraction of information from RDF data sources and offers functionality for the analysis of various data characteristics as well as the exploration of the corresponding ontology graph structure. In addition to some basic data mining features, our interactive semantic data visualisation and exploration tool offers various types of visualisations based on the type of data. In contrast to existing semantic data visualisation solutions, ViDaX also offers non-expert users the possibility to explore semantic data based on powerful automatic visualisation and interaction techniques without the need for any low-level programming. To illustrate some of ViDaX's functionality, we present a use case based on semantic data retrieved from DBpedia, a semantic version of the well-known Wikipedia online encyclopedia, which forms a major component of the emerging linked data initiative.
We present the results of an investigation on software support for the SMUIML multimodal user int... more We present the results of an investigation on software support for the SMUIML multimodal user interaction description language. In particular, we introduce a graphical UIDL editor for the creation of SMUIML scripts. The representation of data as well as the dialogue modelling in our graphical editor is based on the SMUIML language. Due to the event-centered nature of SMUIML, multimodal dialogue modelling is represented by a state machine in the graphical SMUIML dialogue editor. The editor further offers a real-time graphical debugging tool. Compared to existing multimodal dialogue editing solutions, the SMUIML graphical editor offers synchronised dual editing in graphical and textual form as well as a number of operators for the temporal combination of modalities. The presented graphical editor represents the third component of a triad of tools for the development of multimodal user interfaces, consisting of an XML-based modelling language, a framework for the authoring of multimodal interfaces and a graphical editor.
Multimodal interfaces have shown to be ideal candidates for interactive systems that adapt to a u... more Multimodal interfaces have shown to be ideal candidates for interactive systems that adapt to a user either automatically or based on user-defined rules. However, user-based adaptation demands for the corresponding advanced software architectures and algorithms. We present a novel multimodal fusion algorithm for the development of adaptive interactive systems which is based on hidden Markov models~(HMM). In order to select relevant modalities at the semantic level, the algorithm is linked to temporal relationship properties. The presented algorithm has been evaluated in three use cases from which we were able to identify the main challenges involved in developing adaptive multimodal interfaces.
ISWC'14 Adjunct Proceedings of the 2014 International Symposium on Wearable Computers, Sep 14, 2014
This position paper discusses how computer science and fashion design can gracefully enrich each ... more This position paper discusses how computer science and fashion design can gracefully enrich each other to address privacy, non-invasiveness and nondisruptiveness issues. A use case providing intimate, remote communication for a couple is described, as well as potential tracks to solve this use case from a technological as well as fashion design points of view. We show that fashion designers closely collaborating with computer scientists can help address complex issues such as privacy when integrating smart garments together.
Abstract Multimodal interfaces have shown to be ideal candidates for interactive systems that ada... more Abstract Multimodal interfaces have shown to be ideal candidates for interactive systems that adapt to a user either automatically or based on user-defined rules. However, user-based adaptation demands for the corresponding advanced software architectures and algorithms. We present a novel multimodal fusion algorithm for the development of adaptive interactive systems which is based on hidden Markov models (HMM).
Abstract We present SpeeG, a multimodal speech-and body gesture-based text input system targeting... more Abstract We present SpeeG, a multimodal speech-and body gesture-based text input system targeting media centres, set-top boxes and game consoles. Our controller-free zoomable user interface combines speech input with a gesture-based real-time correction of the recognised voice input. While the open source CMU Sphinx voice recogniser transforms speech input into written text, Microsoft's Kinect sensor is used for the hand gesture tracking.
Abstract We present the Visual Data Explorer (ViDaX), a tool for visualising and exploring large ... more Abstract We present the Visual Data Explorer (ViDaX), a tool for visualising and exploring large RDF data sets. ViDaX enables the extraction of information from RDF data sources and offers functionality for the analysis of various data characteristics as well as the exploration of the corresponding ontology graph structure. In addition to some basic data mining features, our interactive semantic data visualisation and exploration tool offers various types of visualisations based on the type of data.
In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP... more In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP interaction styles. Existing multimodal fusion engines and frameworks range from low-level data stream-oriented approaches to high-level semantic in\-fer\-ence-based solutions. However, there is a lack of multimodal interaction engines offering native fusion support across different levels of abstractions to fully exploit the power of multimodal interactions. We present Mudra, a unified multimodal interaction framework supporting the integrated processing of low-level data streams as well as high-level semantic inferences. Our solution is based on a central fact base in combination with a declarative rule-based language to derive new facts at different abstraction levels. Our innovative architecture for multimodal interaction encourages the use of software engineering principles such as modularisation and composition to support a growing set of input modalities as well as to enable the integration of existing or novel multimodal fusion engines.
We present ArtVis, an advanced user interface combining state-of-the-art visualisation techniques... more We present ArtVis, an advanced user interface combining state-of-the-art visualisation techniques and tangible interaction to explore the Web Gallery of Art digital artwork collection consisting of almost 28000 European artworks. The graphical ArtVis interface contains three separate visualisation panels that allow users to temporally, semantically and geographically investigate the digital artwork collection. The goal of ArtVis is that users can gain new insights by visually analysing and exploring a large data set rather than to only provide detailed information about individual artworks. This explorative ArtVis interaction style is further fostered by a tangible user interface where physical artefacts can be used as simple handles to drive the faceted browsing of the rich digital information space.
We present the results of an investigation on software support for the SMUIML multimodal user int... more We present the results of an investigation on software support for the SMUIML multimodal user interaction description language. In particular, we introduce a graphical UIDL editor for the creation of SMUIML scripts. The presented graphical editor is fully based on SMUIML for the representation of the underlying data as well as for the dialogue modelling. Due to the event-centered nature of SMUIML, the representation of the multimodal dialogue modelling in the graphical SMUIML dialogue editor has been realised via a state machine. The editor further offers a real-time graphical debugging tool. Compared to existing multimodal dialogue editors, the SMUIML graphical editor offers a dual graphical and textual editing as well as a number of operators for the temporal combination of modalities.
In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP... more In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP interaction styles. Existing multimodal fusion engines and frameworks range from low-level data stream-oriented approaches to high-level semantic in\-fer\-ence-based solutions. However, there is a lack of multimodal interaction engines offering native fusion support across different levels of abstractions to fully exploit the power of multimodal interactions. We present Mudra, a unified multimodal interaction framework supporting the integrated processing of low-level data streams as well as high-level semantic inferences. Our solution is based on a central fact base in combination with a declarative rule-based language to derive new facts at different abstraction levels. Our innovative architecture for multimodal interaction encourages the use of software engineering principles such as modularisation and composition to support a growing set of input modalities as well as to enable the integration of existing or novel multimodal fusion engines.
We present SpeeG, a multimodal speech- and body gesture-based text input system targeting media c... more We present SpeeG, a multimodal speech- and body gesture-based text input system targeting media centres, set-top boxes and game consoles. Our controller-free zoomable user interface combines speech input with a gesture-based real-time correction of the recognised voice input. While the open source CMU Sphinx voice recogniser transforms speech input into written text, Microsoft's Kinect sensor is used for the hand gesture tracking. A modified version of the zoomable Dasher interface combines the input from Sphinx and the Kinect sensor. In contrast to existing speech error correction solutions with a clear distinction between a detection and correction phase, our innovative SpeeG text input system enables continuous real-time error correction. An evaluation of the SpeeG prototype has revealed that low error rates for a text input speed of about six words per minute can be achieved after a minimal learning phase. Moreover, in a user study SpeeG has been perceived as the fastest of all evaluated user interfaces and therefore represents a promising candidate for future controller-free text input.
We present ArtVis, an application combining advanced visualisation techniques and tangible intera... more We present ArtVis, an application combining advanced visualisation techniques and tangible interaction to explore a large digital collection of almost 28000 European artworks managed by the Web Gallery of Art. In order to get new insights by exploring, analysing and browsing the artworks, our graphical ArtVis user interface offers three complementary but synchronised visualisation components. We further developed a tangible ArtVis user interface for the playful exploration and seamless integration of the digital artwork collection with physical artefacts. A formative evaluation of the ArtVis prototype revealed that users are able to answer relatively difficult questions as well as get some new insights based on the vast amount of data. A second user evaluation of the tangible ArtVis interface has shown that this sort of physical interaction attracts users and stimulates them to further explore the digital artwork collection.
The advent of advanced mobile devices in combination with new interaction modalities and methods ... more The advent of advanced mobile devices in combination with new interaction modalities and methods for the tracking of contextual information, opens new possibilities in the field of context-aware user interface adaptation. One particular research direction is the automatic context-aware adaptation of input modalities in multimodal mobile interfaces. We present existing adaptive multimodal mobile input solutions and position them within closely related research fields. Based on a detailed analysis of the state of the art, we propose eight design guidelines for adaptive multimodal mobile input solutions. The use of these guidelines is further illustrated through the design and development of an adaptive multimodal calendar application.
We present the Visual Data Explorer (ViDaX), a tool for visualising and exploring large RDF data ... more We present the Visual Data Explorer (ViDaX), a tool for visualising and exploring large RDF data sets. ViDaX enables the extraction of information from RDF data sources and offers functionality for the analysis of various data characteristics as well as the exploration of the corresponding ontology graph structure. In addition to some basic data mining features, our interactive semantic data visualisation and exploration tool offers various types of visualisations based on the type of data. In contrast to existing semantic data visualisation solutions, ViDaX also offers non-expert users the possibility to explore semantic data based on powerful automatic visualisation and interaction techniques without the need for any low-level programming. To illustrate some of ViDaX's functionality, we present a use case based on semantic data retrieved from DBpedia, a semantic version of the well-known Wikipedia online encyclopedia, which forms a major component of the emerging linked data initiative.
We present the results of an investigation on software support for the SMUIML multimodal user int... more We present the results of an investigation on software support for the SMUIML multimodal user interaction description language. In particular, we introduce a graphical UIDL editor for the creation of SMUIML scripts. The representation of data as well as the dialogue modelling in our graphical editor is based on the SMUIML language. Due to the event-centered nature of SMUIML, multimodal dialogue modelling is represented by a state machine in the graphical SMUIML dialogue editor. The editor further offers a real-time graphical debugging tool. Compared to existing multimodal dialogue editing solutions, the SMUIML graphical editor offers synchronised dual editing in graphical and textual form as well as a number of operators for the temporal combination of modalities. The presented graphical editor represents the third component of a triad of tools for the development of multimodal user interfaces, consisting of an XML-based modelling language, a framework for the authoring of multimodal interfaces and a graphical editor.
Multimodal interfaces have shown to be ideal candidates for interactive systems that adapt to a u... more Multimodal interfaces have shown to be ideal candidates for interactive systems that adapt to a user either automatically or based on user-defined rules. However, user-based adaptation demands for the corresponding advanced software architectures and algorithms. We present a novel multimodal fusion algorithm for the development of adaptive interactive systems which is based on hidden Markov models~(HMM). In order to select relevant modalities at the semantic level, the algorithm is linked to temporal relationship properties. The presented algorithm has been evaluated in three use cases from which we were able to identify the main challenges involved in developing adaptive multimodal interfaces.
ISWC'14 Adjunct Proceedings of the 2014 International Symposium on Wearable Computers, Sep 14, 2014
This position paper discusses how computer science and fashion design can gracefully enrich each ... more This position paper discusses how computer science and fashion design can gracefully enrich each other to address privacy, non-invasiveness and nondisruptiveness issues. A use case providing intimate, remote communication for a couple is described, as well as potential tracks to solve this use case from a technological as well as fashion design points of view. We show that fashion designers closely collaborating with computer scientists can help address complex issues such as privacy when integrating smart garments together.
Abstract Multimodal interfaces have shown to be ideal candidates for interactive systems that ada... more Abstract Multimodal interfaces have shown to be ideal candidates for interactive systems that adapt to a user either automatically or based on user-defined rules. However, user-based adaptation demands for the corresponding advanced software architectures and algorithms. We present a novel multimodal fusion algorithm for the development of adaptive interactive systems which is based on hidden Markov models (HMM).
Abstract We present SpeeG, a multimodal speech-and body gesture-based text input system targeting... more Abstract We present SpeeG, a multimodal speech-and body gesture-based text input system targeting media centres, set-top boxes and game consoles. Our controller-free zoomable user interface combines speech input with a gesture-based real-time correction of the recognised voice input. While the open source CMU Sphinx voice recogniser transforms speech input into written text, Microsoft's Kinect sensor is used for the hand gesture tracking.
Abstract We present the Visual Data Explorer (ViDaX), a tool for visualising and exploring large ... more Abstract We present the Visual Data Explorer (ViDaX), a tool for visualising and exploring large RDF data sets. ViDaX enables the extraction of information from RDF data sources and offers functionality for the analysis of various data characteristics as well as the exploration of the corresponding ontology graph structure. In addition to some basic data mining features, our interactive semantic data visualisation and exploration tool offers various types of visualisations based on the type of data.
In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP... more In recent years, multimodal interfaces have gained momentum as an alternative to traditional WIMP interaction styles. Existing multimodal fusion engines and frameworks range from low-level data stream-oriented approaches to high-level semantic in\-fer\-ence-based solutions. However, there is a lack of multimodal interaction engines offering native fusion support across different levels of abstractions to fully exploit the power of multimodal interactions. We present Mudra, a unified multimodal interaction framework supporting the integrated processing of low-level data streams as well as high-level semantic inferences. Our solution is based on a central fact base in combination with a declarative rule-based language to derive new facts at different abstraction levels. Our innovative architecture for multimodal interaction encourages the use of software engineering principles such as modularisation and composition to support a growing set of input modalities as well as to enable the integration of existing or novel multimodal fusion engines.
Uploads
Papers by Bruno Dumas
A video about the system can be found at: https://www.academia.edu/11590819/SpeeG
A video about the system can be found at: https://www.academia.edu/11590819/SpeeG