You are here: MyURC.org > publications > Use of User Interface Sockets to Create Naturally Evolving Intelligent Environments

Use of User Interface Sockets to Create Naturally Evolving Intelligent Environments

Gregg C Vanderheiden Ph.D.
Gottfried Zimmermann Ph.D.

Trace R & D Center
University of Wisconsin – Madison

Vanderheiden, G. C., & Zimmermann, G. (2005). Use of user interface sockets to create naturally evolving intelligent environments. 11th International Conference on Human-Computer Interaction, Caesars Palace, Las Vegas, Nevada USA.

Abstract

A new standard under development by the V2 Technical Committee of INCITS may provide a mechanism for creating naturally evolving intelligent environments. The standard is designed to provide a mechanism for any (conforming) product (device or service) to allow other devices to act as an alternate interface (controller) for the product. Controllers include assistive technologies, PDAs and universal remote controls. This also opens the way for intelligent agent control of everyday appliances as well as net appliances. The standard is based on the concept of a "user interface socket" that allows modality independent control of the product. Implications for natural language control are discussed. Such a standard, if implemented widely, would allow people to purchase ‘unintelligent’ products today that could later be controlled intelligently when an intelligent ‘controller’ was brought into their midst. It would also allow for a straightforward evolution of intelligence in the home without having to update either networking or the products themselves.

1. Introduction

Much has been written and proposed regarding the development of ambient intelligence and intelligent environments. Past work however, has often focused on the creation of carefully architect environments with predefined networks and control structures. Some smart houses have been created with special cabling and infrastructure.

However, the costs to custom wire a house or even to purchase a set of smart appliances is beyond the means of most households. Thus, these activities have been confined to those who are both very affluent and technically adventuresome. In order for this to take hold on a mass basis, it must be inexpensive to get started and possible to ease into an intelligent environment gradually through a natural process of attrition and replacement of household technologies.

It is proposed that the use of standard household networking technologies (WiFi and HomePlug, for example) combined with the new user interface socket concept can provide a means for a low cost and natural, gradual evolution of a standard home or office environment into an intelligent controllable environment.

2. The ubiquitous network

The first of these required technologies (ubiquitous network) is already evolving. It consists of a wide variety of interconnected networked technologies in the home. The backbone may either be something like WiFi or Ethernet. Some type of power line network would probably be used to connect smaller appliances (such as alarm clocks, coffee pots, etc.) because of its ease, low cost and transparency. Low cost wireless technologies like Bluetooth or Zigbee could also be used as their costs fall.

We are already seeing cell phones that work as regular cell phones when the user is about in the community, but which change into VoIP/WiFi phones whenever the person approaches their house. Within the home, we have the ability for devices connected with a wide variety of wired and wireless technologies to all interact on the same network transparently. With the advent of powerline Ethernet we will soon have the ability for people to connect to the internet by simply walking into their house and to connect new devices into their home network, by simply plugging them into the electrical outlet.

3. Interface Sockets

The second of the required technologies (interface sockets) is being introduced in the form of a new ANSI/INCITS set of standards from the V2 Technical Committee. These standards specify communications between a Target that a user wishes to access and operate, and a Universal Remote Console (URC) that presents the user with a remote user interface through which they can discover, select, access and operate the Target. The URC is software that is typically hosted on the user's physical device, but a distributed approach is also possible. Figure 1 show a number of possible URCs on the left and some typical "Target" devices on the right.

 

Figure 1: URCs can include PDAs, cell phones, special watches and assistive technologies. Target devices can include home entertainment devices, clocks, thermostats, copiers, ATMs, and elevators as well as invisible targets like Web services.

The user interface socket is a low level description of a particular target product or service and specifies its operational input and output requirements. The socket describes the functionality and state of the target as a set of typed data and command elements. The data elements on the socket provide all of the data that are manipulated by or presented to a user (including all functions, data input, indicators, and displays). Together with the commands, they provide access to the full functionality of the Target device or service in a non-modality specific form (e.g. it does not assume a particular type of user interface or display).

4. Naturally evolving smart home

The introduction of user interface sockets allows individuals to purchase home electronic devices individually over time (and from different manufacturers) and have them work with an intelligent controller they purchase (or have). For example, a person might have a natural language remote control or purchase a phone that had it built in. They could then pick up a digital video recorder and instantly (just by plugging it into the wall and cable) find that they could simply ask for programs by name when they wanted to watch them. Or they could describe which programs they would like to have recorded and automatically have them programmed to record (all this without having to mess with finding the programs, figuring out how to ask the product to record the programs, locate the recorded programs later, etc.). Later, when they replace their clock radio, they could pick one up that was also URC compatible and they would find that they could now set their alarm clock by simply telling their URC (e.g. phone) what time they would like to have the alarm set to. If they like to have their coffee pot turned on automatically in the morning, they would purchase a socket-enabled coffeepot, so that anyone in the family can easily set it, because they can simply ask their URC to do it for them. None of the appliances would need to be intelligent beyond exposing their function and taking commands via their user interface socket.

Because the full functionality of the products is exposed via the user interface socket, it is easy for intelligent controllers to be able to directly access the full functionality of the products, rather than having to navigate all of the menus. The user interface socket also makes the devices "more machine understandable" as well.

Additional layers of intelligence and sensors can then be installed in the home in this incremental way. Using the same architecture, each of these sensors or agents could provide any central controlling unit (or each other) the ability to sense or know things. In addition, Internet services may provide supplemental descriptions of Targets that new intelligent controllers can harness to interact with Targets in a way that was not possible when the Target was purchased. This knowledge can then be used in computer decision processes programmed in the factory or specified by the user. For example, the user may say "Whenever I come into the house and the house is dark, please turn on the kitchen overhead lights".

In addition, the ability to control the household products via a URC means that users can select and change their URC over time to meet their changing abilities. For example, if a person acquires a disability they could buy a new URC that accommodates their new limitations and provides them with the ability to continue to operate their household devices. And as people age, they can secure URCs that match their abilities and that will work with the appliances they already have.

Key to the success of this approach is the fact that is can evolve naturally over time without the user ever having to make a decision to "rewire their home or uninstall a smart house."

5. Past Work

The idea of "abstract UI descriptions" goes back to the concept of model-based User Interface Management Systems (UIMS) (Hayes, Szekely & Lerner, 1985), (Savidis & Stephanidis, 1995) which emphasize the separation between application logic, dialog control, and presentation component according to the Seeheim model (Pfaff, 1985). Myers (1990) uses the notion of interactors to provide a high-level interface for user input in a graphical user environment.

Former work of Trace in the area of universal remote controllers include input emulation efforts (Vanderheiden, 1981), and the development of the infrared-based Universal Remote Console Communication (URCC) Protocol (Vanderheiden, 1998).

The Total Access System uses a personal information appliance (called "accessor") to provide alternative ways of performing keyboard, mouse and/or monitor functions on a computer system (Scott & Gingras 2001).

UIML aims to provide a target platform independent XML-based user interface description language that can be automatically rendered on a diversity of user interface platforms (Abrams, Phanouriou, Batongbacal, Williams, & Shuster 1999). Cross-platform vocabularies for UIML have yet to be defined. However, work in this area seems to be in progress for UIML2.

The Pittsburgh Pebbles PDA Project (Nichols et al., 2002) introduces the "Personal Universal Controller" (PUC), which resembles the "Universal Remote Console" in many aspects, in particular the notion of state variables and commands describing an abstract user interface that can be downloaded from a device or service.

XWeb (Olsen et al., 2000) aims to support an abstract notion of interaction that is independent of a particular set of interactive techniques, by harnessing existing Web technologies.

XForms defines a set of abstract controls for UIs. However, XForms is designed to be embedded in a host container (e.g. an HTML document), which may or may not be accessible. Depending on the technology a particular XForms implementation is built upon, it may or may not support two-way synchronization of state information between server and client, which would be needed for implementing highly interactive UIs on remote consoles.

Sun’s Jini technology transmits UI code from a target device to a remote control (Beard & Korn 2001). However, this technology requires that the target provide different UI implementations for different classes of remote console devices.

6. The ANSI/INCITS URC Standards

The family of URC standards is being developed by the V2 Technical Committee of the InterNational Committee for Information Technology Standards (INCITS). INCITS is an accredited standardization organization of the American National Standards Institute (ANSI).

The title of the family of V2 URC standards is:

"Protocol to Facilitate Operation of Information and Electronic Products
through Remote and Alternative Interfaces and Intelligent Agents"

The first is the URC standard, describing the overall concepts and framework:

1. Universal Remote Console (ANSI/INCITS 389)
   • Specifies the URC framework and technical components of the standard. This should be the first document read by those unfamiliar with the URC standard and its scope. This document provides requirements for Targets, URCs and the network that connects them.

There are four additional standards which complement the URC standard, yielding a set of 5 standards.

2. User Interface Socket Description (ANSI/INCITS 390)
   • Specifies the User Interface Socket Description component of the URC framework. A User Interface Socket Description represents the functionality and state of a Target in a machine interpretable manner.
3. Presentation Template (ANSI/INCITS 391)
   • Specifies a language for providing a Presentation Template. A Presentation Template is a modality-independent scaffold for a user interface that provides the necessary structure and hints to build a browsable user interface.
4. Target Description (ANSI/INCITS 392)
   • Specifies a language for the description of Targets and their Sockets, as used within the URC framework for discovery purposes.
5. Resource Description (ANSI/INCITS 393)
   • Specifies resources and resource descriptions for user interface components that are used in building concrete user interfaces, including labels, help texts, access keys and keywords.

6.1 Major Components of the URC framework

Figure 2 illustrates the major components of the URC framework within the broader context of the URC application. It illustrates both elements for which there are formats specified by the URC framework, and elements that can use these or other standard or proprietary formats. This figure is provided as a conceptual map and is not intended to prescribe a particular architecture. Any alternative architecture may be adopted provided the required functionality is implemented and standard format documents are provided. Technical details for each component are provided in the respective standard documents as referenced in the pertaining sections of this document. See Table 1 and Table 2 for a summary of the purpose and reference to the technical descriptions of the major components.

As shown in Figure 2, the URC architecture can be thought of as comprising three major components and two networks.

1. The Universal Remote Console (URC)
2. The Target
3. Supplemental Resources

The two networks are

• Target-URC Network (TUN) - between the URC and the Target (required)
• Resource-URC Network (RUN) - between the URC and sources of additional resources

Figure 2: Overall structure and components of the URC framework (documents are represented as rectangles with rounded corners)

The URC standards specify communications between a Target that a user wishes to access and operate, and a Universal Remote Console (URC) that presents the user with a remote user interface through which they can discover, select, access and operate the Target. The URC is software that is typically hosted on the user's physical device, but a distributed approach is also possible.

Communications between the Target and URC take place over a network, the Target-URC Network. The standards do not specify the network protocol to be used, and a Target or URC may support any number of appropriate connection protocols. These protocols are used to provide discovery of Targets, and to establish and maintain control sessions between URCs and Targets. Targets and URCs access the Target-URC Network through Target-URC Network Links. Targets support discovery by providing essential information in a Target Description.

Each Target provides a User Interface Socket (or short "Socket"), or set of Sockets, through which a URC can access the Target's state and control it. For each Socket, at Target provides a User Interface Socket Description (or short "Socket Description") which describes the Socket in a machine interpretable manner. Additionally the Target provides Resources that pertain to the user interface of the Target, as viewed through that Socket. The Socket Description and Resources are used by the URC to find or generate an appropriate user interface, given the functionality of the Target, the nature of the URC device, and the user's interaction preferences.

Interaction between a Target and a URC consists of a discovery phase and an optional control phase. The discovery phase initializes the URC to locate and identify all available Targets and their Sockets. The control phase is the time period during which a Target and a URC initiate, maintain and terminate a control session. A control session is a connection between the URC and a Target's Socket for the purpose of the URC controlling a functional unit of the Target via the Socket. When multiple URCs are connected to the same Socket, each has an independent control session, although Target state values may be shared.

One type of Resource is a User Interface Implementation Description (UIID). A UIID is a description of a user interface for a particular Socket. This description may take an abstract or concrete form, and may be very general or specific to a given URC device class, user profile and task. The UIIDs provide a mechanism by which a manufacturer can provide tuned interfaces for their Target Sockets which are predefined and optimized to work on particular URCs, for example Pocket PCs or Palm-based PDAs. UIIDs can be provided by a Target or by external sources.

The URC standards define one, very general, form of UIID - the Presentation Template. A Presentation Template provides general hints as to how to build a usable user interface for a Socket, and does not assume any particular class of URC.

Another type of Resource is an Atomic Resource. An Atomic Resource is an object within a user interface that is used as an atomic entity in the construction of a concrete user interface. Atomic Resources include labels, help text, access keys and keywords. An Atomic Resource may be of any form and modality, including text, images, sounds, animations and video clips.

Socket Descriptions and UIIDs are documents that are independent of any natural language. In order to render their content to a user, Atomic Resources are provided (by the Target and external to it) in the form of Resource Sheets. A Resource Sheet is a file that contains descriptions of Atomic Resources (or "Atomic Resource Descriptions").

A URC may take advantage of Resources from the Target or from a Resource Service. These are referred to as Target Resources and Supplemental Resources respectively. Supplemental Resources can be used by a URC to replace, to supplement, or to help interpret or translate, any Resources provided by a Target.

In general, Resources for building user interfaces may be obtained from the Target, stored on the URC, or gathered from the Internet. A URC uses the Target-URC Network to retrieve Target Resources. It may use any form of networking or other mechanism to access Supplemental Resources via a Resource-URC Network. The URC standards do not specify the mechanism by which a URC accesses Resources external to the Target.

Table 1: Overview of components, their functions

Component	Purpose
Target-URC Network	network connecting the Target and the URC
Target-URC Network Link	link between a Target or a URC and the Target-URC Network
Target Description	information on a Target that is necessary for access to the Target and its Sockets
(User Interface) Socket	machine-operable access and control point for a Target
(User Interface) Socket Description	specification that describes the functions and properties of a Socket
Resource	object that is used as an entity or to support decision making in the construction of a concrete user interface
Target Resource	Resource provided by a Target
Supplemental Resource	Resource made available externally to a Target
Resource-URC Network	network connecting the URC to sources of Supplemental Resources

Table 2: Overview of Resource types defined by this standard and ANSI INCITS 393-2005

Resource Type	Purpose
User Interface Implementation Description (UIID)	description for implementing a user interface to a Target, based on the Target's Socket
Presentation Template	modality-independent presentation information for a Socket Description; subtype of UIID
Atomic Resource	atomic entity in the construction of a concrete user interface (e.g. label, help text, access key, keyword)
Resource Sheet	file containing descriptions of Atomic Resources (Atomic Resource Descriptions)

7. Status of URC Standards

The set of URC standards have been completed, have gone through first public review, been revised and at submission time are out for ballot. It is expected that the standard should be completed and released under ANSI by the time of the conference.

Test implementations of the standard have been created in Java, using UPnP over 802.11b (WiFi) and Ethernet. Simulations involve TVs, VCRs, clock radios, alarm clocks, lamps, blinds and coffee makers. A text input interface has been developed (that also works with speech input) and a natural language interface is under development.

Combined, they hold the opportunity for control of complex devices (and everyday devices with complex interfaces) by people using only their natural language. By allowing things to be controlled with a natural language controller products can be made much easier to use by everyone. Alternate controls of all types as well as agent control of the elements in our environment can be thus achieved without having to completely replace the devices all at once or as a system. Individual products can be purchased over time – and – if they include user interface sockets in their design, they will become part of the smart house as they are brought in. Such a gradual, as-you-need-it or can-afford-it approach is much more likely to be implemented in most households than an approach that requires purchase of sets of appliances and custom wiring or networks. And with implementation in mainstream high volume products the cost for the networking and socket capabilities is likely to be less than 5 -10% of the overall cost of the product.

8. Acknowledgements

This work was funded by NIDRR, US Dept of Education and the National Science Foundation under Grants H133E040013, H133E030012. The opinions herein are those of the authors and not necessarily those of the funding agencies.

9. References

Abrams, M., Phanouriou, C., Batongbacal, A.L., Williams, S., & Shuster, J.E. (1999). UIML: An Appliance-Independent XML User Interface Language. WWW8 conference, May 1999, Toronto, Canada.

Beard, M., & Korn, P. (2001). What I Need is What I Get: Downloadable User Interfaces via Jini and Java. CHI 2001 Extended Abstracts, pp. 15-16.

Hayes, P. J.; Szekely, P. A.; Lerner, R. A. (1985). Design alternatives for user interface management systems based on experience with COUSIN. Proceedings of the CHI '85 conference on Human factors in computing systems, April 1985.

Myers, B.A. (1990). A new model for handling input. ACM Transactions on Information Systems (TOIS), July 1990, Volume 8, Issue 3.

Nichols, J.; Myers, B.A.; Higgins, M.; Hughes, J.; Harris, T.K.; Rosenfeld, R.; Pignol, M. (2002). Generating Remote Control Interfaces for Complex Appliances. CHI Letters: ACM Symposium on User Interface Software and Technology, UIST'02, 27-30 Oct. 2002, Paris, France. http://www-2.cs.cmu.edu/~pebbles/papers/PebblesPUCuist.pdf.

Olsen Jr., D.R., et al. (2000). Cross-modal Interaction Using XWeb. Proceedings UIST’00, 2000, San Diego, CA, pp. 191-200. Retrieved 21 June, 2002, from the WWW: http://icie.cs.byu.edu/ICE/LabPapers/CrossModalXwebInteraction.pdf.

Pfaff, G.E., (1985). User Interface Management Systems. Springer-Verlag, Berlin, 1985.

Savidis, A.; Stephanidis, C. (1995). Developing dual user interfaces for integrating blind and sighted users: the HOMER UIMS. Conference proceedings on Human factors in computing systems, May 1995.

Scott, N.G., & Gingras, I. (2001). The Total Access System. CHI 2001 Extended Abstracts, pp. 13-14.

Vanderheiden, G. (1981). Practical Applications of Microcomputers to Aid the Handicapped. Computer, IEEE Computer Society, January.

Vanderheiden, G. C. (1998). Universal remote console communication protocol (URCC). Proceedings of the 1998 TIDE Conference, Helsinki, Finland: Stakes.