squaremobius dot net

HCI as language

TonyG and I were talking about human computer interaction as a conversation. If you consider a user interface to be a language, with a vocabulary and a grammar, then user actions are sentences. More specifically, taking the example of a direct manipulation style interface: icons or selections of icons are noun (phrases), gestures are verb (phrases). Dragging a file icon to the trash can is saying FILE GOES TRASHCAN.

A few things are apparent when you look at things this way: most direct-manipulation(-of-files) systems have effectively one verb (gesture): that is, GOES, usually indicated with a click and drag gesture. Most sentences are just noun-verb-noun. Also, the computer is mostly a passive partner in the conversation, aside from 'minimal encouragers' — brief sound effects and animations as feedback that the gesture was recognised.

Using more than one verb

I said before that most systems only have one verb GOES; you might contest that there is also INVOKE (intransitive), so let me qualify my position by adding that has meaning within the system. I'm saying that starting an application or opening a file is stepping outside of the language (more on this later, or in another post). In file manipulation, it may be that one verb is sufficient — the rest is interpetation based on the nouns in the sentence and the context. There seems to be a trade-off between a rich set of interpretations and a rich set of verbs, which is heavily weighted in favour of interpretations. This is a little weird to me.

The contrasting situation is to have a rich set of verbs (hence, gestures) and consistent meanings for the verbs. For example, to delete something you could scribble over it; delete should after all be intransitive, surely. If you want to modify something, click in it (this is half-arsedly approximated by the 'file-type associations' pattern in desktop systems). If you want to move it, select (circle?) it and drag it.

More complex sentences

Sentences in contempory graphical user interfaces are very simple; usually NOUN VERB NOUN. There's no recursion, only statefulness. What if you could form complex sentences? Sure, with only one verb there are not that many interesting complex sentences you can form. With many verbs you can make interesting noun phrases: Text similar to this text should be copied here.

Having a conversation

So far the computer is listening to what we say, and interpreting it as commands to update its state. The computer may as well not be there, it's just the user (as TonyG puts it) shuffling paperweights.

If we imagine the computer on the other side of the screen, and the screen itself as the speaking-space between the user and the computer, then we can have a 'proper' conversation. For a start the computer should ask when it doesn't understand or needs more information; and it should ask in the same language as we are using. What does this mean, concretely? Good question.