M4: Evaluation Methods

The aim of an interdisciplinary technology design is to create proposals of design as early as possible to avoid technical constraints. If this happens, the evaluation of the proposals of design however end up in  a paradoxical situation: it should evaluate a technical application, which yet does not exist. In principle, experiences become necessary with the designed technology to identify and assess its effects. However this is not possible because the technology lacks its technical and organizational embedding and its application possibilities. Furthermore, experiences with technology must be made – e.g. with technical failures - that should be avoided just by these designs. The way out of this dilemma is the simulation study whereby this dilemma is bypassed through a realistic but protected environment. Later, real users test the demonstrators under simulated but realistic conditions by using specific, design- and regulations-related test cases in order to derive design proposals.

Simulation studies are an instrument for gaining experiences in (almost) real situations with technology-based social networking. The studies can be used from all disciplines that are working at the design of technical-social networking. Each simulation study is polyvalent. It can impart insights for everybody who observes and interviews the study and its participants and discusses with the latter. Beside the lawyers and computer scientists, the existing simulation studies are also used by ergonomists and psychologists for their scientific objectives.

In M4, the method of simulation study for the design proposals of VENUS has to be further developed. Therefore, the following topics are to be examined:

  • the possibilities of use and limits of simulation studies to gain the necessary experience in the single disciplines,
  • the requirements for the case design
  • in order to gain meaningful results
  • the requirements in proto-typical partial realizations, e.g. of user interfaces in order to analyze them in simulation,
  • the requirements in the demonstrators (functions, surface, networking, environment),
  • the need of expansion or modification in order to optimize the evaluation methods,
  • the need for additional evaluation-methods and -steps,
  • the relation of simulation studies to other evaluation methods of usability engineering like pilot and field test, business game, practical test, participative technology design, computer simulation or thought experiment and their comparative advantages and disadvantages.

Referring to one of the demonstrators, a simulation study is planned, executed and evaluated in Research Field 3.