WS 2008/2009 - Testen von Software - Übung

Latest update: 18.12.2008

Allgemeines zur Übung

Zum erfolgreichen Absolvieren der Übung "Testen von Software" sind folgende drei Kriterien zu erfüllen:
  1. kontinuierliche und aktive Teilnahme an den Übungsveranstaltungen
  2. Präsentation eines Testwerkzeuges
  3. Schreiben einer Kurzzusammenfassung über das referierte Testwerkzeug

Für die Zusammenfassung sollen folgende Templates genutzt werden:
Zusammenfassung Testwerkzeug (MS Word Version)
Zusammenfassung Testwerkzeug (OpenOffice 2.0 Version)

Termine

Neuer Termin: Der erste Vortrag wird vom 05.01. auf den 09.01. in den 2. Block (direkt vor der Vorlesung) verschoben. Der Raum wird noch bekannt gegeben.
Datum Toolbeschreibung Student Vortrag Zusammenfassung
09.01.2009 JUnit + CppUnit + Jester + Zanebug Erik Bergmann
12.01.2009 Valgrind + OProfile + Memory Checker Karsten
19.01.2009 STAF + TestMaster Stephan
26.01.2009 Salome TMF + Bugzilla + JUnit Thimo Hoerster

Tools

static program checkers

lint / splint

www.splint.org
Splint is a tool for statically checking C programs for security vulnerabilities and coding mistakes. With minimal effort, Splint can be used as a better lint. If additional effort is invested adding annotations to programs, Splint can perform stronger checking than can be done by any standard lint.

checkstyle

http://checkstyle.sourceforge.net
Checkstyle can check many aspects of your java source code. Historically it's main functionality has been to check code layout issues, but since the internal architecture was changed in version 3, more and more checks for other purposes have been added. Now Checkstyle provides checks that find class design problems, duplicate code, or bug patterns like double checked locking.

jlint

http://artho.com/jlint/
A Java Programm checker. Jlint will check your Java code and find bugs, inconsistencies and synchronization problems by doing data flow analysis and building the lock graph.

findbugs

http://findbugs.sourceforge.net/
FindBugs is a program to find bugs in Java programs. It looks for instances of "bug patterns" --- code instances that are likely to be errors.

PMD

http://pmd.sourceforge.net/
is a Java source code analyzer. It finds unused variables, empty catch blocks, unnecessary object creation, and so forth. PMD scans Java source code and looks for potential problems. It has plugins for JEdit, JBuilder, Netbeans/Forte/Sun ONE, IntelliJ IDEA, Maven, Ant, Eclipse, Gel, and Emacs.

FxCop

http://www.gotdotnet.com/team/fxcop/
FxCop is a code analysis tool that checks .NET managed code assemblies for conformance to the Microsoft .NET Framework Design Guidelines. It uses reflection, MSIL parsing, and callgraph analysis to inspect assemblies for more than 200 defects in the following areas:: Library design, Localization, Naming conventions, Performance, Security. FxCop includes both GUI and command line versions of the tool.

Regressiontest

Expect / Dejagnu

http://expect.nist.gov/
programmed dialogue with interactive programs
http://www.gnu.org/software/dejagnu/
DejaGnu is a framework for testing other programs. Its purpose is to provide a single front end for all tests. Think of it as a custom library of Tcl procedures crafted to support writing a test harness. A Test Harness is the testing infrastructure that is created to support a specific program or tool.

Junit

http://www.junit.org
JUnit is a regression testing framework written by Erich Gamma and Kent Beck. It is used by the developer who implements unit tests in Java.

CppUnit

http://sourceforge.net/projects/cppunit
CppUnit is the C++ port of the famous JUnit framework for unit testing. Test output is in XML or text format for automatic testing and GUI based for supervised tests.

Jameleon

http://jameleon.sourceforge.net/
Jameleon is a data-driven automated testing tool that is easily extensible via plug-ins. Features of applications are automated in Java and tied together independently in XML, creating self-documenting automated test cases.

csUnit

http://www.csunit.org/
csUnit is a unit testing tool for the Microsoft .NET Framework. You can use it with all .NET languages including C#, Visual Basic .NET, J#, and managed C++.

MbUnit

http://www.mertner.com/confluence/display/MbUnit/Home
MbUnit is an extensible .Net test framework. As in NUnit, tests are created at runtime using Reflection and custom attributes. MbUnit differentiates itself from NUnit in it's extensibility model. It contains a number of tests that go beyond the simple unit testing, such as combinatorial testing, data oriented testing, etc...

Zanebug

http://www.adapdev.com/zanebug/downloads.aspx
Zanebug was developed to provide a unified platform for unit and integration testing. The current release addresses component-level unit testing, with a fair amount of development time focused on performance statistics and a flexible architecture. The long-term view for Zanebug is a tool that provides integrated component, form and web-based testing both locally and remotely with all the necessary performance information necessary for enterprise development.

Self-Testable class

http://www.stclass.org/
STclass allows the definition and evaluation of functional contracts, as defined by B. Meyer in the Design by Contract approach: OCL like invariants on classes, pre- and post-conditions on methods; contracts defined in javadoc comments (can be extracted by javadoc); contracts inheritance from parent class and interfaces following the Meyer's rules;

Mutationstest

Jester

http://jester.sourceforge.net/
Jester - the JUnit test tester. Jester finds code that is not covered by tests. Jester makes some change to your code, runs your tests, and if the tests pass Jester displays a message saying what it changed. Jester includes a script for generating web pages that show the changes made that did not cause the tests to fail.

GUI Test

JFCUnit

http://sourceforge.net/projects/jfcunit/
http://jfcunit.sourceforge.net
jfcUnit enables developers to write test cases for Java Swing based applications. It provides support for: 1.Obtaining handles on Windows/Dialogs opened by the Java code. 2.Locating components within a component hierarchy that occur within the containers found above. 3.Raising events on the found components, e.g. clicking a button, typing text in a TextComponent. 4.Handling testing of components in a thread safe manner. Since version 2.0, jfcUnit provides XML Recording and playback. This allows users to quickly and automatically generate/edit� scripts to drive the testing. The XML API's are open and allow for developers to define there own XML tag handlers. See the example JFCUnit via XML.

Abbot Java GUI Test Framework

http://abbot.sourceforge.net/
The Abbot framework provides automated event generation and validation of Java GUI components, improving upon the very rudimentary functions provided by the java.awt.Robot class (A Better 'Bot). The framework may be invoked directly from Java code or accessed without programming through the use of scripts. It is suitable for use both by developers for unit tests and QA for functional testing.

Jacareto

http://www.ph-ludwigsburg.de/mathematik/personal/spannagel/jacareto/
Java Capture & Replay Toolkit - Jacareto is a capture&replay framework for Java applications and applets. It allows you to easily create your own capture&replay tools.

X11::GUITest / Win32::GuiTest

http://sourceforge.net/projects/x11guitest/
http://triumvir.org/articles/guitest-wd-mag/
http://www.cpan.org/authors/id/E/ER/ERNGUI/
X11::GUITest is a Perl package intended to facilitate the testing of GUI applications by means of user emulation. It can be used to test and interact with GUI applications which have been built (Xlib, GTK, etc.) upon the X toolkit. (GUItest of Ped?)

Pounder (no longer being actively developed)

http://pounder.sourceforge.net
Pounder is a utility for automating Java GUI tests. It allows developers to dynamically load GUI- Components , record scripts, and then use those scripts in a test harness. Pounder differentiates itself from other utilities by allowing you to examine the results of a test run in source, while maintaining a separate GUI script that can be re-recorded if necessary.

Jemmy

http://jemmy.netbeans.org/
Jemmy is a JavaTM library that is used to create automated tests for Java GUI applications. It contains methods to reproduce all user actions which can be performed on Swing/AWT components (i.e. button pushing, text typing, tree node expanding, etc). JemmyTest is a program written in Java which uses the Jemmy API to test applications.

Marathon

http://marathonman.sourceforge.net/
Marathon is a testing framework for GUI applications developed using Java/Swing. Marathon composes of recorder, runner and editor. The testscripts are composed of python code. Marathon captures the semantic actions on components instead of the mouse & keyboard events. This results in scripts that are simple and at a higher granular level. Marathon suites well for applications that mostly depend on forms-paradigm.

Badboy

http://www.badboy.com.au/
Badboy is a tool designed to aid in testing and development of dynamic web applications. It makes web testing and development easier with dozens of features including a simple yet comprehensive capture/replay interface, powerful load testing support, detailed reports, graphs and much more!

Sahi

http://sahi.co.in/w/
Sahi is an automation and testing tool for web applications, with the facility to record and playback scripts. Developed in java and javascript, this tool uses simple javascript to execute events on the browser.

Dogtail

http://people.redhat.com/zcerza/dogtail/
Dogtail is a GUI test tool and automation framework written in Python. It uses Accessibility (a11y) technologies to communicate with Linux desktop applications.

Python Win32 Automation

http://sourceforge.net/projects/pywinauto/
pywinauto is a python package that allows you to automate the windows GUI. Very easy to get started, and quite powerful.

jDiffChaser

http://jdiffchaser.sourceforge.net/
jDiffChaser is a Java Open Source Software that automates visual GUI comparisons between two versions of a Swing application.

AutoIt

http://www.autoitscript.com/
AutoIt v3 is a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting. It uses a combination of simulated keystrokes, mouse movement and window/control manipulation in order to automate tasks in a way not possible or reliable with other languages (e.g. VBScript and SendKeys). AutoIt is also very small, self-contained and will run on all versions of Windows out-of-the-box with no annoying "runtimes" required!

Autohotkey

http://www.autohotkey.com/
AutoHotkey is a free, open-source utility for Windows. With it, you can for example automate almost anything by sending keystrokes and mouse clicks. You can write a mouse or keyboard macro by hand or use the macro recorder.

Profiling

OProfile

http://oprofile.sourceforge.net/
OProfile is a system-wide profiler for Linux systems, capable of profiling all running code at low overhead. OProfile is released under the GNU GPL. It consists of a kernel driver and a daemon for collecting sample data, and several post-profiling tools for turning data into information. OProfile leverages the hardware performance counters of the CPU to enable profiling of a wide variety of interesting statistics, which can also be used for basic time-spent profiling. All code is profiled: hardware and software interrupt handlers, kernel modules, the kernel, shared libraries, and applications.

gprof Gnu Profiler

Material: Buch Programmierwerkzeuge S. 223 - 228 http://www.gnu.org/software/binutils/manual/gprof-2.9.1/gprof.html

Profiling Java Programs

Material: IX- Artikel 04/2004 S 42. http://gd.tuwien.ac.at/languages/java/GoToJava2/html/k100298.html
Tools: Jrat: jrat.sourceforge.net
Hpjmeter: www.hp.com/products1/unix/java/hpjmeter
JunitPerf: http://clarkware.com/software/JUnitPerf.html

Code Coverage

gcov coverage testing tool

http://gcc.gnu.org/onlinedocs/gcc-3.0/gcc_8.html
gcov is a test coverage program. Use it in concert with GNU CC to analyze your programs to help create more efficient, faster running code. You can use gcov as a profiling tool to help discover where your optimization efforts will best affect your code. Material: Buch: Zeller/Krinke, Programmierwekzeuge S.229 ? 235

cobertura

http://cobertura.sourceforge.net/
Cobertura is a free Java tool that calculates the percentage of code accessed by tests. It can be used to identify which parts of your Java program are lacking test coverage. It is based on jcoverage.

EMMA: a free Java code coverage tool

http://emma.sourceforge.net
EMMA is an open-source toolkit for measuring and reporting Java code coverage.

Tools for determining Security defects

Material: Artikel Linuxjournal http://www.linuxjournal.com//article.php?sid=5673
Tools: Flawfinder: http://www.dwheeler.com/flawfinder
Rats http://www.securesw.com/download_rats.htm
PSCAN http://www.striker.ottawa.on.ca/~aland/pscan/ (historically)
bfbtester http://bfbtester.sourceforge.net/

distributed Testing

QAT (Quality Assurance Tests )

http://qat.sourceforge.net/
QAT was developed to ease the issues encountered by having to perform Quality Assurance tests across a variety of hardware and software combinations. The QAT tool can be divided into two main sections, the Agent, responsible for actually running each test or group of tests, and the Harness, which is responsible for test selection, management, result and agent co-ordination.

QMTest

http://www.codesourcery.com/qmtest/
CodeSourcery's QMTest provides a cost-effective general purpose testing solution that allows an organization to implement a robust, easy-to-use testing program tailored to its needs. QMTest's extensible architecture allows it to handle a wide range of application domains: everything from compilers to graphical user interfaces to web-based applications.

STAF

http://staf.sourceforge.net/index.php
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework designed around the idea of reusable components, called services. STAF is a framework designed to improve the level of reuse and automation in test cases and test environments. The goal of STAF is to provide a complete end-to-end automation solution for testers.

Memory Checks

Memory checker / leaktracer

http://freshmeat.net/projects/leaktracer/
LeakTracer is a small tool for checking a C++ program for memory leaks. To use LeakTracer, run your program using the provided LeakCheck script. It uses the LD_PRELOAD feature to "overlay" some functions on top of your functions (no recompile needed). LeakTracer uses gdb to print out the exact line where the memory was allocated and not freed - this of course means you have to free all dynamically allocated data. LeakTracer also overrides the global operator new and operator delete.

Valgrind

http://valgrind.kde.org/
is a GPL'd system for debugging and profiling x86-Linux programs. With the tools that come with Valgrind, you can automatically detect many memory management and threading bugs, avoiding hours of frustrating bug-hunting, making your programs more stable. You can also perform detailed profiling to help speed up your programs.

Systematic Software Testing

VeriSoft

http://cm.bell-labs.com/who/god/verisoft/
is a tool for systematically exploring the state spaces of systems composed of several concurrent processes executing arbitrary code written in any language. The state space of a system is a directed graph that represents the combined behavior of all the components of the system. Paths in this graph correspond to sequences of operations (scenarios) that can be observed during executions of the system. VeriSoft systematically explores the state space of a system by controlling and observing the execution of all the components, and by reinitializing their executions. It searches for coordination problems (deadlocks, divergences, etc.) between concurrent components, and for violations of user-specified assertions. VeriSoft can always guarantee a complete coverage of the state space up to some depth; hence, all possible executions of the system up to that depth are guaranteed to be covered.

org.tigris.mbt

http://mbt.tigris.org/
org.tigris.mbt is an implementation of Model-based testing built in Java. It allows you to generate test sequences from a finite-state machine (graph). The test sequences can be created statically, or run dynamically.

Java PathFinder

http://javapathfinder.sourceforge.net/
Java PathFinder is a system to verify executable Java bytecode programs. In its basic form, it is a Java Virtual Machine (JVM) that is used as an explicit state software model checker.

Spec Explorer

http://research.microsoft.com/projects/specexplorer/
Spec Explorer is a software-development tool for advanced model-based specification and conformance testing. Spec Explorer can help software-development teams detect errors in the design, the specification, and the implementation of their systems. The tool is intended to be used by software testers, designers, and implementers.

Testmanagement und Bug Tracking

Salome TMF

http://fitnesse.org/
Salome-TMF is an independent Test Management Tool, which helps you to manage your entire testing process ? by creating tests, executing manual or automatic tests, tracking results, managing requirements and defects and producing HTML documentation. Salome-TMF is compatible with Junit, Abbot and Beanshell to define your automatic tests, and with Bugzilla and Mantis to manage your defects. Salome-TMF can also be extended by plug-in according to your requirements.

TestMaster

http://testmaster.sourceforge.net/
A testcase management, logging, reporting and test automation tool, similar to the commercial product Test Director. Features: Progress stats, reports, test case import from CSV,doc,web or SQL, STAF plugin.

Fitnesse

https://wiki.objectweb.org/salome-tmf/
FitNesse is a collaborative testing and documentation tool. It provides a very simple way for teams to collaboratively create documents, specify tests, and run those tests.

Bugzilla

www.bugzilla.org
Bugzilla is a bug- or issue-tracking system. Bug-tracking systems allow individual or groups of developers effectively to keep track of outstanding problems with their product. Bugzilla was originally written by Terry Weissman in a programming language called TCL, to replace a rudimentary bug-tracking database used internally by Netscape Communications. Terry later ported Bugzilla to Perl from TCL, and in Perl it remains to this day. Most commercial defect-tracking software vendors at the time charged enormous licensing fees, and Bugzilla quickly became a favorite of the open-source crowd (with its genesis in the open-source browser project, Mozilla). It is now the de-facto standard defect-tracking system against which all others are measured.

Flyspray

http://flyspray.rocks.cc/
Flyspray is an uncomplicated, web-based bug tracking system for assisting with software development.

General Testframeworks

Eclipse Test and Performance Tools Platform

http://www.eclipse.org/tptp/
The Eclipse Test and Performance Tools Platform (TPTP) Project is an open source Top Level Project of the Eclipse Foundation. TPTP is divided into four projects: TPTP Platform, Monitoring Tools, Testing Tools, Tracing and Profiling Tools.