Package buildbot :: Package steps :: Module python_twisted :: Class Trial
[frames | no frames]

Class Trial

   BuildStep --+        
               |        
LoggingBuildStep --+    
                   |    
        ShellCommand --+
                       |
                      Trial

Known Subclasses:
TwistedTrial

I run a unit test suite using 'trial', a unittest-like testing framework that comes with Twisted. Trial is used to implement Twisted's own unit tests, and is the unittest-framework of choice for many projects that use Twisted internally.

Projects that use trial typically have all their test cases in a 'test' subdirectory of their top-level library directory. I.e. for my package 'petmail', the tests are in 'petmail/test/test_*.py'. More complicated packages (like Twisted itself) may have multiple test directories, like 'twisted/test/test_*.py' for the core functionality and 'twisted/mail/test/test_*.py' for the email-specific tests.

To run trial tests, you run the 'trial' executable and tell it where the test cases are located. The most common way of doing this is with a module name. For petmail, I would run 'trial petmail.test' and it would locate all the test_*.py files under petmail/test/, running every test case it could find in them. Unlike the unittest.py that comes with Python, you do not run the test_foo.py as a script; you always let trial do the importing and running. The 'tests' parameter controls which tests trial will run: it can be a string or a list of strings.

You can also use a higher-level module name and pass the --recursive flag to trial: this will search recursively within the named module to find all test cases. For large multiple-test-directory projects like Twisted, this means you can avoid specifying all the test directories explicitly. Something like 'trial --recursive twisted' will pick up everything.

To find these test cases, you must set a PYTHONPATH that allows something like 'import petmail.test' to work. For packages that don't use a separate top-level 'lib' directory, PYTHONPATH=. will work, and will use the test cases (and the code they are testing) in-place. PYTHONPATH=build/lib or PYTHONPATH=build/lib.$ARCH are also useful when you do a'setup.py build' step first. The 'testpath' attribute of this class controls what PYTHONPATH= is set to.

Trial has the ability (through the --testmodule flag) to run only the set of test cases named by special 'test-case-name' tags in source files. We can get the list of changed source files from our parent Build and provide them to trial, thus running the minimal set of test cases needed to cover the Changes. This is useful for quick builds, especially in trees with a lot of test cases. The 'testChanges' parameter controls this feature: if set, it will override 'tests'.

The trial executable itself is typically just 'trial' (which is usually found on your $PATH as /usr/bin/trial), but it can be overridden with the 'trial' parameter. This is useful for Twisted's own unittests, which want to use the copy of bin/trial that comes with the sources. (when bin/trial discovers that it is living in a subdirectory named 'Twisted', it assumes it is being run from the source tree and adds that parent directory to PYTHONPATH. Therefore the canonical way to run Twisted's own unittest suite is './bin/trial twisted.test' rather than 'PYTHONPATH=. /usr/bin/trial twisted.test', especially handy when /usr/bin/trial has not yet been installed).

To influence the version of python being used for the tests, or to add flags to the command, set the 'python' parameter. This can be a string (like 'python2.2') or a list (like ['python2.3', '-Wall']).

Trial creates and switches into a directory named _trial_temp/ before running the tests, and sends the twisted log (which includes all exceptions) to a file named test.log . This file will be pulled up to the master where it can be seen as part of the status output.

There are some class attributes which may be usefully overridden by subclasses. 'trialMode' and 'trialArgs' can influence the trial command line.
Method Summary
  __init__(self, reactor, python, trial, testpath, tests, testChanges, recurse, randomly, trialMode, trialArgs, **kwargs)
  addTestResult(self, testname, results, text, tlog)
  commandComplete(self, cmd)
This is a general-purpose hook method for subclasses.
  createSummary(self, loog)
  evaluateCommand(self, cmd)
Decide whether the command was SUCCESS, WARNINGS, or FAILURE.
  getText(self, cmd, results)
  getText2(self, cmd, results)
We have decided to add a short note about ourselves to the overall build description, probably because something went wrong.
  rtext(self, fmt)
  setupEnvironment(self, cmd)
  start(self)
Begin the step.
    Inherited from ShellCommand
  checkForOldSlaveAndLogfiles(self)
  describe(self, done)
Return a list of short strings to describe this step, for the status display.
  setCommand(self, command)
    Inherited from LoggingBuildStep
  checkDisconnect(self, f)
  getColor(self, cmd, results)
  interrupt(self, reason)
Halt the command, either because the user has decided to cancel the build ('reason' is a string), or because the slave has disconnected ('reason' is a ConnectionLost Failure).
  maybeGetText2(self, cmd, results)
  setStatus(self, cmd, results)
  setupLogfiles(self, cmd, logfiles)
Set up any additional logfiles= logs.
  startCommand(self, cmd, errorMessages)
    Inherited from BuildStep
  acquireLocks(self, res)
  addCompleteLog(self, name, text)
  addHTMLLog(self, name, html)
  addLog(self, name)
  addLogObserver(self, logname, observer)
  addURL(self, name, url)
Add a BuildStep URL to this step.
  failed(self, why)
  finished(self, results)
  getProperty(self, propname)
  getSlaveName(self)
  releaseLocks(self)
  runCommand(self, c)
  setProgress(self, metric, value)
BuildSteps can call self.setProgress() to announce progress along some metric.
  setProperty(self, propname, value)
  setStepStatus(self, step_status)
  setupProgress(self)
  slaveVersion(self, command, oldversion)
Return the version number of the given slave command.
  slaveVersionIsOlderThan(self, command, minversion)
  startStep(self, remote)
Begin the step.

Instance Variable Summary
  logfiles: a dict mapping log NAMEs to workdir-relative FILENAMEs of their corresponding logfiles.
    Inherited from ShellCommand
NoneType command: a list of argv strings (or WithProperties instances).
    Inherited from BuildStep
buildbot.process.base.Build build: the parent Build which is executing this step
buildbot.status.progress.StepProgress progress: tracks ETA for the step
buildbot.status.builder.BuildStepStatus step_status: collects output status

Class Variable Summary
bool flunkOnFailure = True
str name = 'trial'
tuple progressMetrics = ('output', 'tests', 'test.log')
NoneType python = None                                                                  
bool randomly = False
NoneType reactor = None                                                                  
bool recurse = False
bool testChanges = False
tuple testpath = ()
NoneType tests = None                                                                  
str trial = 'trial'
list trialArgs = []
list trialMode = ['--reporter=bwverbose']
    Inherited from ShellCommand
NoneType description = None                                                                  
NoneType descriptionDone = None                                                                  
    Inherited from LoggingBuildStep
list parms = ['build', 'name', 'locks', 'haltOnFailure', 'flu...
    Inherited from BuildStep
bool flunkOnWarnings = False
bool haltOnFailure = False
list locks = []
bool useProgress = True
bool warnOnFailure = False
bool warnOnWarnings = False

Method Details

__init__(self, reactor=(), python=None, trial=None, testpath=(), tests=None, testChanges=None, recurse=None, randomly=None, trialMode=None, trialArgs=None, **kwargs)
(Constructor)

Parameters:
reactor - which reactor to use, like 'gtk' or 'java'. If not provided, the Twisted's usual platform-dependent default is used.
           (type=string)
python - which python executable to use. Will form the start of the argv array that will launch trial. If you use this, you should set 'trial' to an explicit path (like /usr/bin/trial or ./bin/trial). Defaults to None, which leaves it out entirely (running 'trial args' instead of 'python ./bin/trial args'). Likely values are 'python', ['python2.2'], ['python', '-Wall'], etc.
           (type=string (without spaces) or list)
trial - which 'trial' executable to run. Defaults to 'trial', which will cause $PATH to be searched and probably find /usr/bin/trial . If you set 'python', this should be set to an explicit path (because 'python2.3 trial' will not work).
           (type=string)
testpath - use in PYTHONPATH when running the tests. If None, do not set PYTHONPATH. Setting this to '.' will cause the source files to be used in-place.
           (type=string)
tests - a list of test modules to run, like ['twisted.test.test_defer', 'twisted.test.test_process']. If this is a string, it will be converted into a one-item list.
           (type=list of strings)
testChanges - if True, ignore the 'tests' parameter and instead ask the Build for all the files that make up the Changes going into this build. Pass these filenames to trial and ask it to look for test-case-name tags, running just the tests necessary to cover the changes.
           (type=boolean)
recurse - If True, pass the --recurse option to trial, allowing test cases to be found in deeper subdirectories of the modules listed in 'tests'. This does not appear to be necessary when using testChanges.
           (type=boolean)
randomly - if True, add the --random=0 argument, which instructs trial to run the unit tests in a random order each time. This occasionally catches problems that might be masked when one module always runs before another (like failing to make registerAdapter calls before lookups are done).
           (type=boolean)
trialMode - a list of arguments to pass to trial, specifically to set the reporting mode. This defaults to ['-to'] which means 'verbose colorless output' to the trial that comes with Twisted-2.0.x and at least -2.1.0 . Newer versions of Twisted may come with a trial that prefers ['--reporter=bwverbose'].
           (type=list of strings)
trialArgs - a list of arguments to pass to trial, available to turn on any extra flags you like. Defaults to [].
           (type=list of strings)
kwargs - parameters. The following parameters are inherited from ShellCommand and may be useful to set: workdir, haltOnFailure, flunkOnWarnings, flunkOnFailure, warnOnWarnings, warnOnFailure, want_stdout, want_stderr, timeout.
           (type=dict)
Overrides:
buildbot.steps.shell.ShellCommand.__init__

commandComplete(self, cmd)

This is a general-purpose hook method for subclasses. It will be called after the remote command has finished, but before any of the other hook functions are called.
Overrides:
buildbot.process.buildstep.LoggingBuildStep.commandComplete (inherited documentation)

evaluateCommand(self, cmd)

Decide whether the command was SUCCESS, WARNINGS, or FAILURE. Override this to, say, declare WARNINGS if there is any stderr activity, or to say that rc!=0 is not actually an error.
Overrides:
buildbot.process.buildstep.LoggingBuildStep.evaluateCommand (inherited documentation)

getText2(self, cmd, results)

We have decided to add a short note about ourselves to the overall build description, probably because something went wrong. Return a short list of short strings. If your subclass counts test failures or warnings of some sort, this is a good place to announce the count.
Overrides:
buildbot.process.buildstep.LoggingBuildStep.getText2 (inherited documentation)

start(self)

Begin the step. Override this method and add code to do local processing, fire off remote commands, etc.

To spawn a command in the buildslave, create a RemoteCommand instance and run it with self.runCommand:
 c = RemoteCommandFoo(args)
 d = self.runCommand(c)
 d.addCallback(self.fooDone).addErrback(self.failed)
As the step runs, it should send status information to the BuildStepStatus:
 self.step_status.setColor('red')
 self.step_status.setText(['compile', 'failed'])
 self.step_status.setText2(['4', 'warnings'])
To have some code parse stdio (or other log stream) in realtime, add a LogObserver subclass. This observer can use self.step.setProgress() to provide better progress notification to the step.:
 self.addLogObserver('stdio', MyLogObserver())
To add a LogFile, use self.addLog. Make sure it gets closed when it finishes. When giving a Logfile to a RemoteShellCommand, just ask it to close the log when the command completes:
 log = self.addLog('output')
 cmd = RemoteShellCommand(args)
 cmd.useLog(log, closeWhenFinished=True)
You can also create complete Logfiles with generated text in a single step:
 self.addCompleteLog('warnings', text)

When the step is done, it should call self.finished(result). 'result' will be provided to the buildbot.process.base.Build, and should be one of the constants defined above: SUCCESS, WARNINGS, FAILURE, or SKIPPED.

If the step encounters an exception, it should call self.failed(why). 'why' should be a Failure object. This automatically fails the whole build with an exception. It is a good idea to add self.failed as an errback to any Deferreds you might obtain.

If the step decides it does not need to be run, start() can return the constant SKIPPED. This fires the callback immediately: it is not necessary to call .finished yourself. This can also indicate to the status-reporting mechanism that this step should not be displayed.
Overrides:
buildbot.steps.shell.ShellCommand.start (inherited documentation)

Instance Variable Details

logfiles

a dict mapping log NAMEs to workdir-relative FILENAMEs of their corresponding logfiles. The contents of the file named FILENAME will be put into a LogFile named NAME, ina something approximating real-time. (note that logfiles= is actually handled by our parent class LoggingBuildStep)
Value:
{'test.log': '_trial_temp/test.log'}                                   

Class Variable Details

flunkOnFailure

Type:
bool
Value:
True                                                                   

name

Type:
str
Value:
'trial'                                                                

progressMetrics

Type:
tuple
Value:
('output', 'tests', 'test.log')                                        

python

Type:
NoneType
Value:
None                                                                  

randomly

Type:
bool
Value:
False                                                                  

reactor

Type:
NoneType
Value:
None                                                                  

recurse

Type:
bool
Value:
False                                                                  

testChanges

Type:
bool
Value:
False                                                                  

testpath

Type:
tuple
Value:
()                                                                     

tests

Type:
NoneType
Value:
None                                                                  

trial

Type:
str
Value:
'trial'                                                                

trialArgs

Type:
list
Value:
[]                                                                     

trialMode

Type:
list
Value:
['--reporter=bwverbose']                                               

Generated by Epydoc 2.1 on Sun Dec 10 22:04:45 2006 http://epydoc.sf.net