Class Trial
BuildStep --+
|
LoggingBuildStep --+
|
ShellCommand --+
|
Trial
- Known Subclasses:
-
TwistedTrial
I run a unit test suite using 'trial', a unittest-like testing
framework that comes with Twisted. Trial is used to implement Twisted's
own unit tests, and is the unittest-framework of choice for many projects
that use Twisted internally.
Projects that use trial typically have all their test cases in a
'test' subdirectory of their top-level library directory. I.e. for my
package 'petmail', the tests are in 'petmail/test/test_*.py'. More
complicated packages (like Twisted itself) may have multiple test
directories, like 'twisted/test/test_*.py' for the core functionality and
'twisted/mail/test/test_*.py' for the email-specific tests.
To run trial tests, you run the 'trial' executable and tell it where
the test cases are located. The most common way of doing this is with a
module name. For petmail, I would run 'trial petmail.test' and it would
locate all the test_*.py files under petmail/test/, running every test
case it could find in them. Unlike the unittest.py that comes with
Python, you do not run the test_foo.py as a script; you always let trial
do the importing and running. The 'tests' parameter controls which tests
trial will run: it can be a string or a list of strings.
You can also use a higher-level module name and pass the --recursive
flag to trial: this will search recursively within the named module to
find all test cases. For large multiple-test-directory projects like
Twisted, this means you can avoid specifying all the test directories
explicitly. Something like 'trial --recursive twisted' will pick up
everything.
To find these test cases, you must set a PYTHONPATH that allows
something like 'import petmail.test' to work. For packages that don't use
a separate top-level 'lib' directory, PYTHONPATH=. will work, and will
use the test cases (and the code they are testing) in-place.
PYTHONPATH=build/lib or PYTHONPATH=build/lib.$ARCH are also useful when
you do a'setup.py build' step first. The 'testpath' attribute of this
class controls what PYTHONPATH= is set to.
Trial has the ability (through the --testmodule flag) to run only the
set of test cases named by special 'test-case-name' tags in source files.
We can get the list of changed source files from our parent Build and
provide them to trial, thus running the minimal set of test cases needed
to cover the Changes. This is useful for quick builds, especially in
trees with a lot of test cases. The 'testChanges' parameter controls this
feature: if set, it will override 'tests'.
The trial executable itself is typically just 'trial' (which is
usually found on your $PATH as /usr/bin/trial), but it can be overridden
with the 'trial' parameter. This is useful for Twisted's own unittests,
which want to use the copy of bin/trial that comes with the sources.
(when bin/trial discovers that it is living in a subdirectory named
'Twisted', it assumes it is being run from the source tree and adds that
parent directory to PYTHONPATH. Therefore the canonical way to run
Twisted's own unittest suite is './bin/trial twisted.test' rather than
'PYTHONPATH=. /usr/bin/trial twisted.test', especially handy when
/usr/bin/trial has not yet been installed).
To influence the version of python being used for the tests, or to add
flags to the command, set the 'python' parameter. This can be a string
(like 'python2.2') or a list (like ['python2.3', '-Wall']).
Trial creates and switches into a directory named _trial_temp/ before
running the tests, and sends the twisted log (which includes all
exceptions) to a file named test.log . This file will be pulled up to the
master where it can be seen as part of the status output.
There are some class attributes which may be usefully overridden by
subclasses. 'trialMode' and 'trialArgs' can influence the trial command
line.
| Method Summary |
| |
__init__(self,
reactor,
python,
trial,
testpath,
tests,
testChanges,
recurse,
randomly,
trialMode,
trialArgs,
**kwargs) |
| |
addTestResult(self,
testname,
results,
text,
tlog)
|
| |
commandComplete(self,
cmd)
This is a general-purpose hook method for subclasses. |
| |
createSummary(self,
loog)
|
| |
evaluateCommand(self,
cmd)
Decide whether the command was SUCCESS, WARNINGS, or FAILURE. |
| |
getText(self,
cmd,
results)
|
| |
getText2(self,
cmd,
results)
We have decided to add a short note about ourselves to the overall
build description, probably because something went wrong. |
| |
rtext(self,
fmt)
|
| |
setupEnvironment(self,
cmd)
|
| |
start(self)
Begin the step. |
| Inherited from ShellCommand |
| |
checkForOldSlaveAndLogfiles(self) |
| |
describe(self,
done)
Return a list of short strings to describe this step, for the status
display. |
| |
setCommand(self,
command) |
| Inherited from LoggingBuildStep |
| |
checkDisconnect(self,
f) |
| |
getColor(self,
cmd,
results) |
| |
interrupt(self,
reason)
Halt the command, either because the user has decided to cancel the
build ('reason' is a string), or because the slave has disconnected
('reason' is a ConnectionLost Failure). |
| |
maybeGetText2(self,
cmd,
results) |
| |
setStatus(self,
cmd,
results) |
| |
setupLogfiles(self,
cmd,
logfiles)
Set up any additional logfiles= logs. |
| |
startCommand(self,
cmd,
errorMessages) |
| Inherited from BuildStep |
| |
acquireLocks(self,
res) |
| |
addCompleteLog(self,
name,
text) |
| |
addHTMLLog(self,
name,
html) |
| |
addLog(self,
name) |
| |
addLogObserver(self,
logname,
observer) |
| |
addURL(self,
name,
url)
Add a BuildStep URL to this step. |
| |
failed(self,
why) |
| |
finished(self,
results) |
| |
getProperty(self,
propname) |
| |
getSlaveName(self) |
| |
releaseLocks(self) |
| |
runCommand(self,
c) |
| |
setProgress(self,
metric,
value)
BuildSteps can call self.setProgress() to announce progress along some
metric. |
| |
setProperty(self,
propname,
value) |
| |
setStepStatus(self,
step_status) |
| |
setupProgress(self) |
| |
slaveVersion(self,
command,
oldversion)
Return the version number of the given slave command. |
| |
slaveVersionIsOlderThan(self,
command,
minversion) |
| |
startStep(self,
remote)
Begin the step. |
__init__(self,
reactor=(),
python=None,
trial=None,
testpath=(),
tests=None,
testChanges=None,
recurse=None,
randomly=None,
trialMode=None,
trialArgs=None,
**kwargs)
(Constructor)
-
- Parameters:
reactor -
which reactor to use, like 'gtk' or 'java'. If not provided,
the Twisted's usual platform-dependent default is used.
(type=string)
python -
which python executable to use. Will form the start of the
argv array that will launch trial. If you use this, you should
set 'trial' to an explicit path (like /usr/bin/trial or
./bin/trial). Defaults to None, which leaves it out entirely
(running 'trial args' instead of 'python ./bin/trial args').
Likely values are 'python', ['python2.2'], ['python', '-Wall'],
etc.
(type=string (without spaces) or list)
trial -
which 'trial' executable to run. Defaults to 'trial', which
will cause $PATH to be searched and probably find /usr/bin/trial
. If you set 'python', this should be set to an explicit path
(because 'python2.3 trial' will not work).
(type=string)
testpath -
use in PYTHONPATH when running the tests. If None, do not set
PYTHONPATH. Setting this to '.' will cause the source files to be
used in-place.
(type=string)
tests -
a list of test modules to run, like
['twisted.test.test_defer', 'twisted.test.test_process']. If this
is a string, it will be converted into a one-item list.
(type=list of strings)
testChanges -
if True, ignore the 'tests' parameter and instead ask the
Build for all the files that make up the Changes going into this
build. Pass these filenames to trial and ask it to look for
test-case-name tags, running just the tests necessary to cover
the changes.
(type=boolean)
recurse -
If True, pass the --recurse option to trial, allowing test
cases to be found in deeper subdirectories of the modules listed
in 'tests'. This does not appear to be necessary when using
testChanges.
(type=boolean)
randomly -
if True, add the --random=0 argument, which instructs trial to
run the unit tests in a random order each time. This occasionally
catches problems that might be masked when one module always runs
before another (like failing to make registerAdapter calls before
lookups are done).
(type=boolean)
trialMode -
a list of arguments to pass to trial, specifically to set the
reporting mode. This defaults to ['-to'] which means 'verbose
colorless output' to the trial that comes with Twisted-2.0.x and
at least -2.1.0 . Newer versions of Twisted may come with a trial
that prefers ['--reporter=bwverbose'].
(type=list of strings)
trialArgs -
a list of arguments to pass to trial, available to turn on any
extra flags you like. Defaults to [].
(type=list of strings)
kwargs -
parameters. The following parameters are inherited from ShellCommand and may be useful to
set: workdir, haltOnFailure, flunkOnWarnings, flunkOnFailure,
warnOnWarnings, warnOnFailure, want_stdout, want_stderr,
timeout.
(type=dict)
- Overrides:
buildbot.steps.shell.ShellCommand.__init__
|
evaluateCommand(self,
cmd)
Decide whether the command was SUCCESS, WARNINGS, or FAILURE.
Override this to, say, declare WARNINGS if there is any stderr
activity, or to say that rc!=0 is not actually an error.
-
- Overrides:
buildbot.process.buildstep.LoggingBuildStep.evaluateCommand (inherited documentation)
|
getText2(self,
cmd,
results)
We have decided to add a short note about ourselves to the overall
build description, probably because something went wrong. Return a
short list of short strings. If your subclass counts test failures or
warnings of some sort, this is a good place to announce the count.
-
- Overrides:
buildbot.process.buildstep.LoggingBuildStep.getText2 (inherited documentation)
|
start(self)
Begin the step. Override this method and add code to do local
processing, fire off remote commands, etc.
To spawn a command in the buildslave, create a RemoteCommand
instance and run it with self.runCommand:
c = RemoteCommandFoo(args)
d = self.runCommand(c)
d.addCallback(self.fooDone).addErrback(self.failed)
As the step runs, it should send status information to the
BuildStepStatus:
self.step_status.setColor('red')
self.step_status.setText(['compile', 'failed'])
self.step_status.setText2(['4', 'warnings'])
To have some code parse stdio (or other log stream) in realtime, add
a LogObserver subclass. This observer can use self.step.setProgress()
to provide better progress notification to the step.:
self.addLogObserver('stdio', MyLogObserver())
To add a LogFile, use self.addLog. Make sure it gets closed when it
finishes. When giving a Logfile to a RemoteShellCommand, just ask it to
close the log when the command completes:
log = self.addLog('output')
cmd = RemoteShellCommand(args)
cmd.useLog(log, closeWhenFinished=True)
You can also create complete Logfiles with generated text in a
single step:
self.addCompleteLog('warnings', text)
When the step is done, it should call self.finished(result).
'result' will be provided to the buildbot.process.base.Build, and should
be one of the constants defined above: SUCCESS, WARNINGS, FAILURE, or
SKIPPED.
If the step encounters an exception, it should call
self.failed(why). 'why' should be a Failure object. This automatically
fails the whole build with an exception. It is a good idea to add
self.failed as an errback to any Deferreds you might obtain.
If the step decides it does not need to be run, start() can return
the constant SKIPPED. This fires the callback immediately: it is not
necessary to call .finished yourself. This can also indicate to the
status-reporting mechanism that this step should not be displayed.
-
- Overrides:
buildbot.steps.shell.ShellCommand.start (inherited documentation)
|
| Instance Variable Details |
logfiles
a dict mapping log NAMEs to workdir-relative FILENAMEs of their
corresponding logfiles. The contents of the file named FILENAME will be
put into a LogFile named NAME, ina something approximating real-time.
(note that logfiles= is actually handled by our parent class
LoggingBuildStep)
-
- Value:
{'test.log': '_trial_temp/test.log'}
|
|
flunkOnFailure
-
- Type:
-
bool
- Value:
|
progressMetrics
-
- Type:
-
tuple
- Value:
('output', 'tests', 'test.log')
|
|
python
-
- Type:
-
NoneType
- Value:
|
randomly
-
- Type:
-
bool
- Value:
|
reactor
-
- Type:
-
NoneType
- Value:
|
recurse
-
- Type:
-
bool
- Value:
|
testChanges
-
- Type:
-
bool
- Value:
|
testpath
-
- Type:
-
tuple
- Value:
|
tests
-
- Type:
-
NoneType
- Value:
|
trialArgs
-
- Type:
-
list
- Value:
|
trialMode
-
- Type:
-
list
- Value:
|