Continuous Integration testing (with TravisCI)
- juzna.cz
- Member | 248
Intro
We've got a decent number of test cases, which helps us to avoid many errors. They cover most essential functionality of the framework, and they try to cover as many use cases as possible. When they fail, it indicates an essential part of the framework is not working properly.
Before any change gets into master branch or into a release, someone should run the test cases and make sure nothing got broken. Ideally every developer should run them before sending a pull request. Otherwise, a pull request which breaks something is usually useless and only adds boring work to the one who is trying to merge it.
I dare to estimate that number of contributors who run all tests cases before a PR, who check they didn't break anything, and who do it all for various versions of PHP, is zero. Not approximately, but precisely zero! On the other hand, the number of those who merge PRs is approximately zero ;).
One may say that he sometimes wants to merge on iPhone and that it is not possible to check the tests there, but that's not correct. It's possible to merge safely even on iPhone…
Let's recap.
Problems
- difficult to tell if a change broke something
- breaking change can get to master or a release
- difficult to check if a pull request passes
- running test is time consuming, and boring
- different versions of PHP
- existing tests are failing, but nobody cares
That being said, test cases are half useless when we cannot rely on them.
Goals
- see whether master branch is passing, or which test cases don't; always, immediately, with history
- get notification when a change causes a test broken
- in a fork, let the developer know about test failure before he creates pull request
- easy way to check if a pull request is passing; i.e. immediately
- all that for different versions of PHP
Solution
Continuous Integration with Travis CI.
It means automatically executing all test cases. After every push. Also available for forks. And also for pull requests. And for various versions of PHP.
Implementation
Travis CI has a tutorial on how to enable it for a project. The only
requirement is to add file .travis.yml to project root directory
and define PHP related configuration.
This file is probably also the only disadvantage – it's one more file in project root directory, which is not inherently related to the framework itself. It may not seem like a problem, but one file here and another there and we'll end up with lots of mess in root like symfony.
Note that .travis.yml is helpful only in git/GitHub repository
and has no meaning in distribution packages. Therefore it should be removed by
build-tools.
Limitation
We need to have all existing tests passing to gain all benefits. Or at least they have to be skipped, so that final result is passing. Only then, when something gets broken, we'll get failed result which will indicate a problem.
Current status
There are many test cases failing, some due to bugs in PHP itself. These would need to be marked as skipped for that particular version of PHP.
The rest of test cases needs to be fixed to pass.
Summary
There are pros and cons for this solution, but I believe it's worth having it included. If that happens, we'll be able to spot problems early and save our time.
Pull Request is available, but still needs improvements.
So, do we want TravisCI to run tests for Nette?
Last edited by juzna.cz (2012-08-28 17:20)
- juzna.cz
- Member | 248
Known failures
on PHP 5.4
-> Nette\Utils\Strings::fixEncoding()
file: /www/nette/tests/Nette/Utils/Strings.fixEncoding().phpt
-> Nette\Http\Request invalid encoding.
file: /www/nette/tests/Nette/Http/Request.invalidEncoding.phpt
-> Nette\Forms invalid input.
file: /www/nette/tests/Nette/Forms/Forms.invalidInput.002.phptRelated to iconv bug
-> Nette\Diagnostics\Debugger E_ERROR in HTML.
file: /www/nette/tests/Nette/Diagnostics/Debugger.E_ERROR.html.phpt
Failed asserting that "
Fatal error: Call to undefined function missing_funcion() in /www/nette/tests/Nette/Diagnostics/Deb" ... matches expected "
Fatal error: Call to undefined function missing_funcion() in %a% on line %d%
<!DOCTYPE html><!-- "'" ... in file /www/nette/tests/Nette/Diagnostics/Debugger.E_ERROR.html.phpt on line 26Dunno
Last edited by juzna.cz (2012-09-01 09:46)
- juzna.cz
- Member | 248
One more thing which should be solved: loading proper php.ini file.
At the moment, we need to pass options via -d parameter which is
not very nice.
Better solution would be giving tests/php.ini to all tests by
default and having configuration there.
Edit: solved
Last edited by juzna.cz (2012-09-01 15:53)
- juzna.cz
- Member | 248
There is one test with Memcache cache storage failing on Travis, but I've got no idea why. On my local machine it works just fine.
-> Nette\Caching\Storages\MemcachedStorage sliding expiration test.
file: /home/travis/builds/juzna/nette/tests/Nette/Caching/Memcached.sliding.phptEdit: Solved by compiling latest memcached server on
travis
Edit 2: It still fails sometimes, probably a race
condition :(
Last edited by juzna.cz (2012-09-04 10:14)
- juzna.cz
- Member | 248
TODO: configure Travis to run database tests also against PostgreSQL (and/or other databases).
Anyone knows how to do that?
We can either run all tests against several databases, which would however
waste resources. We should run only database tests. For that reason, we
could extends TestRunner to run Database directory several
times, each time with different ENV. Then, connect.inc.php would
connect to that particular db specified in ENV (or skip if that db is
not available).
But this solution looks like a hack for TestRunner. Can you find something better?
- HosipLan
- Moderator | 4668
Something like this should work (not tested):
env:
- DB=mysql,TESTS_DIR=tests/
- DB=postgres,TESTS_DIR=tests/Nette/Database
script: "./tests/run-tests.sh -s $TESTS_DIR -c tests/php.ini-unix"Inspired here: http://about.travis-ci.org/…nguages/php/
Detailed database setup: http://about.travis-ci.org/…abase-setup/
- Milo
- Nette Core | 1283
It may works… but whole test suite will be environment dependent. I have done some work on multi-database tests but I suggest a little bit different solution.
Make default file databases.ini
[mysql-1]
dsn=mysql:.....
user=
password=
;[postgresql-1]
;dsn=pgsql:....
;user=
;password=Add new command line parameter:
-db-conf path/to/databases.iniMark all .phpt with @database annotation. All these tests add
multiple times (e.g. two times when two databases defined) to test queue and use
something like…
$db = TestHelpers::getDatabase('path/to/dump-%driver.sql'); // not sure how to import correct dump- juzna.cz
- Member | 248
@HosipLan: that looks very easy, but I dont like it
much for several reasons:
1/ it's not right to run all tests against mysql, as most of
them don't care about db
2/ it would give us two separate stats on results, though it would be better so
see all failing tests in one output.
Anyway, this causes a problem for us ;)
@Milo: that looks nice, I imagined it very similarly. Would you like to rebase it to latest master so that we can try it out easily? Thx
- Vojtěch Dobeš
- Gold Partner | 1316
Good news: just today travisbot stopped to spam comments. Github added first-class support for Travis.
- juzna.cz
- Member | 248
I just came back from vacation and I noticed that multiple databases tests are already pushed to master.
Also readme has been changed to markdown and has build status icon in it. That's great.
Thanks guys for all the effort and work done here!
Last edited by juzna.cz (2012-09-11 14:18)