We've got a decent number of test cases, which helps us to avoid many errors. They cover most essential functionality of the framework, and they try to cover as many use cases as possible. When they fail, it indicates an essential part of the framework is not working properly.
Before any change gets into master branch or into a release, someone should run the test cases and make sure nothing got broken. Ideally every developer should run them before sending a pull request. Otherwise, a pull request which breaks something is usually useless and only adds boring work to the one who is trying to merge it.
I dare to estimate that number of contributors who run all tests cases before a PR, who check they didn't break anything, and who do it all for various versions of PHP, is zero. Not approximately, but precisely zero! On the other hand, the number of those who merge PRs is approximately zero ;).
One may say that he sometimes wants to merge on iPhone and that it is not possible to check the tests there, but that's not correct. It's possible to merge safely even on iPhone…
- difficult to tell if a change broke something
- breaking change can get to master or a release
- difficult to check if a pull request passes
- running test is time consuming, and boring
- different versions of PHP
- existing tests are failing, but nobody cares
That being said, test cases are half useless when we cannot rely on them.
- see whether master branch is passing, or which test cases don't; always, immediately, with history
- get notification when a change causes a test broken
- in a fork, let the developer know about test failure before he creates pull request
- easy way to check if a pull request is passing; i.e. immediately
- all that for different versions of PHP
Continuous Integration with Travis CI.
It means automatically executing all test cases. After every push. Also available for forks. And also for pull requests. And for various versions of PHP.
This file is probably also the only disadvantage – it's one more file in project root directory, which is not inherently related to the framework itself. It may not seem like a problem, but one file here and another there and we'll end up with lots of mess in root like symfony.
.travis.yml is helpful only in git/GitHub repository
and has no meaning in distribution packages. Therefore it should be removed by
We need to have all existing tests passing to gain all benefits. Or at least they have to be skipped, so that final result is passing. Only then, when something gets broken, we'll get failed result which will indicate a problem.
The rest of test cases needs to be fixed to pass.
There are pros and cons for this solution, but I believe it's worth having it included. If that happens, we'll be able to spot problems early and save our time.
Pull Request is available, but still needs improvements.
So, do we want TravisCI to run tests for Nette?
Last edited by juzna.cz (2012-08-28 17:20)
on PHP 5.4
-> Nette\Utils\Strings::fixEncoding() file: /www/nette/tests/Nette/Utils/Strings.fixEncoding().phpt -> Nette\Http\Request invalid encoding. file: /www/nette/tests/Nette/Http/Request.invalidEncoding.phpt -> Nette\Forms invalid input. file: /www/nette/tests/Nette/Forms/Forms.invalidInput.002.phpt
Related to iconv bug
-> Nette\Diagnostics\Debugger E_ERROR in HTML. file: /www/nette/tests/Nette/Diagnostics/Debugger.E_ERROR.html.phpt Failed asserting that " Fatal error: Call to undefined function missing_funcion() in /www/nette/tests/Nette/Diagnostics/Deb" ... matches expected " Fatal error: Call to undefined function missing_funcion() in %a% on line %d% <!DOCTYPE html><!-- "'" ... in file /www/nette/tests/Nette/Diagnostics/Debugger.E_ERROR.html.phpt on line 26
Last edited by juzna.cz (2012-09-01 09:46)
One more thing which should be solved: loading proper php.ini file.
At the moment, we need to pass options via
-d parameter which is
not very nice.
Better solution would be giving
tests/php.ini to all tests by
default and having configuration there.
Last edited by juzna.cz (2012-09-01 15:53)
There is one test with Memcache cache storage failing on Travis, but I've got no idea why. On my local machine it works just fine.
-> Nette\Caching\Storages\MemcachedStorage sliding expiration test. file: /home/travis/builds/juzna/nette/tests/Nette/Caching/Memcached.sliding.phpt
Edit: Solved by compiling latest memcached server on
Edit 2: It still fails sometimes, probably a race condition :(
Last edited by juzna.cz (2012-09-04 10:14)
TODO: configure Travis to run database tests also against PostgreSQL (and/or other databases).
Anyone knows how to do that?
We can either run all tests against several databases, which would however
waste resources. We should run only database tests. For that reason, we
could extends TestRunner to run
Database directory several
times, each time with different ENV. Then,
connect to that particular db specified in ENV (or skip if that db is
But this solution looks like a hack for TestRunner. Can you find something better?
Something like this should work (not tested):
env: - DB=mysql,TESTS_DIR=tests/ - DB=postgres,TESTS_DIR=tests/Nette/Database script: "./tests/run-tests.sh -s $TESTS_DIR -c tests/php.ini-unix"
Inspired here: http://about.travis-ci.org/…nguages/php/
Detailed database setup: http://about.travis-ci.org/…abase-setup/
It may works… but whole test suite will be environment dependent. I have done some work on multi-database tests but I suggest a little bit different solution.
Make default file databases.ini
[mysql-1] dsn=mysql:..... user= password= ;[postgresql-1] ;dsn=pgsql:.... ;user= ;password=
Add new command line parameter:
Mark all .phpt with
@database annotation. All these tests add
multiple times (e.g. two times when two databases defined) to test queue and use
$db = TestHelpers::getDatabase('path/to/dump-%driver.sql'); // not sure how to import correct dump
@HosipLan: that looks very easy, but I dont like it
much for several reasons:
1/ it's not right to run all tests against mysql, as most of them don't care about db
2/ it would give us two separate stats on results, though it would be better so see all failing tests in one output.
Anyway, this causes a problem for us ;)
@Milo: that looks nice, I imagined it very similarly. Would you like to rebase it to latest master so that we can try it out easily? Thx
I just came back from vacation and I noticed that multiple databases tests are already pushed to master.
Also readme has been changed to markdown and has build status icon in it. That's great.
Thanks guys for all the effort and work done here!
Last edited by juzna.cz (2012-09-11 14:18)