Continuous Integration testing (with TravisCI)

1. 2 years ago

juzna.cz
Member
Registered: 2010-12-03
Location: Brno, CZ
Posts: 243

Continuous Integration testing (with TravisCI)

Intro

We've got a decent number of test cases, which helps us to avoid many errors. They cover most essential functionality of the framework, and they try to cover as many use cases as possible. When they fail, it indicates an essential part of the framework is not working properly.

Before any change gets into master branch or into a release, someone should run the test cases and make sure nothing got broken. Ideally every developer should run them before sending a pull request. Otherwise, a pull request which breaks something is usually useless and only adds boring work to the one who is trying to merge it.

I dare to estimate that number of contributors who run all tests cases before a PR, who check they didn't break anything, and who do it all for various versions of PHP, is zero. Not approximately, but precisely zero! On the other hand, the number of those who merge PRs is approximately zero ;).

One may say that he sometimes wants to merge on iPhone and that it is not possible to check the tests there, but that's not correct. It's possible to merge safely even on iPhone…

Let's recap.

Problems

  • difficult to tell if a change broke something
  • breaking change can get to master or a release
  • difficult to check if a pull request passes
  • running test is time consuming, and boring
  • different versions of PHP
  • existing tests are failing, but nobody cares

That being said, test cases are half useless when we cannot rely on them.

Goals

  • see whether master branch is passing, or which test cases don't; always, immediately, with history
  • get notification when a change causes a test broken
  • in a fork, let the developer know about test failure before he creates pull request
  • easy way to check if a pull request is passing; i.e. immediately
  • all that for different versions of PHP

Solution

Continuous Integration with Travis CI.

It means automatically executing all test cases. After every push. Also available for forks. And also for pull requests. And for various versions of PHP.

Implementation

Travis CI has a tutorial on how to enable it for a project. The only requirement is to add file .travis.yml to project root directory and define PHP related configuration.

This file is probably also the only disadvantage – it's one more file in project root directory, which is not inherently related to the framework itself. It may not seem like a problem, but one file here and another there and we'll end up with lots of mess in root like symfony.

Note that .travis.yml is helpful only in git/GitHub repository and has no meaning in distribution packages. Therefore it should be removed by build-tools.

Limitation

We need to have all existing tests passing to gain all benefits. Or at least they have to be skipped, so that final result is passing. Only then, when something gets broken, we'll get failed result which will indicate a problem.

Current status

There are many test cases failing, some due to bugs in PHP itself. These would need to be marked as skipped for that particular version of PHP.

The rest of test cases needs to be fixed to pass.

Summary

There are pros and cons for this solution, but I believe it's worth having it included. If that happens, we'll be able to spot problems early and save our time.

Pull Request is available, but still needs improvements.

So, do we want TravisCI to run tests for Nette?

Last edited by juzna.cz (2012-08-28 17:20)


Develop with pleasure

 

2. 2 years ago

Ondřej Mirtes
Moderator
Registered: 2009-01-08
Location: Praha
Posts: 1537

Re: Continuous Integration testing (with TravisCI)

+1

 

3. 2 years ago

redhead
Member
Registered: 2009-05-02
Posts: 1318

Re: Continuous Integration testing (with TravisCI)

Yes!

 

4. 2 years ago

hrach
Moderator
Registered: 2007-03-11
Location: Brno
Posts: 1303

Re: Continuous Integration testing (with TravisCI)

Yes, and for the record, I always runs nette\database tests before pull-requesting :)

 

5. 2 years ago

HosipLan
Moderator
Registered: 2009-06-01
Location: Brno
Posts: 4035

Re: Continuous Integration testing (with TravisCI)

Yes, we do want Travis-CI for automated testing. And I think, the advantages overcomes the presence of one yaml file.

 

6. 2 years ago

juzna.cz
Member
Registered: 2010-12-03
Location: Brno, CZ
Posts: 243

Re: Continuous Integration testing (with TravisCI)

hrach wrote:

I always runs nette\database tests before pull-requesting :)

Both PHP 5.3 and 5.4? ;)


Develop with pleasure

 

7. 2 years ago

juzna.cz
Member
Registered: 2010-12-03
Location: Brno, CZ
Posts: 243

Re: Continuous Integration testing (with TravisCI)

Known failures

on PHP 5.4

-> Nette\Utils\Strings::fixEncoding()
   file: /www/nette/tests/Nette/Utils/Strings.fixEncoding().phpt

-> Nette\Http\Request invalid encoding.
   file: /www/nette/tests/Nette/Http/Request.invalidEncoding.phpt

-> Nette\Forms invalid input.
   file: /www/nette/tests/Nette/Forms/Forms.invalidInput.002.phpt

Related to iconv bug


-> Nette\Diagnostics\Debugger E_ERROR in HTML.
   file: /www/nette/tests/Nette/Diagnostics/Debugger.E_ERROR.html.phpt

Failed asserting that "
Fatal error: Call to undefined function missing_funcion() in /www/nette/tests/Nette/Diagnostics/Deb" ... matches expected "
Fatal error: Call to undefined function missing_funcion() in %a% on line %d%
<!DOCTYPE html><!-- "'" ... in file /www/nette/tests/Nette/Diagnostics/Debugger.E_ERROR.html.phpt on line 26

Dunno

Last edited by juzna.cz (2012-09-01 09:46)


Develop with pleasure

 

8. 2 years ago

HosipLan
Moderator
Registered: 2009-06-01
Location: Brno
Posts: 4035

Re: Continuous Integration testing (with TravisCI)

$ php-cgi -v
PHP 5.4.6 (cgi-fcgi) (built: Aug 28 2012 21:47:50)

Same as @juzna.cz, but the last one works for me, and the DiscoveredReflection is fixed already too.

FAILURES! (506 tests, 3 failures, 33 skipped)

 

9. 2 years ago

juzna.cz
Member
Registered: 2010-12-03
Location: Brno, CZ
Posts: 243

Re: Continuous Integration testing (with TravisCI)

On 5.3.16 is still one test failing due to another bug in PHP :(

Edit: Test updated not to fail, and php bug already fixed (hopefully in 5.3.17)

Last edited by juzna.cz (2012-09-01 15:53)


Develop with pleasure

 

10. 2 years ago

juzna.cz
Member
Registered: 2010-12-03
Location: Brno, CZ
Posts: 243

Re: Continuous Integration testing (with TravisCI)

One more thing which should be solved: loading proper php.ini file.

At the moment, we need to pass options via -d parameter which is not very nice.

Better solution would be giving tests/php.ini to all tests by default and having configuration there.

Edit: solved

Last edited by juzna.cz (2012-09-01 15:53)


Develop with pleasure

 

11. 2 years ago

juzna.cz
Member
Registered: 2010-12-03
Location: Brno, CZ
Posts: 243

Re: Continuous Integration testing (with TravisCI)

Another todo: setup memcache on Travis, so that it can be tested as well

Edit: Solved

Last edited by juzna.cz (2012-09-01 15:50)


Develop with pleasure

 

12. 2 years ago

juzna.cz
Member
Registered: 2010-12-03
Location: Brno, CZ
Posts: 243

Re: Continuous Integration testing (with TravisCI)

There is one test with Memcache cache storage failing on Travis, but I've got no idea why. On my local machine it works just fine.

-> Nette\Caching\Storages\MemcachedStorage sliding expiration test.
   file: /home/travis/builds/juzna/nette/tests/Nette/Caching/Memcached.sliding.phpt

Edit: Solved by compiling latest memcached server on travis
Edit 2: It still fails sometimes, probably a race condition :(

Last edited by juzna.cz (2012-09-04 10:14)


Develop with pleasure

 

13. 2 years ago

juzna.cz
Member
Registered: 2010-12-03
Location: Brno, CZ
Posts: 243

Re: Continuous Integration testing (with TravisCI)

TODO: configure Travis to run database tests also against PostgreSQL (and/or other databases).

Anyone knows how to do that?

We can either run all tests against several databases, which would however waste resources. We should run only database tests. For that reason, we could extends TestRunner to run Database directory several times, each time with different ENV. Then, connect.inc.php would connect to that particular db specified in ENV (or skip if that db is not available).

But this solution looks like a hack for TestRunner. Can you find something better?


Develop with pleasure

 

14. 2 years ago

HosipLan
Moderator
Registered: 2009-06-01
Location: Brno
Posts: 4035

Re: Continuous Integration testing (with TravisCI)

Something like this should work (not tested):

env:
  - DB=mysql,TESTS_DIR=tests/
  - DB=postgres,TESTS_DIR=tests/Nette/Database

script: "./tests/run-tests.sh -s $TESTS_DIR -c tests/php.ini-unix"

Inspired here: http://about.travis-ci.org/…nguages/php/

Detailed database setup: http://about.travis-ci.org/…abase-setup/

 

15. 2 years ago

Milo
Moderator
Registered: 2010-05-04
Posts: 736

Re: Continuous Integration testing (with TravisCI)

It may works… but whole test suite will be environment dependent. I have done some work on multi-database tests but I suggest a little bit different solution.

Make default file databases.ini

[mysql-1]
dsn=mysql:.....
user=
password=

;[postgresql-1]
;dsn=pgsql:....
;user=
;password=

Add new command line parameter:

-db-conf path/to/databases.ini

Mark all .phpt with @database annotation. All these tests add multiple times (e.g. two times when two databases defined) to test queue and use something like…

$db = TestHelpers::getDatabase('path/to/dump-%driver.sql'); // not sure how to import correct dump

 

16. 2 years ago

juzna.cz
Member
Registered: 2010-12-03
Location: Brno, CZ
Posts: 243

Re: Continuous Integration testing (with TravisCI)

@HosipLan: that looks very easy, but I dont like it much for several reasons:
1/ it's not right to run all tests against mysql, as most of them don't care about db
2/ it would give us two separate stats on results, though it would be better so see all failing tests in one output.

Anyway, this causes a problem for us ;)

@Milo: that looks nice, I imagined it very similarly. Would you like to rebase it to latest master so that we can try it out easily? Thx


Develop with pleasure

 

17. 2 years ago

Milo
Moderator
Registered: 2010-05-04
Posts: 736

Re: Continuous Integration testing (with TravisCI)

I'll prepare a new branch by cherry-pick in a few days. I have some work done locally.

 

18. 2 years ago

vojtech.dobes
Member
Registered: 2010-09-27
Location: Brno
Posts: 1187

Re: Continuous Integration testing (with TravisCI)

Good news: just today travisbot stopped to spam comments. Github added first-class support for Travis.


Dělat věci dobře. Github @vojtechdobes

 

19. 2 years ago

Milo
Moderator
Registered: 2010-05-04
Posts: 736

Re: Continuous Integration testing (with TravisCI)

I created new branch. Last commits allows run multiple database tests based on databases.ini. All current tests passing but I'm not sure if it is a proper way.

Last edited by Milo (2012-09-05 19:53)

 

20. 2 years ago

juzna.cz
Member
Registered: 2010-12-03
Location: Brno, CZ
Posts: 243

Re: Continuous Integration testing (with TravisCI)

I just came back from vacation and I noticed that multiple databases tests are already pushed to master.

Also readme has been changed to markdown and has build status icon in it. That's great.

Thanks guys for all the effort and work done here!

Last edited by juzna.cz (2012-09-11 14:18)


Develop with pleasure

 

21. about a year ago

Milo
Moderator
Registered: 2010-05-04
Posts: 736

Re: Continuous Integration testing (with TravisCI)

What about runs tests for PHP 5.2 and 5.5? It's a little bit tricky.

 

Board footer

Switch to desktop