#74 Run a "behave" test with the CI pipeline (Proof of Concept)
Opened by esakaiev. Modified
git://fedorapeople.org/~esakaiev/dnf_tests master

no initial comment

Justification

Purpose: This pull request only for reviewing changes for dnf_dcoker tests, adopting it for standard-test-roles for running it with ci-pipeline.

GitHub source:

       https://github.com/rpm-software-management/ci-dnf-stack

Link to clone dnf_docker tests:

     https://github.com/rpm-software-management/ci-dnf-stack.git

Dependencies

Current patch is aimed at adopting dnf_docker tests to standard_test_roles
- Link to the pull request for standard-test-behave-role (tests need it for execution)

    https://pagure.io/fedora-ci/AtomicCi/pull-request/73

Patches:

  1. patch for Dockerfile (Dockerfile.patch):
- https://github.com/rpm-software-management/ci-dnf-stack/blob/master/Dockerfile
includes "--skip-broken " flag to skip broken rpm during installation
  1. patch for dnf-testing.sh (dnf-testing.sh.patch)
     - https://github.com/rpm-software-management/ci-dnf-stack/blob/master/dnf-testing.sh
     This script is used for executing behave features under docker image. 
     The main idea is to run each feature under separate docker image. 
  1. standard-inventory-behave
   -  new file, need to locate under root folder
   Standard-inventory-behave inventory uses for building main docker image with all dependencies and rpm packages for behave tests
   Each behave test will clone this docker image and start execution under separate docker container. 
  1. download_rpmy.py
   -  new file, need to locate under rpmy folder
   - Download all rpm (taken from Jenkins job) 
   - need to execute python download_rpmy.py
  1. runtest.sh
   -  new file, need to locate under root folder
   -  main script, that will be trigered from "standard-test-behave" role during "Running behave tests" step
   -  it includes iteration for checking cpu usage, if cpu > 80%, script will wait while previous feature will finish execution and release cpu. 
      This step has been included for running tests in parallel.
  1. tests.yml
   - ansible playbook file
   - tests_folder variable - explicit way to determine folder with runtest.sh executable file
   - run_in_background - specify the way for execution (true means run tests in background, in parallel)
   - features -list with behave features

Run tests for Classic

    # export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
        # python rpmy/download_rpmy.py
    # export TEST_SUBJECTS=behave:dnf-bot/dnf-testing:latest
        # ansible-playbook -i standard-inventory-behave --tags=classic tests.yml

Tests results:

TASK [standard-test-behave : Check the results] *******************************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost                  : ok=14   changed=9    unreachable=0    failed=0   

PASS autoremove
PASS builddep
PASS check-1
PASS check-2
PASS check-3
PASS clean-1
PASS clean-2
PASS clean-requirements-on-remove
PASS config-1
PASS config
PASS deplist-1
PASS downgrade-1
PASS download-1
PASS gpg-1
PASS gpg-2
PASS gpg-3
PASS gpg-4
PASS gpg-checksum
PASS group-1
PASS group-2
PASS group-conditional-pkgs
PASS help-1
PASS history-1
PASS history-2
PASS history_undo
PASS install-1
PASS install-2
PASS install-downgrade-1
PASS install-obsoletes
PASS install_remove-1
PASS installroot-1
PASS installroot-2
PASS installroot-3
PASS installroot-4
PASS installroot-5
PASS list-1
PASS list-repo-installed
PASS mark-3
PASS mark-install
PASS mark-remove
PASS obsoletes-1
PASS obsoletes-2
PASS obsoletes-3
PASS obsoletes-4
PASS obsoletes-keep-reason
PASS obsoletes-one-to-one
PASS obsoletes-reason
PASS persistdir-1
PASS persistdir-2
PASS plugin-path-1
PASS plugin-path-2
PASS plugin-path-3
PASS plugin-path-4
PASS plugin-path-5
PASS plugin-path-6
PASS protected_packages
PASS provides-1
PASS reinstall-notavailable
PASS reinstall-reason
PASS remove-1
PASS remove-or-distro-sync
PASS repolist-1
PASS repolist-2
PASS repolist-disabled
PASS repolist-enabled-disabled
PASS repolist-no-repos
PASS repoquery-1
PASS repoquery-2
PASS repoquery-3
PASS repoquery-4
PASS repository-1
PASS repository-2
PASS repository-packages-info
PASS repository-packages-reinstall
PASS repository-packages-remove
PASS repository-packages-update
PASS rich-1
PASS rich-2
PASS rich-3
PASS rich-4
PASS rich-5
PASS rich-6
PASS rich-7
PASS rich-8
PASS rich-9
PASS scriptlets-1
PASS security_1
PASS security_2
PASS security_3
PASS security_4
PASS security_5
PASS setopt-1
PASS shell-1
PASS shell-2
PASS shell-3
PASS shell-4
PASS shell-5
PASS shell-6
PASS shell-7
PASS shell-8
PASS ssl-1
PASS ssl-2
PASS swap-2
PASS swap-2-shell
PASS swap-3
PASS swap-3-shell
PASS swap
PASS updateinfo-1
PASS updateinfo-2
PASS upgrade-1
PASS upgrade-2
PASS upgrade-3
PASS upgrade-to-1
PASS vars-releaseve   

1 new commit added

  • Modifying runtest.sh according with changes instandard-test-behave model

Sounds nice, although still didn't look into code

I think we need to make this feature to work dynamically. just run behave to find all tests. and then run them