Skip to content

Fix failing integration tests #96

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
adangel opened this issue Nov 29, 2021 · 0 comments · Fixed by #97
Closed

Fix failing integration tests #96

adangel opened this issue Nov 29, 2021 · 0 comments · Fixed by #97
Assignees
Labels
bug Something isn't working
Milestone

Comments

@adangel
Copy link
Member

adangel commented Nov 29, 2021

Integration tests need to be updates, they are failing right now: https://github.com/pmd/pmd-regression-tester/actions/workflows/manual-integration-tests.yml

The update of the test projects (see pmd/pmd#3640) is applied already, since this integration test just runs on pmd from master branch. We probably need to update our patch files, which simulate some changes.

In detail:

  • test_case_1_single_java_rule_changed
2021-11-29T10:12:00.4054476Z {:errors=>{:changed=>0, :new=>0, :removed=>1, :base_total=>40, :patch_total=>39}, :violations=>{:changed=>0, :new=>0, :removed=>512, :base_total=>512, :patch_total=>0}, :configerrors=>{:changed=>0, :new=>0, :removed=>3, :base_total=>3, :patch_total=>0}}
2021-11-29T10:12:00.4055486Z #############################
2021-11-29T10:12:00.4056013Z F
2021-11-29T10:12:00.4056599Z ===============================================================================
2021-11-29T10:12:00.4057716Z Failure: test_case_1_single_java_rule_changed(ManualIntegrationTests)
2021-11-29T10:12:00.4059145Z test/manual_integration_tests.rb:48:in `test_case_1_single_java_rule_changed'
2021-11-29T10:12:00.4060281Z      45:     assert_equal(0, @summary[:violations][:changed], 'found changed violations')
2021-11-29T10:12:00.4061383Z      46:     assert_equal(0, @summary[:violations][:new], 'found new violations')
2021-11-29T10:12:00.4063044Z      47:     # These are the artificially created false-negatives for AbstractClassWithoutAbstractMethod rule
2021-11-29T10:12:00.4064646Z   => 48:     assert_equal(34 + 234, @summary[:violations][:removed], 'found removed violations')
2021-11-29T10:12:00.4065449Z      49: 
2021-11-29T10:12:00.4066137Z      50:     # errors might have been caused in the baseline for other rules (only visible in the stacktrace)
2021-11-29T10:12:00.4066976Z      51:     # hence they might appear as removed
2021-11-29T10:12:00.4067543Z found removed violations
2021-11-29T10:12:00.4068034Z <268> expected but was
2021-11-29T10:12:00.4068433Z <512>

We have now many more removed violations (512 instead of 268). This could be just because of the changed projects - the baseline probably has now more violations that are affected...

  • test case 2 - single xpath rule changed
2021-11-29T10:21:56.1686223Z �[0;0m#############################: test_case_2_single_xpath_rule_changed
2021-11-29T10:21:56.1687267Z {:errors=>{:changed=>0, :new=>0, :removed=>1, :base_total=>40, :patch_total=>39}, :violations=>{:changed=>0, :new=>15, :removed=>477, :base_total=>64130, :patch_total=>63668}, :configerrors=>{:changed=>0, :new=>0, :removed=>3, :base_total=>3, :patch_total=>0}}
2021-11-29T10:21:56.1688397Z #############################
2021-11-29T10:21:56.1688730Z F
2021-11-29T10:21:56.1689251Z ===============================================================================
2021-11-29T10:21:56.1689942Z Failure: test_case_2_single_xpath_rule_changed(ManualIntegrationTests)
2021-11-29T10:21:56.1691218Z test/manual_integration_tests.rb:81:in `test_case_2_single_xpath_rule_changed'
2021-11-29T10:21:56.1691974Z      78:     print "#############################: test_case_2_single_xpath_rule_changed\n" \
2021-11-29T10:21:56.1692664Z      79:           "#{@summary}\n#############################\n"
2021-11-29T10:21:56.1693470Z      80:     assert_equal(0, @summary[:violations][:changed], 'found changed violations')
2021-11-29T10:21:56.1694413Z   => 81:     assert_equal(0, @summary[:violations][:new], 'found new violations')
2021-11-29T10:21:56.1695266Z      82:     # There are no violations, that have been removed for AvoidMessageDigestField
2021-11-29T10:21:56.1696337Z      83:     assert_equal(0, @summary[:violations][:removed], 'found removed violations')
2021-11-29T10:21:56.1696872Z      84: 
2021-11-29T10:21:56.1697229Z found new violations
2021-11-29T10:21:56.1697645Z <0> expected but was
2021-11-29T10:21:56.1697997Z <15>

That's weird - by disabled one xpath rule we got new violations - that shouldn't happen.

  • test case 3 - change in pmd-core
2021-11-29T10:46:27.0962509Z �[0;0m#############################: test_case_3_change_in_core
2021-11-29T10:46:27.0964183Z {:errors=>{:changed=>0, :new=>0, :removed=>0, :base_total=>40, :patch_total=>40}, :violations=>{:changed=>0, :new=>36, :removed=>474, :base_total=>826504, :patch_total=>826066}, :configerrors=>{:changed=>0, :new=>0, :removed=>0, :base_total=>3, :patch_total=>3}}
2021-11-29T10:46:27.0965298Z #############################
2021-11-29T10:46:27.0965958Z F
2021-11-29T10:46:27.0966875Z ===============================================================================
2021-11-29T10:46:27.0967612Z Failure: test_case_3_change_in_core(ManualIntegrationTests)
2021-11-29T10:46:27.0969303Z test/manual_integration_tests.rb:114:in `test_case_3_change_in_core'
2021-11-29T10:46:27.0971638Z      111:     print "#############################: test_case_3_change_in_core\n" \
2021-11-29T10:46:27.0972240Z      112:           "#{@summary}\n#############################\n"
2021-11-29T10:46:27.0973910Z      113:     assert_equal(0, @summary[:violations][:changed], 'found changed violations')
2021-11-29T10:46:27.0975061Z   => 114:     assert_equal(0, @summary[:violations][:new], 'found new violations')
2021-11-29T10:46:27.0977058Z      115:     assert_equal(0, @summary[:violations][:removed], 'found removed violations')
2021-11-29T10:46:27.0978172Z      116:     assert_equal(0, @summary[:errors][:removed], 'found removed errors')
2021-11-29T10:46:27.0979827Z      117:     assert_equal(0, @summary[:errors][:changed], 'found changed errors')
2021-11-29T10:46:27.0980502Z found new violations
2021-11-29T10:46:27.0980987Z <0> expected but was
2021-11-29T10:46:27.0981916Z <36>

Here we got again new violations, while we expected no change at all.

  • test case 4 - unrelated change

That's fine, this test case still works.

@adangel adangel added the bug Something isn't working label Nov 29, 2021
@adangel adangel added this to the 1.2.1 milestone Nov 29, 2021
@adangel adangel self-assigned this Nov 30, 2021
adangel added a commit to adangel/pmd-regression-tester that referenced this issue Nov 30, 2021
adangel added a commit that referenced this issue Dec 1, 2021
Update integration tests #97

* pr-97:
  Update integration tests after pmd/pmd#3666 has been merged
  [ci] Use new cache
  Update release notes (#96)
  Update integration tests
  [ci] Run these integration tests monthly automatically
  Update test case 5
  Use en_US.UTF-8 as locale, don't sign the commits
  Fix test case 2
  Fix test case 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant