Cuanto
  1. Cuanto
  2. CUANTO-70

Randomly Cuanto reports fewer test results than actually ran on the "Test Run History for <group/project>" page.

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0
    • Component/s: Database
    • Labels:
      None
    • Environment:
      Ubuntu 8.04.4 LTS on a P4 @ 2.80GHz w/ 3 Gigs of RAM
    • Number of attachments :
      0

      Description

      Randomly (about every week) Cuanto reports fewer test results than actually ran on the "Test Run History for <group/project>" page. The problem happens with multiple group/projects. We're submitting via the API.

        Activity

        Hide
        Todd Wells added a comment -

        So the totals listed on the history page are not the same as the total number of tests when viewed in the analysis view? Or are the tests missing altogether (in both views)? If the former, then when on the analysis table view, click the "Recalc" link in the upper-right to force statistics to recalculate and see if that resolves the issue.

        Show
        Todd Wells added a comment - So the totals listed on the history page are not the same as the total number of tests when viewed in the analysis view? Or are the tests missing altogether (in both views)? If the former, then when on the analysis table view, click the "Recalc" link in the upper-right to force statistics to recalculate and see if that resolves the issue.
        Hide
        Keith Felton added a comment -

        Yes, the totals on the history page are randomly not the same number of tests as viewed in the Analysis tab. Clicking the Recalc link does workaround the problem, but the below JavaScript error occurs.

        ---------------------------------------------------------------------------------------------------
        Webpage error details
        User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64;
        Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729;
        Media Center PC 6.0)
        Timestamp: Fri, 5 Nov 2010 18:26:34 UTC

        Message: Invalid argument.
        Line: 16
        Char: 2542
        Code: 0
        URI: http://nightcrawler:8080/nightcrawler/js/yui/2.6.0/datatable/cuanto-datatable-min.js
        ---------------------------------------------------------------------------------------------------

        Show
        Keith Felton added a comment - Yes, the totals on the history page are randomly not the same number of tests as viewed in the Analysis tab. Clicking the Recalc link does workaround the problem, but the below JavaScript error occurs. --------------------------------------------------------------------------------------------------- Webpage error details User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0) Timestamp: Fri, 5 Nov 2010 18:26:34 UTC Message: Invalid argument. Line: 16 Char: 2542 Code: 0 URI: http://nightcrawler:8080/nightcrawler/js/yui/2.6.0/datatable/cuanto-datatable-min.js ---------------------------------------------------------------------------------------------------
        Hide
        Todd Wells added a comment -

        I can't repro the javascript error – it occurs when you click on the Recalc link? Does it always happen or did it happen once? I tried Firefox and Chrome latest and IE7.

        Show
        Todd Wells added a comment - I can't repro the javascript error – it occurs when you click on the Recalc link? Does it always happen or did it happen once? I tried Firefox and Chrome latest and IE7.
        Hide
        Keith Felton added a comment -

        I duplicated the problem using IE8 but couldn't duplicate with Chrome or Firefox. I just noticed that the "Invalid argument" JS error occurs displaying the "Test Run <date/time>" page. So clicking on the Recalc link doesn't generate the error; it's the Test Run page that generates the error.

        Show
        Keith Felton added a comment - I duplicated the problem using IE8 but couldn't duplicate with Chrome or Firefox. I just noticed that the "Invalid argument" JS error occurs displaying the "Test Run <date/time>" page. So clicking on the Recalc link doesn't generate the error; it's the Test Run page that generates the error.
        Hide
        Todd Wells added a comment -

        My local branch is using a newer version of YUI (2.8.2) which seems like where the error is occurring (if IE is telling you correctly), so perhaps whatever the error is has been fixed in YUI 2.8.2. I know some of the datatable stuff went through substantial refactoring and I know at least one IE-specific error that I used to get has now gone away.

        Show
        Todd Wells added a comment - My local branch is using a newer version of YUI (2.8.2) which seems like where the error is occurring (if IE is telling you correctly), so perhaps whatever the error is has been fixed in YUI 2.8.2. I know some of the datatable stuff went through substantial refactoring and I know at least one IE-specific error that I used to get has now gone away.
        Hide
        Keith Felton added a comment -

        Regarding incorrectly tallying the totals on the history page (this problem is not browser specific), do you need any more info from me?

        Show
        Keith Felton added a comment - Regarding incorrectly tallying the totals on the history page (this problem is not browser specific), do you need any more info from me?
        Hide
        Todd Wells added a comment -

        No, I think it's likely a race condition.

        Show
        Todd Wells added a comment - No, I think it's likely a race condition.
        Hide
        Keith Felton added a comment -

        Are you planning on fixing the bug?

        Show
        Keith Felton added a comment - Are you planning on fixing the bug?
        Hide
        Todd Wells added a comment -

        I didn't resolve it "Won't Fix", so yes, I hope so.

        Show
        Todd Wells added a comment - I didn't resolve it "Won't Fix", so yes, I hope so.
        Hide
        Todd Wells added a comment -

        Keith, by any chance have tried 2.8.0b1 and seen the same problem there?

        Show
        Todd Wells added a comment - Keith, by any chance have tried 2.8.0b1 and seen the same problem there?
        Hide
        Todd Wells added a comment -

        Please reopen if you see this again.

        Show
        Todd Wells added a comment - Please reopen if you see this again.
        Hide
        Keith Felton added a comment -

        Todd, we're not running 2.8.0b1 yet. We're still running 2.7.1.

        Show
        Keith Felton added a comment - Todd, we're not running 2.8.0b1 yet. We're still running 2.7.1.
        Hide
        dawn hemminger added a comment -

        We recently upgraded to 2.8b2 and we're still seeing this problem. I took some snapshots if you want to see them. I don't see a way to attach them in this interface. I also don't know how to reopen an issue, so I hope you get this comment.

        Show
        dawn hemminger added a comment - We recently upgraded to 2.8b2 and we're still seeing this problem. I took some snapshots if you want to see them. I don't see a way to attach them in this interface. I also don't know how to reopen an issue, so I hope you get this comment.
        Hide
        Todd Wells added a comment -

        Is there any matter to the incorrect totals? For instance, the total not including skipped tests or something like that? Can you tell if one subset of the results is wrong? Like passed/failed/skipped? Like if you look at the total "passed" in the summary does it match the total passed when you add them up in the analysis table? Or are all the values randomly incorrect?

        Show
        Todd Wells added a comment - Is there any matter to the incorrect totals? For instance, the total not including skipped tests or something like that? Can you tell if one subset of the results is wrong? Like passed/failed/skipped? Like if you look at the total "passed" in the summary does it match the total passed when you add them up in the analysis table? Or are all the values randomly incorrect?
        Hide
        dawn hemminger added a comment -

        In this case, there were 397 tests run. We're using the cuanto API with a wrapper we created to write the tests to Cuanto. All tests ran and all passed (no fails, no skips, etc). I manually counted the number of tests displayed under the analysis tab and it's 397 just like the analysis page states on the bottom left corner, so all tests are accounted for. It's only when you're looking at the "Most recent runs" or "Test Runs History" view, that it's showing 396 tests run and passed. In the analysis view, if I hit "recalc" and then go back to the "most recent runs" view and refresh a few times, the count eventually comes up to 397 as expected.

        Note, this is similar to the problem that still exists in this build where when you enter any analysis information, e.g. "Test Bug" under the "Reason" column, the analyzed results do not show up in the "most recent runs" or "Test Runs History" view. You have to hit "recalc" in the Analysis view first to get that analyzed data to update in those other views.

        For this particular issue, I'm running Firefox 7.0.1.

        Show
        dawn hemminger added a comment - In this case, there were 397 tests run. We're using the cuanto API with a wrapper we created to write the tests to Cuanto. All tests ran and all passed (no fails, no skips, etc). I manually counted the number of tests displayed under the analysis tab and it's 397 just like the analysis page states on the bottom left corner, so all tests are accounted for. It's only when you're looking at the "Most recent runs" or "Test Runs History" view, that it's showing 396 tests run and passed. In the analysis view, if I hit "recalc" and then go back to the "most recent runs" view and refresh a few times, the count eventually comes up to 397 as expected. Note, this is similar to the problem that still exists in this build where when you enter any analysis information, e.g. "Test Bug" under the "Reason" column, the analyzed results do not show up in the "most recent runs" or "Test Runs History" view. You have to hit "recalc" in the Analysis view first to get that analyzed data to update in those other views. For this particular issue, I'm running Firefox 7.0.1.
        Hide
        Todd Wells added a comment -

        I've disabled caching for TestRunStats in commit 937a0ec in an attempt to resolve this issue.

        Show
        Todd Wells added a comment - I've disabled caching for TestRunStats in commit 937a0ec in an attempt to resolve this issue.
        Hide
        Todd Quessenberry added a comment -

        There is a race condition due to the request for stats calculations not being removed from the queue until the calculation is complete. If the last results for a test run came in while the calculation for that test run was running, there's a decent chance they wouldn't get included. A fix that dequeues the request before handling it is at https://github.com/toddq/cuanto/commit/bad43a71fad6c06479aaa0b3e4acbba634faa34a.

        Show
        Todd Quessenberry added a comment - There is a race condition due to the request for stats calculations not being removed from the queue until the calculation is complete. If the last results for a test run came in while the calculation for that test run was running, there's a decent chance they wouldn't get included. A fix that dequeues the request before handling it is at https://github.com/toddq/cuanto/commit/bad43a71fad6c06479aaa0b3e4acbba634faa34a .

          People

          • Assignee:
            Todd Wells
            Reporter:
            Keith Felton
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: