Failing tests marked as skipped upgrading from 13.1.0 to 13.2.0 (up to 14.1.0) [arm/aarch64]

[lemonad]

Update:

I've added a minimal test setup below and it seems like this might be a combination of two things. Starting with 13.2.0, I get an error

1) My First Test
       Does not do much and fails!:
     Error: EFAULT: bad address in system call argument, write

and when also adding defaultCommandTimeout: 8000 to the config, failing tests are marked as skipped silently.

My First Test

  0 passing (8s)


  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        2                                                                                │
  │ Passing:      0                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      2                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     7 seconds                                                                        │
  │ Spec Ran:     spec.cy.js

In 13.1.0, I get the expected response

My First Test
    1) Does not do much and fails!
    ✓ Does not do much! (621ms)


  1 passing (14s)
  1 failing

  1) My First Test
       Does not do much and fails!:
     AssertionError: Timed out retrying after 8000ms: Expected to find element: `#DoesNotExist`, but never found it.
      at Context.eval (webpack:///./cypress/e2e/spec.cy.js:4:7)

Current behavior

After upgrading to version 14.1.0 we noticed failing tests silently being marked as skipped. Once a failing test is executed, that test and all subsequent tests are marked as skipped. Thus CI will always give a green checkmark.

Not making any modifications and running on macOS works as expected, leading me to believe that the problem might have to do with arm/aarch64 but not only that. Now, we are executing the cypress/included docker containers on an embedded device with a custom Yocto distribution built on Ubuntu 18.04.5 LTS. However, as the tests are being run inside the container, I think that has little bearing on the problem. I binary searched through all versions and found that 13.1.0 is the last version that works as expected and 13.2.0 skips tests.

Our tests are very basic, like below. The configuration is without support files or hooks. We start the web server externally so cypress is just pointing to http://localhost:80 via baseUrl.

beforeEach(() => {
    cy.visit("/start");
  });

  it("can enter settings from main", () => {
    cy.get("#ExpandButton").click();
    // Avoid double clicking by waiting.
    cy.wait(500).get("#SettingsButton").click();
    cy.get("#OptionsNav");
  });

Example with a failing test at the top of the test file:

docker run -it --rm --network host -v /media/data/testing:/e2e -w /e2e --entrypoint bash cypress/included:13.2.0 -c "cypress run"


DevTools listening on ws://127.0.0.1:37863/devtools/browser/6bf62af7-1e97-4868-9df3-6514f5be3bb2

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        13.2.0                                                                         │
  │ Browser:        Electron 114 (headless)                                                        │
  │ Node Version:   v20.6.1 (/usr/local/bin/node)                                                  │
  │ Specs:          1 found (settings.cy.ts)                                                       │
  │ Searched:       cypress**/*.cy.{ts,tsx}                                                        │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  settings.cy.ts                                                                  (1 of 1)


  Settings Page

  0 passing (12s)


  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        3                                                                                │
  │ Passing:      0                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      3                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     12 seconds                                                                       │
  │ Spec Ran:     settings.cy.ts                                                                   │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  settings.cy.ts                           00:12        3        -        -        -        3 │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:12        3        -        -        -        3

Example with the same test moved last

docker run -it --rm --network host -v /media/data/testing:/e2e -w /e2e --entrypoint bash cypress/included:13.2.0 -c "cypress run"


DevTools listening on ws://127.0.0.1:46709/devtools/browser/67599640-b306-477d-97b2-8321d5de2ba8

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        13.2.0                                                                         │
  │ Browser:        Electron 114 (headless)                                                        │
  │ Node Version:   v20.6.1 (/usr/local/bin/node)                                                  │
  │ Specs:          1 found (settings.cy.ts)                                                       │
  │ Searched:       cypress**/*.cy.{ts,tsx}                                                        │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  settings.cy.ts                                                                  (1 of 1)


  Settings Page
    ✓ can enter settings from start (3329ms)
    ✓ can enter settings from main (5257ms)

  2 passing (10s)


  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        3                                                                                │
  │ Passing:      2                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      1                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     10 seconds                                                                       │
  │ Spec Ran:     settings.cy.ts                                                                   │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  settings.cy.ts                           00:10        3        2        -        -        1 │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:10        3        2        -        -        1

Using exactly the same test files with cypress/included:13.1.0, the results are as expected

docker run -it --rm --network host -v /media/data/testing:/e2e -w /e2e --entrypoint bash cypress/included:13.1.0 -c "cypress run"


DevTools listening on ws://127.0.0.1:38371/devtools/browser/46e0f31f-5c0d-4a13-8a19-ffdf1fba74dd
[434:0228/082302.734027:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported.

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        13.1.0                                                                         │
  │ Browser:        Electron 106 (headless)                                                        │
  │ Node Version:   v20.5.0 (/usr/local/bin/node)                                                  │
  │ Specs:          1 found (settings.cy.ts)                                                       │
  │ Searched:       cypress**/*.cy.{ts,tsx}                                                        │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  settings.cy.ts                                                                  (1 of 1)


  Settings Page
    ✓ can enter settings from start (3681ms)
    ✓ can enter settings from main (5273ms)
    (Attempt 1 of 3) can render setup settings page
    (Attempt 2 of 3) can render setup settings page
    1) can render setup settings page


  2 passing (1m)
  1 failing

  1) Settings Page
       can render setup settings page:
     AssertionError: Timed out retrying after 8000ms: Expected to find element: `#WifiHotspotParameters2`, but never found it.
      at Context.eval (webpack:///./cypress/e2e/settings.cy.ts:30:0)




  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        3                                                                                │
  │ Passing:      2                                                                                │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  3                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     1 minute, 6 seconds                                                              │
  │ Spec Ran:     settings.cy.ts                                                                   │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Screenshots)

  -  /e2e/cypress/screenshots/settings.cy.ts/Settings Page -- can render setup settin     (1280x720)
     gs page (failed).png
  -  /e2e/cypress/screenshots/settings.cy.ts/Settings Page -- can render setup settin     (1280x720)
     gs page (failed) (attempt 2).png
  -  /e2e/cypress/screenshots/settings.cy.ts/Settings Page -- can render setup settin     (1280x720)
     gs page (failed) (attempt 3).png


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  settings.cy.ts                           01:06        3        2        1        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 1 failed (100%)                     01:06        3        2        1        -        -

Desired behavior

No response

Test code to reproduce

cypress.config.js:

const { defineConfig } = require('cypress')

module.exports = defineConfig({
  projectId: 'Test',
  e2e: {
    baseUrl: 'http://localhost:80',
    specPattern: "cypress/e2e/**/*.cy.{js,jsx}",
    supportFile: false,
  },
  defaultCommandTimeout: 8000,
})

cypress/e2e/spec.cy.js:

describe('My First Test', () => {
  it('Does not do much and fails!', () => {
    cy.visit("/")
    cy.get("#DoesNotExist")
  })

  it('Does not do much!', () => {
    cy.visit("/")
  })
})

Cypress Version

14.1.0

Node version

The versions that come with the cypress/included docker containers

Operating System

Ubuntu 18.04.5 LTS on aarch64

Debug Logs

I can add these but not sure if I should add the results from 13.2.0 or 14.1.0.

Other

No response

[jennifer-shehane]

@lemonad We don't have any specific logic to handle things differently in aarch64. I'd suggest narrowing this down to a super simple example - no app code, no CI setup, no plugins (no grep/fail fast), no 'rerunning in CI', no external reporters. I'm having a hard time reasoning how Cypress could be executing logic to skip all the tests until a normal first time run.

I wasn't able to reproduce it on my macOS, but you also said it's not reproducible there.

[lemonad]

@jennifer-shehane I don't think I know how to make it simpler than the two files I included above (I basically followed the Cypress install/create your first test instructions and then added the defaultCommandTimeout). Even if our host is a custom Yocto build (running on a Nvidia Jetson Xavier NX), it seems the cypress/included Docker container actually running the Cypress tests should be isolated from all that.

Our Github CI is also running on the same hardware/Yocto so that doesn't help with replication. It might just be a special case unfortunately. We can use 13.1.0 for now, but it would be nice to be able to upgrade in the future.

The changelog for 13.2.0 is short and it feels like the Electron update might have something to do with it. The 13.1.0 container runs Electron 106 and Node 20.5.0 while the 13.2.0 container runs Electron 114 and Node 20.6.1.

[jennifer-shehane]

@lemonad Have you tried running the tests in Chrome to exclude Electron as a browser issue?

[lemonad]

@jennifer-shehane Oh, that's a good suggestion! I just have to figure out which docker image to use and then I'll try it.

[MikeMcC399]

• There is no Chrome browser for arm64 - see Add Chrome to Linux/arm64 browser images cypress-docker-images#1188
• Chromium and Firefox ESR browsers can be added to a custom Cypress Docker image . See the examples directory in the Cypress Docker repo.

In Discord https://discord.com/channels/755913899261296641/1344786011510542356 we discussed that the repro

git clone --branch test/fail https://github.com/MikeMcC399/cypress-docker-images
cd cypress-docker-images
cd examples/basic-mini
docker run -it --rm -v .:/app -w /app cypress/included

works as expected on GitHub-hosted runners and fails with the error

     Error: EFAULT: bad address in system call argument, write

on the problematic Ubuntu 18 host.

Docker no longer lists Ubuntu 18 as actively supported (see https://docs.docker.com/engine/install/ubuntu/ with minimum Ubuntu Focal 20.04 (LTS)). Docker removed support for Ubuntu 18.04 with docker/docs@c770500 on Jun 21, 2023.

[MikeMcC399]

The issue is not reproducible on CircleCI with the machine image ubuntu-2004:2023.10.1 containing Docker 24.0.6.

This is an active runner on Ubuntu 20.04 with the closest matching Docker version 24.0.6 compared to the final version Docker 24.0.2 released for Ubuntu 18.04. CircleCI no longer has any active Ubuntu 18.04 machine images available due to end of standard support of this operating system in June 2023.

Possibly there is a Docker compatibility issue with the custom Yocto distribution built on Ubuntu 18.04.5 LTS. This version of Ubuntu is now unsupported by Docker.

[MikeMcC399]

@lemonad

Do you have an alternate NVIDIA Jetson Xavier NX module to test on, in case the EFAULT memory error is caused by a hardware issue?

[MikeMcC399]

@lemonad

Please let us know your plans for this issue. It appears that you are using Docker in an unsupported environment and unless the Cypress issue can be reproduced in a supported environment it is going to be difficult for the Cypress.io team or the community to help you further.

[MikeMcC399]

I suggest to close this issue as there hasn't been any further feedback.