Momentic Product and use-case questions

Question 1

Do you have customers with >3,000 UI tests?

Answer 1

Webflow has over 10,000 UI tests

Question 2

Do you have customers with >300,000 test executions per month?

Answer 2

Notion executes >200,000 tests per day as part of CI/CD on every commit

Question 3

Can you support >2,500 test executions in a 15-minute period from a single customer?

Answer 3

At peak, Notion executes >10,000 tests in a 15-minute period as part of CI/CD on every commit

Question 4

Can you support >10,000 test executions in a 15-minute period from a single customer?

Answer 4

At peak, Notion executes >10,000 tests in a 15-minute period as part of CI/CD on every commit

Question 5

Do your customers who already have automed UI tests see a significant boost in productivity? Provide a % productivity boost from a customer with previously automated UI tests and how you calculated this in the comments.

Answer 5

Notion migrated from Selenium and saw a 70% decrease in time to automate, 30% decrease in execution time, and 99% decrease in test flakiness. This is self-reported from the customer using their Datadog dashboards and analytics.

Question 6

Do your customers who already have automated UI tests see significant cost savings when moving over from previous systems / products? Provide a % cost savings from a customer with previously automated UI tests and how you calculated this in the comments.

Answer 6

Yes. Customers with previously automated UI tests have seen dramatic cost savings after switching, largely due to improvements in test reliability and reductions in maintenance and debugging time.

For Customer 1, moving from Cypress to Momentic, savings were calculated at approximately 98.7% reduction in engineering costs associated with addressing flaky tests (from $225,000/month to $975/month). This was based on total number of executions, the percentage of tests that were flaky, the debugging time required per failure, and the hourly engineering rate.⁠
⁠
Calculation example (Customer 1):
Before: 150 tests, 3% failure rate, 50 executions/day
225 failed tests/day × 30 days = 6,750 failed tests/month
30 min to resolve each = 3,375 engineer-hours/month
At $100/hr = $337,500/month on flaky test maintenance

After: Failure rate drops to 0.65%
0.975 failed tests/day × 30 days = 29.25/month
29.25 × 30 min = 14.625 engineer-hours/month
$1,462.50/month on maintenance
Cost savings: ($337,500 - $1,462.50)/$337,500 ≈ 99.6% cost savings on flaky test maintenance

Even accounting for variations in engineer cost, execution volume, or failure rates, customers consistently report 60–99% cost savings post-migration, especially in time spent maintaining and debugging tests.

Question 7

Do you support no code / low code options for creating automated UI tests?

Answer 7

Yes. Momentic’s tests are all completely built using our low-code editor. Test definitions in Momentic are all in natural language. If you can describe the scenario to a manual tester, you can automate it in Momentic.

https://momentic.ai/docs/editor/test-definition

Question 8

Do you support porting over existing UI tests written in python?

Answer 8

Yes. Momentic supports importing any tests in any language. Momentic translates imported tests using codebase context into natural language test plans and then automatically generates Momentic tests.

Question 9

Is the average rework time of a ported UI test <10 minutes per test? Put the average time in the comments section.

Answer 9

The average time to port a UI test depends on the complexity of the test.

However, Notion was able to migrate 200 Selenium tests into Momentic with an average time of 6 minutes per test.

Question 10

Can you upload tests in Google Docs or PDF format?

Answer 10

Yes. Momentic’s test generation ingests text and turns them into working Momentic tests.

Question 11

(Google docs or PDF) Dev Designs

Answer 11

Yes. Momentic’s test generation ingests text and turns them into working Momentic tests.

Question 12

(Figma) UX Mocks

Answer 12

No. Figma UX mocks do not include enough information to accurately determine user flows under test.

Question 13

Do you support self-healing tests that can adapt to UI / selector changes?

Answer 13

Yes. Momentic tests are defined in natural language. This means that they are not tied to hardcoded attributes like XPath or CSS selectors. As long as there is an element on the page that matches the input description, our AI will find it and adapt.

For larger user flow changes, Momentic is able to make edits to tests as part of the Failure recovery process. These changes will require human approval to be accepted.

https://momentic.ai/docs/prompting/finding-elements
https://momentic.ai/docs/failure-recovery

Question 14

Do you support an audit log of changes made by the self-healing features?

Answer 14

Yes. Every test execution has a corresponding test trace. We surface information such as targeted element, AI thoughts, screenshots, HTML & a11y snapshots to users to share how self-healing was done.

Question 15

Do you support the ability to review and approve the changes made by the self-healing features?

Answer 15

Yes. Every test execution has a corresponding test trace. Proposed changes made by Momentic’s AI as part of Failure recovery will show up as a diff that needs to be reviewed by a human and accepted.

https://momentic.ai/docs/failure-recovery

Question 16

Do you support strictly defined asserts?

Answer 16

Yes. Momentic supports a variety of different assertions. We support JavaScript assertions, element-based assertions, page content assertions, as well as multi-modal AI assertions.

https://momentic.ai/docs/prompting/writing-assertions

Question 17

Do you support fuzzy asserts (for testing AI features)?

Answer 17

Yes. Fuzzy assertions are supported through multi-modal AI assertions.

Quora uses Momentic to test poe.com which is an AI chatbot platform. They use AI assertions to assert on dynamically generated content and whether it is correct and on topic.

Question 18

Do you support reusable steps / setup (i.e. steps to log in)?

Answer 18

Yes. Momentic supports reusable steps through modules. Modules are shared steps that can be parameterized just like functions. You can use modules to share and reuse common logic like log in.

https://momentic.ai/docs/organization/modules

Question 19

Do you support the ability to see and modify the code behind the human readable test case steps?

Answer 19

Yes. Momentic’s natural language test cases serialize to YAML files which are human readable and reviewable.

https://github.com/momentic-ai/examples/blob/main/single-project/create-react-app-test.test.yaml

Question 20

Do you support multiple browsers? Include in the comments which browsers you support.

Answer 20

Yes. We currently support Chrome, Chromium and Chrome for Testing. We plan on adding support for Safari, Firefox, and Edge in H2 2025.

https://momentic.ai/docs/browsers

Question 21

Do you support mobile browser sizes? Include in the comments which browser sizes you support or if it is completely customizable.

Answer 21

Yes. We support arbitrary browser sizes that is completely customizable by our customers.

Question 22

Do you support multiple operating systems? Include in the comments which operating systems you support.

Answer 22

Yes. We support running in multiple operating systems.

Windows 10+, Windows Server 2016+, or Windows Subsystem for Linux (WSL).
macOS 13 Ventura, or macOS 14 Sonoma.
Debian 11, Debian 12, Ubuntu 20.04 or Ubuntu 22.04, Ubuntu 24.04, on x86-64 and arm64 architecture.

Question 23

Do you support calling API’s as part of test setup?

Answer 23

Yes. Momentic supports sending API calls as a part of test setup.

Question 24

Do you support calling API’s as part of test assertions?

Answer 24

Yes. Momentic supports sending API calls as a part of test assertions.

Question 25

Do you support calling API's as part of test cleanup?

Answer 25

Yes. Momentic supports sending API calls as a part of test cleanup.

Question 26

Do you support image comparison testing (testing dashboard widgets / graphs look as expected)?

Answer 26

Yes. Momentic supports image comparisons using our Visual Diff step. https://momentic.ai/docs/steps/visual-diff

Question 27

Do you support pixel perfect image comparison testing?

Answer 27

Yes. Momentic supports image comparisons using our Visual Diff step. https://momentic.ai/docs/steps/visual-diff

Question 28

Do you support globalization testing? (ability to support multi-byte char sets as well as right-to-left languages)

Answer 28

Yes. Momentic supports multi-byte char sets as well as right-to-left languages. We have customers who are using Momentic to test globalization and internationalization in their applications.

Question 29

Do you support file uploads? (ability to define a test resource of various file formats that need to be uploaded during the test execution)

Answer 29

Yes. Momentic supports uploading files in different formats. https://momentic.ai/docs/files

Question 30

Do you support hosting the files that need to be uploaded during test execution?

Answer 30

Yes. Momentic supports hosting the files that need to be uploaded during test execution. https://momentic.ai/docs/files#cloud-hosted-files

Question 31

Do you support entire test brand clean-up?

Answer 31

Yes. You can orchestrate setup and teardown through Momentic or as part of your CI/CD process.

Question 32

Do you support parametrized tests? Can a single test run across multiple environments (parametrized by different URLs and different credentials)?

Answer 32

Yes. Momentic tests can be fully parameterized and executed against different URLs and different environments. Environments can be configured with custom URLs and variables to store information like credentials. https://momentic.ai/docs/cli/configuration#environments https://support.momentic.ai/articles/4803493941-how-do-i-run-a-test-against-different-inputs-in-cli https://momentic.ai/docs/cli/commands/run#env-%3Cenv%3E https://momentic.ai/docs/cli/commands/run#url-override-%3Curloverride%3E

Question 33

Do you support pausing a test (manually marking a test to be skipping during test execution to not break CI/CD for a known issue)?

Answer 33

Yes. Momentic tests can be marked as skipped and it will not be executed.

Question 34

Do you support a test utilizing random values each time to create fuzzy tests and increase test coverage?

Answer 34

Yes. Momentic supports executing JavaScript and utilizing libraries like Faker.js to generate fake data and random values. https://momentic.ai/docs/javascript#third-party-packages

Question 35

Do you support data driven tests (data file iteration for input to tests)?

Answer 35

Yes. Momentic supports parameterized tests and using a data file to iterate. The data file must be in CSV format. https://support.momentic.ai/articles/4803493941-how-do-i-run-a-test-against-different-inputs-in-cli

Question 36

Do you support accessibility testing? Please into comments what type of accessibility testing you support.

Answer 36

No. We currently don't support accessibility testing in Momentic. However, this is an active area of feature development for our customers and we expect to launch it in H2 2025. https://momentic-ai.notion.site/a11y-testing

Question 37

Do you support the ability to pull from a pool of test credentials during parallelization so a single credential is only having 1 concurrent test at a time (checking out a credential while it is being used)?

Answer 37

Yes. You can specify which environment and credentials a test is using during execution. This will be orchestrated as part of your CI/CD pipeline.

Question 38

Do you support testing mobile apps?

Answer 38

No. Mobile testing is in development with key partners like Notion, Quora, Eventbrite, Compass and set to launch H2 2025.

Question 39

Is the backend language of AI-generated tests in a portable language (playwright, selenium, cypress, etc.)? Please name the language in the comments.

Answer 39

No. Momentic is a proprietary, no-code AI testing framework and does not directly generate tests in a portable language like Playwright, Selenium, or Cypress.

Question 40

Do you have any features that replace / enhance manual ad-hoc testing? Please describe that functionality in the comments.

Answer 40

No. Momentic supports agentic testing where you can instruct an AI agent to complete a task in natural language. It can be used for automating exploratory or manual ad-hoc testing. https://momentic.ai/docs/steps/ai-action

Question 41

Do you support Retrieval-Augmented Generation (RAG) to get additional context from internal & public docs at test generation time?

Answer 41

Yes. You can ingest external and internal documentation into Momentic's AI knowledge base so that it can understand your application and product terminology.

Question 42

Do you support Retrieval-Augmented Generation (RAG) to get additional context from internal & public docs while running tests?

Answer 42

Yes. You can ingest external and internal documentation into Momentic's AI knowledge base so that it can understand your application and product terminology.

Question 43

Can you execute a single test suite with 300-500 UI tests in <30 minutes? Describe in comments if this is primarily done through creating faster UI tests or if this is done via parallelization.

Answer 43

Yes. Tests executed on Momentic is fully parallelized on our Kubernetes cluster in Google Cloud. Our average wall time is 5 minutes for node startup. The exact duration will depend on how long and complex your tests our. The total duration would be 5 minutes + max(duration of each test).

Question 44

Do you support Spinnaker? The ability for Spinnaker to kick off test suites as part of CI/CD pipelines?

Answer 44

Yes. Any CI/CD platform that supports running Node.js 18+ will be able to execute Momentic tests.

Question 45

Do you support sending test results to Slack when a test run finishes? Please add in comments the level of detail that the Slack report contains.

Answer 45

Yes. Notification details includes test name, trace URL, start time, end time, duration, AI root cause analysis, failure type, failure reason, and failure message. The message can be customized per-test to include arbitrary data and text from the test itself.

Question 46

Do you support a Jira integration?

Answer 46

No. We don't currently have a 1st party Jira integration as you can link test execution results and traces to a Jira ticket via URL.

Question 47

Can you link a Jira ID to a test failure?

Answer 47

Yes. All Momentic test traces are linkable to a Jira ticket via URL.

Question 48

Can you write jira compatible / friendly bug reports based off of failures?

Answer 48

Yes. All Momentic test traces include an AI root cause analysis of the failure that summarizes the test execution and the cause of the defect as well as the type of the defect. This can be linked directly to a Jira ticket or bug report.

Question 49

Can you support SDE reproduction of the defects either by generating steps to reproduce or push-button rerun for non-testers?

Answer 49

Yes. All Momentic tests can be re-executed with a push of a button. Every test trace in Momentic includes screenshots, video, console logs, network logs, HTML snapshots, and accessibility snapshots to aid in debugging and reproducing the defect.

Question 50

Can you support creating a UI automated test from a Jira bug report?

Answer 50

Yes. Momentic's test generation ingests text and turns them into working Momentic tests.

Question 51

Do you support a full export of test results? Please respond in comments if this can be put on a schedule or if it is API based

Answer 51

Yes. All test results are able to be exported as a zip file including JSON logs, screenshots, video, and execution results.

Question 52

Do you support reporting on average test pass rate of a test project?

Answer 52

Yes. This is surfaced in our analytics dashboard. This includes average pass rate, average duration, median duration, p95 duration, flake rate, and number of executions.

Question 53

Do you support reporting on test flakiness? (not for updates based on UI changes, but more around timeouts / error messages / intermittent issues)?

Answer 53

Yes. This is surfaced in our analytics dashboard. We define test flakiness as test executions within the same CI pipeline that has varying results.

Question 54

Do you record screenshots on test failure?

Answer 54

Yes. Every test trace in Momentic includes screenshots, video, console logs, network logs, HTML snapshots, and accessibility snapshots to aid in debugging.

Question 55

Do you record a video of the entire test execution on test failure?

Answer 55

Yes. Every test trace in Momentic includes screenshots, video, console logs, network logs, HTML snapshots, and accessibility snapshots to aid in debugging.

Question 56

Do you report on the network/console logging on test failure?

Answer 56

Yes. Every test trace in Momentic includes screenshots, video, console logs, network logs, HTML snapshots, and accessibility snapshots to aid in debugging.