Objections & FAQs

Question 1

You’re a small team - how are you going to support me? Are you going to care about me when you get bigger deals?

Answer 1

Our QA Engineers can comfortably handle up to 5 accounts. This takes into consideration that at any moment all 5 accounts could require 200% effort. In the rare instance that we need additional help fast, we have flex consultants on standby we have standing relationships with that can help.

Question 2

When you say you charge per developer, what constitutes a ‘developer’?

Answer 2

We’ve found that the number of developers on a team serves as a good proxy for the complexity of the app. Therefore, the spirit of our pricing model is to ensure we can appropriately service your account. I.e., As your team grows, we’re able to scale along with you.

That being said, a ‘developer’ refers to an engineer that has a direct influence on the functionality of the application or could potentially break it. A backend and frontend engineer would be considered a developer we support, but a data engineer may not.

Question 3

What is functional testing?

Answer 3

Functional testing tests the frontend as well as its interaction with the backend. This includes APIs and integrations. Essentially, functional testing tests anything a user can interact with.

Question 4

How much time will it require on my side?

Answer 4

Our typical partnership requires minimal effort from the client-side. The QA Wolf team is fully capable of outlining a comprehensive test matrix along with executing the automation and maintenance of those tests. The primary effort from your end will be fixing bugs we catch. Typically, clients assign one-to-a-few point persons on their end to manage the QA Wolf relationship. Most of the heavy lifting is during the first 2 weeks of the onboarding phase where we require access to testing environments.

Question 5

Do you have SOCII/Infosec/security protocols in place?

Answer 5

We don’t have official protocols like SOCII in place because we’ve found working with other clients that they are not needed. For example, we work with a number of fintech and healthcare companies where we exclusively have access to their testing environment which does not contain sensitive user information. If there ever is a case where sensitive data is in question, we can mask the data during any type of information collection. We also take our infrastructure very seriously. We only use servers located in the US and our work is never outsourced - all of our QA Engineers are in-house and based in the US.

Question 6

Where are your servers located?

Answer 6

We use Azure servers and they’re all located within the US. We chose Azure because of their container instances functionality. We are in the process of switching over the AWS as we restructure our infrastructure.

Question 7

My app is too complex for an outside firm to QA it

Answer 7

We work with a range of web apps from simple to very complex, and I’d agree that your web app skews on the side of the complex. This has never been an issue for us in the past and we don’t foresee it being one with your app either. Our QA Engineers are extremely capable and if helpful, our analytics will convert your user interactions into actual test cases which will reveal every test case in need of automation.

Question 8

Why do you need 12 weeks to get to 80% coverage?

Answer 8

80% coverage in 12 weeks is what see as the average SLA across our clients. Some companies have simpler apps and some have much more complex web apps. The reason for coverage not being instant is because we still have QA Engineers writing and maintaining your tests.

Question 9

How many days do you need to get enough data for your analytics?

Answer 9

Short answer is test cases begin rolling in within a few days and more complex and long-tail cases continue to come in over the course of a few weeks and beyond. It really depends on the size and complexity of the app along with the number of users using it on a daily basis. But even apps with only a few users can generate enough analytics for test cases.

Question 10

What happens if you go out of business?

Answer 10

We’re a VC-backed business with stable revenue and long-term contracts from reputable companies like Mainstreet, Basecamp, and Gumroad. In addition to this, we have written in our agreements with other clients that as a contingency plan, in the instance QA Wolf were to go out of business, we will implement QA Wolf on-premise at your location under an SLA.

Question 11

Why can you reach 80% coverage, and not 100%?

Answer 11

There are somethings that we can’t test, such as google reCaptcha. Also, there is technically an unlimited paths you can test (ex. on an e-commerce site, you can add and remove something from a cart over and over again). 80% is representative of all the functional and edge case tests.

Question 12

Do we have an SLA/How do SLAs work?

Answer 12

Yes, we have SLAs. SLAs are custom to what our clients want. For example, some clients have SLAs in place for maintaining tests. Others have SLAs in place for creating and maintaining tests over the weekend and early morning hours. Are there any SLAs you specifically had in mind?

Question 13

What will I do with my existing QA Team?

Answer 13

It depends on what your QA team is doing right now - the main value we provide to existing QA teams is that we remove the repetitive task of creating and maintaining tests. Some clients we work with will have their QA Engineers be our main points of contact. These QA Engineers then relay everything back to your internal product and engineering teams. This also frees up the QA Engineer’s time to pick up extra tasks and expand their bandwidth for other types of testing.

Question 14

Do you have a customer reference I can get in touch with?

Answer 14

We are being mindful of our customers’ time, as many are startup founders, and for that reason are not engaging them for reference calls unless that is the absolute last thing preventing you from signing with us. [The case studies link in that follow-up email has several success stories from companies of various sizes, and there are a number of startup logos on our site who also use Dover successfully.] That said, if QA Wolf has checked every box but that one, let me know and I can arrange an intro.

Question 15

What are Playwright’s strengths/weaknesses as compared to other testing frameworks?

Answer 15

Strength
Playwright controls the browser instead of running inside the browser like Cypress does. So it has full access to features like iFrames/multiple tabs/multiple domains. Along with the auto-waiting that Cypress has.

Weakness
A weakness Playwright has is that since it is newer it has a slightly smaller community, but it is growing quickly. 28k Playwright vs 34k Cypress GitHub stars as of right now.

Question 16

Can you create some sample tests for us?

Answer 16

[If they are a large or likely deal and you feel close to winning them
Hi [Name], Typically, we reserve test creation for a paid pilot program since it requires developer resources from our end. If these sample tests are the last pieces you’re considering before exploring a pilot with us then we’d be happy to waive any fees for up to 3 tests. Please let us know the test cases you’d like automated along with access to your test environment and we’ll start working on them. Thank you, [Name]

Question 17

We prefer to have our developers test their own code.

Answer 17

This is a fair approach we’ve seen from a lot of our current customers as ‘shifting left’ becomes more popular. Typically, we see this happen because engineering teams want their developers to have more responsibility for the quality of the code they write.

Our opinion on this is that your developers are high-value resources that should be building, writing, and shipping new code - not testing it. The end result of any business is customer satisfaction and acquisition and this is done by innovating quickly with a quality product.

Anything that stands in the way of doing that is an obstacle to overcome.

Question 18

We’ve already done so much in terms of testing [E.g. we’ve created 50 Cypress tests] is it worth it to outsource to a new company?

Answer 18

[It’s important here to ask what their current coverage is and who’s time is spent writing these tests]
The first thing I want to respond with is that we’re only here to figure out if we can help. If the tests you’ve created are already providing you with comprehensive test coverage, which we consider 80%+, and you’re able to maintain those tests while building out new ones in the future for new features, then I’d say you’re in a pretty good spot. The next question I’d have, is who is creating and maintaining those tests. If it’s your developers, then I’d say that time could be better spent writing code and building products. If it’s your QA Engineers, then I’d say their time could be freed up to focus on other areas of automation and QA.

Question 19

What’s the difference between QA Wolf and Playwright?

Answer 19

There are several differences between the bones of Playwright and QA Wolf.

For starters, we placed a lot of focus on user experience and team collaboration. With QA Wolf, all of your tests are easily accessible for your entire company right on the web - from test results to video replays of the tests.

You also get a birds-eye-view of your coverage and can filter by category of test case. Each test run also comes with a video, detailed logs, and the ability to re-run tests with one click.

With Playwright, videos need to be downloaded from GitHub Actions and test cases are presented in a non-friendly way.

Question 20

QA Wolf vs Outsourcing?

Answer 20

There are a few differences between traditional outsourced firms and QA Wolf. The two major differences are how we deliver value and how we charge for that value.

Outsourced firms focus on a narrow set of test cases handed down to them by engineering and product teams. It is a prescriptive process that will get you some automated coverage, but not very much. QA Wolf focuses on a fully comprehensive set of test cases. We typically create an exhaustive test matrix ourselves or build upon one provided by the client. Regardless of how the matrix is completed, our goal is to get you to 80% coverage within 4 months - and to keep you there indefinitely - even when new features are added. For some companies, this could mean 100 test cases and for others, it could mean 1,000. The takeaway here is that QA Wolf gets you predictable coverage in a predictable time frame while outsourced teams cannot promise the same.

The second difference is how we charge for the value we provide. Outsourced firms charge hourly, typically in 4-hour increments. The problem with hourly as I’m sure you know is that you could get a good engineer and receive more value or a junior engineer and receive less value but that 4 hours still costs you the same amount regardless of who you get. The other problem is that hourly rates penalize you for wanting more coverage because more coverage = more tests = more hours worked. QA Wolf takes a different approach and charges exclusively for coverage, not for hours worked. We base our pricing on the size of your team, which acts as a proxy for the complexity of app, and from there we promise a certain level of coverage in a certain time. So, for example, if we were supporting a team of 5 engineers we would charge $5,000/month and you would receive 80% coverage.

Where these two values really meet is in test maintenance. We’ve found that maintaining tests requires about 40% of the time it took to first create it. So when you work with an outsourced firm and that firm creates a large suite of tests for you, those tests will inevitably need maintenance and you’ll be billed at the same hourly rate. With QA Wolf, all maintenance is included with our plans.

Question 21

QA Wolf vs Hiring Internally?

Answer 21

There are two parts to this answer: (1) Value of the person creating tests and (2) Value of the test infrastructure.

Value of the person creating tests
We can only answer this question based on information we’ve gathered from previous conversations with engineering teams. That being said, every QA Engineer is different and every web application varies in complexity. For companies we’ve spoken with, we’ve found that an efficient QA Engineer typically automates 1-2 test cases per week (~50-100 per year). We’ve also found that a single QA Engineer caps out at about 50-100 test cases because test maintenance begins accounting for about 40% of their time. For comparison, similar-sized teams we’re working with (6 engineers) are on track to have 150-250 test cases automated within their first 16 weeks with us. This translates to about 10-15 newly automated tests per week. Compared to our in-house reference, this is 7-10x more efficient than hiring your own QA Engineer.

Value of the test infrastructure
QA Wolf’s monthly pricing includes an unlimited number of monthly parallel test runs. For comparison, 25 parallel tests with BrowserStack costs $3,200/month.

Other advantages for QA Wolf include (1) no recruiting costs, (2) minimal ramp-up time, (3) predictable results, and (4) low management overhead which are outlined below:

• Recruiting: Finding, interviewing, hiring, and onboarding a QA Engineer can be time-consuming and risky. QA Wolf can start immediately and we have a proven track record with existing clients
• Management: QA Wolf has a team of extremely capable engineers. We proactively identify test cases, write new tests, and maintain existing ones. We’re also very responsive and typically respond within minutes to any request. Finally, we report out weekly on the progress we’ve made and what we have planned for the upcoming week.
• Ramp-up time: QA Wolf hits the ground running with extremely capable engineers. We can begin creating tests the same day as our onboarding and we require little-to-no intervention from your end.
• Predictability: QA Wolf delivers a specified level of coverage in a specified time frame. We are bound by an agreement and tied to SLAs to make sure you’re receiving the value we promised.
• Results-driven: By the nature of us being an outsourced solution, we understand terminating our relationship is much easier than terminating a full-time employee. Because of this, we understand we survive by providing exceptional and ongoing value to our clients. We do not have the luxury of missing deadlines or taking vacations in the middle of a sprint.
• Cost-effectiveness: QA Wolf works out to be about 1/2 the cost of an automation engineer if you were to hire someone internally. In addition to being more affordable, the value is also significantly higher as our output is well above the average output of a single in-house hire.
• Headcount: If you don’t already have an open headcount for a QA Engineer, it can typically be easier to work with an outside solution such as QA Wolf.

Question 22

Can you do few tests for free as a proof of concept (PoC)?

Answer 22

Typically, we reserve test creation for a paid pilot program since it requires developer resources from our end. If these sample tests are the last pieces you’re considering before exploring a pilot with us then we’d be happy to waive any fees for up to 3 tests. Please let us know the test cases you’d like automated along with access to your test environment and we’ll start working on them.

Question 23

Is there a pilot program? Is it free or is it paid?

Answer 23

Yes, we have a 2-month pilot program. We request payment during this time since writing these tests requires an investment of engineering resources on our end. This pilot period serves as a good trial period for evaluating our effectiveness. After this time, you can decide to continue with us on a month-to-month basis, an annual basis for a discounted rate, or not continue with us at all - at which point you may export the code we wrote and use it with Microsoft’s Playwright.

Question 24

Your email/marketing material says QA Wolf automatically creates tests. Why does it take 12 weeks to get to 80% coverage?

Answer 24

The experience from your end, as the customer, is that the tests are indeed automatically created. I.e., The QA Wolf team creates test cases and automates those test cases for you. While our internal technology is becoming better every day, we are still required to have a human-in-the-loop that is writing significant portions of your tests.

Question 25

Will you sign an NDA (Non-Disclosure Agreement) before I demo my platform?

Answer 25

NOTE: If this is a large prospect, we may entertain the idea of signing an NDA. Check with your manager when in doubt.
For product demos, we do not typically sign NDAs. If we were to move forward and work together for a pilot, our standard agreement has an MNDA (Mutual Non-Disclosure Agreement) to protect the proprietary information of both parties.

Question 26

I have X external engineers [Agency/contractors] that aren’t full-time. How do you bill for these?

Answer 26

NOTE: Answer this based on how many hours they commit. E.g., If two people work 20 hrs/week then that’s equivalent to one FTE.

Question 27

Is the QA Engineer we are assigned on your side permanent? I.e., for the foreseeable future do we get the same engineer we communicate with, or is it short-term engineers that rotate through?

Answer 27

A QA Lead will be permanently assigned to your account. This person will interact with you and your team on a daily basis. The QA Lead manages our QA Engineering team and documents all knowledge regarding your account.

Question 28

Are the QA Engineers assigned to our account available to talk anytime during our workday? How quickly should we expect a response when we send a question or request or advise on a new feature coming out?

Answer 28

Yes, our QA Lead is available to talk anytime during your workday. Our goal is to be as responsive as possible and typically, we respond within minutes. We’re most active via Slack and Zoom, but if your team has alternative preferences we can accommodate those as well.

Question 29

At what level of service do we basically have an engineer working full time for us? I.e. is it 12 engineers ($12,000 a month)?

Answer 29

Our goal is to guarantee you a certain level of test coverage (E.g., 80%) and maintain or increase that level of coverage over time. The experience from your end, regardless of your current monthly rate, should be as if you have a full-time engineer writing tests on your behalf. If the spirit of this question pertains to delivering results faster, we can certainly accommodate this as well. E.g., If you’d prefer to have 80% coverage in 2 months instead of 4, we can have a discussion around how that may impact monthly rates.

Question 30

Do you have helpers/code that you can share across tests?

Answer 30

Yes, we have helpers that can be shared globally across all tests. This is helpful for use cases like common logins or other repetitive tasks.

Question 31

Do you integrate into our process? Can we run tests on each deployment?

Answer 31

You can run tests on every deployment. This is possible by integrating QA Wolf into your development process with one line of code which typically only takes a few minutes to complete.

Question 32

How quickly can you get started?

Answer 32

Typically, we can move at your pace. We’ve done kickoffs with clients within the same week of signing an agreement and we’ve also started with clients several weeks later after they’d completed a major sprint. We may be a bit biased, but starting sooner rather than later is ideal because it takes time to create tests and this was we can start ramping up your coverage as soon as possible.

Question 33

Do you support multiple device sizes?

Answer 33

Yes, we support mobile and desktop browser sizes.

Question 34

Do you support multiple browsers? Which ones?

Answer 34

Yes, we support Chrome, Firefox, and Safari.

Question 35

Is Playwright open source?

Answer 35

Yes

Question 36

What is Playwright?

Answer 36

Playwright is Microsoft’s answer to Selenium and Cypress. Playwright is built and maintained by the same team which created Puppeteer over at Google (Headless Browser Testing Framework for Google Chrome). Playwright too is an open-source NodeJS based framework.

Question 37

Can we use your platform?

Answer 37

During our engagement, your team has full access to our platform. If you choose to write tests as part of this access then you may do so.

Question 38

What if we’re not satisfied with the results of the QA Engineer assigned to our account?

Answer 38

I’d first like to add that this has not yet been an issue with any of our clients and we’ll continue doing everything we can to ensure it doesn’t become an issue for you, either.

A proactive step we take in mitigating this issue is clear and frequent communication. If there is ever an issue or concern on your end, we will do our best to quickly course correct and adapt. E.g., Writing tests differently or assigning a new engineer to your account.

Question 39

How do you ensure the accuracy of the tests being written by your QA Engineers?

Answer 39

The QA Lead assigned to your account will review every test the QA engineer writes in order to verify their accuracy.

Additionally, your entire team will have access to the QA Wolf platform through the entirety of our engagement. This means your team can view and run tests at their discretion and leave feedback as they wish.

Question 40

Will you put a proposal together and email it to me? We’ll take a look internally and get back to you.

Answer 40

I’d be happy to send a proposal. But can I make a suggestion? Would you be okay if I put together a draft of an agreement? let’s put a meeting on the calendar. let’s go through it together and make sure we fine-tune it so we end up with a customized agreement. Would you be okay with it?
If the answer is no, it’s probably a disqualification. If answer is yes, then schedule a meeting.

Question 41

We’re evaluating a few other players/solutions and will get back to you.

Answer 41

Got it, thanks for being transparent about where you’re at. What are you hoping they’re able to provide that would interest you?

Question 42

Got it, thanks for being transparent about where you’re at. What are you hoping they’re able to provide that would interest you?

Answer 42

Well, I’m not sure if you should go with us. I’d love to learn more about your current situation to see if we’ll be able to significantly add value.