Rufous

{
  "post": {
    "title": "Should you performance test if you have no dedicated environment?",
    "selftext": "We are currently trying to implement performance testing for a client, however we have numerous issues/blockers that won't allow us to have a dedicated environment for perf testing.\n\nMy question is basically what benefit would we get from running perf tests in a QA environment that has nowhere near the same capabilities as prod (no load balancer and reduced  specs) as opposed to running that in a  dedicated environment that matches prod?\n\nMy thoughts are that it provides little to no value, especially considering we've received emails in the past from the client asking us to slow down our manual testing since we are putting strain on the QA environment.",
    "url": "https://www.reddit.com/r/QualityAssurance/comments/13iyxn1/should_you_performance_test_if_you_have_no/"
  },
  "comments": [
    {
      "body": "One thing is basically having factors to gauge the smaller test environment vs. prod - but it is kind of hard. Eg. Prod will start to get issues with 2k users and test only needs 200 etc..., but this needs experience and knowledge about the infrastructure. \n\nYou can still do quantitativ performance testing, looking at how performance develops over time - if an api suddenly takes 2x the time in testing, someone should check. But as you see this can not help with all the performance related tests."
    },
    {
      "body": "If your QA environment doesn’t even have a load balancer and your production environment has a full budget of resources behind it, it will be a giant pain in the ass to get accurate results. I mean you might be able to find out *something* but in my opinion it’s really apples and oranges.\n\nAlso I would personally never run a performance test in production, because quite frankly it can have a direct impact on the business for a myriad of reasons."
    },
    {
      "body": "Nope, unless you feel like messing up the existing envs."
    },
    {
      "body": "Generally yes.  \nProduction environments can be expensive and not practical to manage, instead it's better to have an environment 1/xth the size of production and try to test it with 1/xth the production throughput. \n\nWhat's more challenging imo by not having a standalone dedicated environment is test result investigation. If you see a degradation o results, is it because of some concurrent manual testing? Or because you aren't resetting your data baselines for tests? All of that will make you doubt test results and likely doubt the value of the performance testing."
    },
    {
      "body": "It's absolutely still worth while.  However, both test setup and results interpretation will take much longer.  Start by understanding how different your non-production setup is from your production setup.  Looking at this, try to come up with a useful goal that will scale to your production usage.  During the test, make sure that you are monitoring logs and resources for absolutely every application and system that is touched by the test, right down to any auth or telemetry services.\n\nAfter this test is complete, start analyzing the results.  Are there areas that seem as though they need some further investigation?  Maybe run a small load test directly against that. Some soak testing might really help out in these areas.  Using this, you can start to get a better picture of it all.  Sometimes we need to test in isolation in order to see the whole system.\n\nIf you have customers complaining about this happening and degrading their non-prod environments, may be you need to look at either doing this off hours, or advise them that there will be a system degradation during a particular window.  I'm in a not too different situation, and do most of my performance testing between 2am and 6am for one of the systems that I'm responsible for.\n\nThe next part, some people might disagree with, and I certainly have at particular points in my career as well.  Are your tests destructive?  Will they create data that may impact financial or regulatory numbers that can't be marked separately in quarterly reporting?  If no, maybe you want to find an expected quiet period, alert your customers to an expected system degradation during that time, and execute your test in production.\n\n​\n\nHonestly though, if your manual testing is slowing down the system, I think that you might have bigger problems.  The above can still get you some useful information though."
    }
  ]
}