Problem Solving Approach: Production and Non Prod behavior is different

In this article:

  • You will able to answer how to debug issue when production code behavior is not as expected as in non-production.
  • You will able to learn some basic causes due to that same code base behave differently in production and non-production.
  • You will get idea how to approach for this kind of scenario.
  • This will help to resolve production issues in quick time to avoid SLA.

newidea

Scenario:

Some functionality which is working as expected in non-production environment, but does not work in production.

The background:

Some time we come to those situations when production code behavior on some functionality (which was part of current release) is not as expected. QA already gave sign off, UAT also performed their testing, RTP team followed proper migration document. But still that functionality does not work.  What got wrong? In production we can’t enable debug info to check everything. Then how to resolve the issue? How to make production code working as expected? Following may be the solution which are truly based on real time experiences.

Explore this:

Below are the sequential steps we usually follow. At any point of time when gap found stop the analysis and correct the migration document, follow the rest steps to make sure there is no more gap and go for release.

1.       Verify WAR file version

  • Check the WAR file version in production and non-prod where sign off was given.
  • Check the code base. If it is J2EE, verify if the java class is up-to-date in version control system.
  • Check if the above changes are in WAR file.

2.       Missing Property file/Database changes

  • Based on the project management, check the tracking document which contains all the changes required to implement particular changes.
  •  It may be design document or kind of solution approach document.

3.       Enable same level log in Non-prod

  • Based on observation found, sometime few developers write some part of code in debug/info logger level by mistake. And that may also cause this situation where same code base is working fine on non-prod but not in production.
  • To identify this, set the logger level to same level as in Production.
  • Usually Error level is enabled in production and in non-prod, it is debug.
  • Read this article for more info.

4.       Hit Node URL-

  • Production code base usually deployed on more than one server or node (may be same server with different JVM as well). So rather hitting direct URL (like google.com), hit node specific URL (like nodeid:8080).
  • If any node URL gives desired result that means there is some problem with server and/or deployment process. May be temp directory did not get cleared in case of web logic server. Clear the temp directory and restart the application.
  • Check whether startup parameter is in sync.

5.       Stop external scripting on specific page

  • To get the user experience and/or to make changes in site look and feel using campaigns, companies are using external scripts like Adobe TestNTarget etc.
  • These script runs on our application core data returned. Hence these can be also responsible for the behavior of production code base.
  • To make sure if external scripts are creating problem, just stop on specific pages where desired functionality is not in working in production.

What Next:

  • Just apply above tips; these are based on personal experiences. May be you came across different scenario. Then please share it.
  • In next article planning to share real-time tools to resolve production issue.

Till then Happy Reading,

Nirbhaya Bhava!!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s