This month, we as a team implemented the requirements as decided during sprint planning.
Actually, our project works on AGILE methodologies, where each release is scheduled after 4 weeks.
So in this lesser time, fulfill the client demand is quite tuff. Sometime it’s too hectic.
Anyways, as a standard process, before closing the release lifecycle, we execute the migration doc (this contains all the property file changes, script, the build number etc.),
on production clone environment, and if needed we go for LT (load testing).
During the current release, when QA team was about give sign-off on QA environment, we found issue on our one of the application.
This was deployed on production clone environment. User was not able to log-in the application. Even no errors were in log (what the heck? Eh!!!)
All developer started debug the problem and time was passing like anything (whenever we IT people get severe problem, each second passes, feels like an hour gone. Isn’t it?)
Suddenly, The GOD opened our door. We identified, that yesterday somebody updated files with missing isInfoEnabled statements.
“If that is so, if we modify the log level will resolve the problem for the time being”, one of the team member thought.
So, we followed our man of thought, modified the log level- from ERROR to DEBUG.
And, luckily we were able to log-in to application. Hurray!!
Next step was to identify such piece of code and we corrected.
But what we learned from this? What if it was caught on production?
Obviously, there could be escalations. Only having exact database and running migration doc properly may insure the correctness of release going on,
But what about the configuration of log file? It really matters. From that incident onwards, we made it standard to keep everything like production environment, before giving sign-off, especially on QA environment.
We can try our level best to make life easy but standardizing such processes, which can be learned from experience. Please share if you have faced any such issue, as sharing can spread smiles.