Right, it may have just changed the state of the system to mask the problem, or it may have just swept it under the rug. But you also may have just avoided a complex, long debugging session which would not result in reusable knowledge.
Debugging something that ultimately ends up being a 24-hour one-time glitch in AWS is a waste of time unless you learn something useful in the process or have sufficient reason to believe you altered the speed with which it was fixed, no matter how fun the journey. Sure, if it results in you deciding you need to be resilient under AWS failures more, that's great. But often times you just burned man hours and lost the opportunity to work on something else. I don't think it's reasonable to say that all debugging results in useful learning. (I'd probably argue the contrary, given the number of times I end up grokking incidental implementation details of some third party software or system during debugging that in just a few short months later becomes useless as that code or service is paved over.)
Trying to tilt the odds in your favor so you make educated guesses around when this will be the case is part of the skill. Part of this skill is also being able to recognize when a bug you thought had been routed around was simply swept under the rug, and changing course to understand it properly instead of sweeping it under the rug again. ("Fool me once, shame on you, fool me twice... you can't get fooled again.") I find these are the exception not the rule, and are probably worth the cost. The joys of shipping outweigh the joys of understanding every facet of a misbehaving system, at least for me.
> But you also may have just avoided a complex, long debugging session which would not result in reusable knowledge.
I find this is becoming increasingly rare for me, to the point where I'm having difficulty remembering the last time it occurred. I run into the same "once off" problems more frequently, and getting more value out of debugging them. I'm getting better at using my tools, better at tying symptoms to causes, better at coming up with appropriate defensive coding techniques, and writing more (reusable) code that makes them easier to debug when they (and entire categories of similar bugs) reoccur.
> Part of this skill is also being able to recognize when a bug you thought had been routed around was simply swept under the rug, and changing course to understand it properly instead of sweeping it under the rug again.
Frequently not possible on two fronts:
On recognizing it: I've seen a distressing number of bugs which are hideously context sensitive (meaning "works fine on my machine" is the punchline because it's 100% broken on another, only now you've got a terrible bug report and no good repro case)
On changing course: Any offline distribution channel. Any embedded system. Any onerously regulated environment. Once it's in the wild, it can become very expensive to fix.
> The joys of shipping outweigh the joys of understanding every facet of a misbehaving system, at least for me.
That's fair. I simply hope you avoid the sorrows of burned customers and data loss.
Debugging something that ultimately ends up being a 24-hour one-time glitch in AWS is a waste of time unless you learn something useful in the process or have sufficient reason to believe you altered the speed with which it was fixed, no matter how fun the journey. Sure, if it results in you deciding you need to be resilient under AWS failures more, that's great. But often times you just burned man hours and lost the opportunity to work on something else. I don't think it's reasonable to say that all debugging results in useful learning. (I'd probably argue the contrary, given the number of times I end up grokking incidental implementation details of some third party software or system during debugging that in just a few short months later becomes useless as that code or service is paved over.)
Trying to tilt the odds in your favor so you make educated guesses around when this will be the case is part of the skill. Part of this skill is also being able to recognize when a bug you thought had been routed around was simply swept under the rug, and changing course to understand it properly instead of sweeping it under the rug again. ("Fool me once, shame on you, fool me twice... you can't get fooled again.") I find these are the exception not the rule, and are probably worth the cost. The joys of shipping outweigh the joys of understanding every facet of a misbehaving system, at least for me.