In such a scenario, it may be only a matter of time before enough backup commands are run to cause the SCN to eclipse the soft limit -- at which point every interlinked Oracle database server suffers some significant problem, refuses connections from other servers, or simply crashes. (Oracle notes, correctly, that the SCN cannot "eclipse" the soft limit. An escalating SCN can rise as far as the limit and no further. If an Oracle database reaches this maximum, it will abort transactions or shut down.)
Oracle released a patch to fix the arbitrary SCN growth rate bug in the hot backup code before InfoWorld began researching this story. The backup bug is listed as 12371955: "High SCN growth rate from ALTER DATABASE BEGIN BACKUP in 11g." If you have not already done so, Oracle recommends that you install this patch immediately. (The hot backup bug is confined to Oracle Database 11g releases 11.1.0.7 and 11.2.0.2.)
The saboteur
But the risk of incrementing the SCN via the backup bug is not the only cause for concern. Perhaps the most important part of our finding is that the SCN can be incremented by anyone who can issue commands on an interconnected database.
Say there's a low-privilege reporting database system that has a read-only database link to a high-privilege database somewhere down the line. All someone has to do is issue a few administrative commands on that low-privilege database to raise the SCN value by, for example, 1 trillion; the next time the high-privilege database receives a connection from that box, its SCN increases to the higher level. It's simple enough to verify the current SCN value, so running a command to push it a few ticks shy of the soft limit is trivial.
Using this technique, a bad actor could potentially cause a systemwide Oracle database communications failure, a shutdown, or a crash with only a few commands on a sideline server -- or even craft a bit of code to mimic a connecting instance. It may be possible, though more challenging, to cause the same problems though a SQL injection attack on a vulnerable application.
Our discovery of this attack method and our communication of this information to Oracle resulted in Oracle's request that InfoWorld hold this story until today, when a patch could be made available. In addition to the method we discovered, we learned of two other, similar means of incrementing the SCN by searching the open Web, although these required higher privilege levels. The patch blocks all three methods that we are aware of, though there may be others.
The reality
The SCN is a moving line that cannot be crossed. The line moves up by 16,384 every second; as long as the SCN growth rate is slower, all should be well. (Note: Oracle has provided a script that allows customers to identify which databases are at risk. The script is referenced in support document 1393363.1.)
But what happens when your SCN moves closer and closer to that line due to the spurious jumps caused by the backup bug, a simple mistake by an admin, or other means? How do you deal with this impending problem?
The answer: Shut down the database servers for a while so that the number stops incrementing.