If you are like me you may well have some clone apps databases as a development or training environment. If you haven't I suggest you read Steven Chan on the subject here. As well as cloning our development databases we make a habit of turning on flashback database so that in the event that a developer messes up development or needs to revert to an older state then we can do quick point in time recovery.

When I came in this morning I noticed that our recently cloned training environment was down. having ascertained that in fact the clone had been completed 'successfully' last thursday I started to investigate. Starting the database manually gave me this interesting error.

comp-dev-db-1:ora.TRAIN.TRAIN1.inst:Done removing ipc resources for sid "TRAIN1"
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:SQL*Plus: Release 10.1.0.4.0 - Production on Mon Jan 14 09:46:57 2008
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:Copyright (c) 1982, 2005, Oracle.  All rights reserved.
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:Enter user-name: Connected to an idle instance.
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:SQL> ORACLE instance started.
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:Total System Global Area 1073741824 bytes
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:Fixed Size               782648 bytes
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:Variable Size                 265555656 bytes
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:Database Buffers      805306368 bytes
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:Redo Buffers                    2097152 bytes
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:Database mounted.
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:ORA-38760: This database instance failed to turn on flashback database
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:
comp-dev-db-1:ora.TRAIN.TRAIN1.inst:

That is a new error to me so I started digging around in the alert.log. There I found some interesting ORA-600 errors as follows.

Errors in file $ORACLE_HOME/admin/$CONTEXT_NAME/bdump/train1_smon_32219.trc:
ORA-00600: internal error code, arguments: [15709], [29], [1], [], [], [], [], []
ORA-38701: Flashback database log 96 seq 12 thread 2: "/<flashback location>/flashback/o1_mf_3rfh3tkd_.flb"
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory

It seems what had happened was that after the cleanup the DBA who had carried out the cloning recalled that he hadn't deleted the old flashback logs and so he deleted all the files in our flashback location. Oracle crashed at this point and would not restart. There are similar notes on metalink regarding this same behaviour if the flashback log becomes corrupt. Lessons learned

1. Make sure that you complete all your cloning activities in the correct order before handing a clone over to users.
2. Corrupt flashback logs are a threat to database availability - something I wouldn't have expected.

For starting the blogtag game which as Howard points out here is a pyramid scheme which is illegal in my country and elsewhere. This one however doesn't promise money but an overwhelming amount of useless trivia about people you don't know. As such it seems to me both an excellent bit of web 2.0 and harmless entertainment. Besides Lisa tagged me and she throws a mean glass of water. So 8 things you didn't know or want to about me.

1.  I'm married to a wonderful lady called Laura - she's a professional pedant, sorry patent attorney - given that I'm a dba we argue about the precise nature of words we are using when arguing, ever wanted to know the difference between constantly,continuously and continually - come to our house it's a blast.
2. I attend this church here. If you are ever in Basingstoke, and who would be voluntarily, come along we don't bite - much.
3. I started professional life as a bean-counter for KPMG. Consequently I'm a DBA who respects auditors, not least because they have to stand in unheated warehouses in the middle of winter watching other people count things- I know I was there.  
4. I have also extensively used microsoft technologies which makes me an Oracle-head who likes both Windows and SQL server. 
5. My grandfather played rugby in the early years of the 20th century for the Lansdowne Road rugby club 
6. Despite this I don't like rugby at all, much preferring 5 days of rain followed by no result.
7. Hearing the sound of automatic weapons fire, whilst pretty rare in the UK, is a regular part of life for me as I live no more than about 1/2 a mile from an area regular used by the SAS reserves for live weapons training.
8. I once ate a meal with 4 separate fish courses in Oslo - right here in fact. I thoroughly recommend it.  
9. I can't count!

Anyway  that's a bit of me. Now lets see if we can't poke Alex, Hans, Richard, Noons, Shrek and see what happens. - See I told you I couldn't count.