Creating an asynchronous writethrough (AWT) cache group automatically creates a replication scheme that allows the database to communicate with the Oracle database. You must start the replication agent after you create an AWT cache group and start the cache agent. See "Creating an AWT cache group" in the Oracle In-Memory Database Cache User's Guide.
Material in Chapter 6, "Troubleshooting Replication " is useful for troubleshooting AWT cache group problems. Useful replication topics are summarized in the current chapter in these sections:
This chapter also contains the following sections:
This section describes what to check if you are unable to start or stop a replication agent.
| Possible cause | What to do |
|---|---|
You do not have ADMIN privileges. |
You must have ADMIN privileges to use the ttAdmin utility or the ttRepStart or ttRepStop procedures to start or stop a replication agent. |
| TimesTen daemon not started | Check the state of the TimesTen daemon, as described in "Check the TimesTen user error log". If necessary, start the TimesTen daemon as described in "Working with the Oracle TimesTen Data Manager Daemon" in the Oracle TimesTen In-Memory Database Operations Guide. |
If you are unable to get replication working, the problem may be one or more of the following:
| Possible Cause | See... |
|---|---|
| TimesTen daemon or replication agents not running | "Check status of TimesTen daemon and replication agents" |
| Replication agents not communicating | "Check that replication agents are communicating" |
| Replication not in Start state | "Check replication state" |
TimesTen can send SNMP traps for certain replication events to enable network management software to take immediate action. TimesTen can send the following SNMP traps:
ttRepAgentExitingTrap
ttRepAgentDiedTrap
ttRepAgentStartingTrap
These traps are described in "Diagnostics through SNMP Traps" in the Oracle TimesTen In-Memory Database Error Messages and SNMP Traps.
You can monitor the performance of asynchronous writethrough (AWT) cache groups to determine how much time is spent performing tasks in the AWT workflow. Use the ttCacheAwtMonitorConfig built-in procedure to enable monitoring.
For example, enable monitoring and set the sampling frequency to 16. A sampling factor of 16 is recommended for accuracy and performance.
Command> CALL ttCacheAwtMonitorConfig('ON',16);
< ON, 16 >
1 row found.
Use the ttRepAdmin utility with the -awtmoninfo and -showstatus commands to display the monitoring results. The AWT monitoring statistics include:
TimesTen processing time: The total number of milliseconds spent in processing AWT transaction data since monitoring was enabled.
Oracle bookmark management time: The total number of milliseconds spent in managing AWT metadata on Oracle since monitoring was enabled.
Oracle execute time: The total number of milliseconds spent in OCI preparation, binding and execution for AWT SQL operations since monitoring was enabled. This statistic includes network latency between TimesTen and Oracle.
Oracle commit time: The total number of milliseconds spent in committing AWT updates on Oracle since monitoring was enabled. This statistic includes network latency between TimesTen and Oracle.
Time since monitoring was started
Total number of TimesTen row operations: The total number of rows updated in AWT cache groups since monitoring was enabled.
Total number of TimesTen transactions: The total number of transactions in AWT cache groups since monitoring was enabled.
Total number of flushes to Oracle: The total number of times that TimesTen data has been sent to Oracle.
The output also includes the percentage of time spent on TimesTen processing, Oracle bookmark management, Oracle execution and Oracle commits.
For example:
ttRepAdmin -showstatus -awtmoninfo myDSN
[other -showstatus output]
...
AWT Monitoring statistics
-------------------------
TimesTen processing time : 0.689000 millisecs (0.164307 %)
Oracle bookmark management time : 3.229000 millisecs (0.770027%)
Oracle execute time : 342.908000 millisecs (81.774043 %)
Oracle commit time : 72.450000 millisecs (17.277315 %)
Time since monitoring was started: 8528.641000 millisecs
Cache-connect Operational Stats :
Total Number of TimesTen row operations : 2
Total Number of TimesTen transactions : 2
Total Number of flushes to Oracle : 2
This section addresses issues that may degrade AWT performance.
| Possible cause | See... |
|---|---|
| Slow network | "Check network bandwidth" |
| Log buffer too small | "Check size of log buffer" |
| Frequent or inefficient disk writes | "Check durability settings" |
| Reading from transaction log files on disk instead of the log buffer | "Check for reads from transaction log files" |
Insert, update, or delete errors that occur while applying changes to Oracle are saved in an error file located in the database directory with the following name:
DatastoreName.awterr
Errors reported to this file are permanent errors. TimesTen does not retry the transaction. The errors may be reported in the AWT error file long after the commit to TimesTen occurs.
The format of the messages in the AWT error file is similar to those generated for conflict and transaction errors in replication, as shown in Example 5-1. Oracle error messages are also reported in the support log and the user log.
Example 5-1 Cache violation occurs when update is propagated to Oracle
If a constraint violation occurs when a cache group update is propagated to Oracle, the message in the AWT error file is similar to the following:
Error occurred 14:48:55 on 03-22-2007 Datastore: c:\temp\cgDSN Oracle Id: system1 Transmitting name: cgDSN Error message: TT5210: Oracle unique constraint violation error in OCIStmtExecute(): ORA-00001: unique constraint (GUSER.SYS_C00357240) violated rc = -1 -- file "bdbTblH.c", lineno 1205, procedure "ttBDbStmtForce()" TT5025: Commit failure in Oracle. Transaction must be rolled back in TimesTen. -- file "bdbConnect.c", lineno 885, procedure "ttBDbXact()" Operation that caused the error: Insert into table TESTUSER.T1 <9,1000> Failed transaction: Insert into table TESTUSER.T1 <9, 1000> End of failed transaction
Example 5-2 An object that TimesTen has placed on Oracle is dropped
If an object that TimesTen has placed on Oracle is dropped, the message in the AWT error file is similar to the following:
May 04 18:12:36 HOST1 TimesTen Replication 7.0[2136]: [Err ] DEFAULT:meta.c(639): TT16062: Failed to compile command: select p.commit_timestamp, p.commit_seqnum, p.protocol from owner1.TT_03_REPPEERS p where p.replication_name = :rname and p.replication_owner = :rowner and p.tt_store_id = :oid and p.subscriber_id = :sid May 04 18:12:36 HOST1 TimesTen Replication 7.0[2136]: [Err ] DEFAULT:meta.c(639): TT5221: TT5221: Oracle syntax error in OCIStmtExecute(): ORA-00942: table or view does not exist rc = -1 -- file "bdbStmt.c", lineno 1041, procedure "getOraOutTypesNLengths()"
In this example, the TT_03_REPPEERS table does not exist. To recover from this error, perform the following tasks:
Stop the replication agent.
Drop and re-create the cache group.
Restart the replication agent.
The support log for databases with AWT cache groups may contain Oracle errors if the replication agent encounters a problem on the Oracle database. If the replication agent encounters one of these errors, AWT rolls back the transaction and retries it. If the support log becomes full, the oldest messages are deleted and replaced by new messages.
The Oracle errors in the support log are considered transient because AWT retries the transaction.
Some transient errors indicate an underlying problem on the Oracle database must be solved before AWT operations can continue. For example:
ORA-01536: space quota exceeded for tablespace ORA-01034: ORACLE not available
After the underlying problem has been fixed, AWT retries the operation.
For more information about the Oracle errors, see Oracle Database Error Messages for the Oracle release you are using.
The following Oracle errors are transient:
ORA-00018: maximum number of sessions exceeded ORA-00019: maximum number of session licenses exceeded ORA-00020: maximum number of processes (%s) exceeded ORA-00025: failed to allocate %s ORA-00028: your session has been killed ORA-00038: Cannot create session: server group belongs to another user ORA-00051: timeout occurred while waiting for a resource ORA-00052: maximum number of enqueue resources (%s) exceeded ORA-00053: maximum number of enqueues exceeded ORA-00054: resource busy and acquire with NOWAIT specified ORA-00055: maximum number of DML locks exceeded ORA-00057: maximum number of temporary table locks exceeded ORA-00058: DB_BLOCK_SIZE must be %s to mount this database (not %s) ORA-00059: maximum number of DB_FILES exceeded ORA-00060: deadlock detected while waiting for resource ORA-00063: maximum number of LOG_FILES exceeded ORA-00064: object is too large to allocate on this O/S (%s,%s) ORA-00099: timed out while waiting for resource, potential PDML deadlock ORA-00104: deadlock detected; all public servers blocked waiting for resources ORA-00107: failed to connect to ORACLE listener process ORA-00115: connection refused; dispatcher connection table is full ORA-00125: connection refused; invalid presentation ORA-00126: connection refused; invalid duplicity ORA-00284: recovery session still in progress ORA-00370: potential deadlock during kcbchange operation ORA-00371: not enough shared pool memory ORA-00376: file %s cannot be read at this time ORA-00379: no free buffers available in buffer pool %s for block size %sK ORA-00384: Insufficient memory to grow cache ORA-00568: Maximum number of interrupt handlers exceeded ORA-00579: osndnt: server received malformed connection request ORA-00600: internal error code, arguments: [%s], [%s], [%s], [%s], [%s], [%s], [%s], [%s] ORA-00603: ORACLE server session terminated by fatal error ORA-01000: maximum open cursors exceeded ORA-01012: not logged on ORA-01014: ORACLE shutdown in progress ORA-01019: unable to allocate memory in the user side ORA-01031: insufficient privileges ORA-01033: ORACLE initialization or shutdown in progress ORA-01034: ORACLE not available ORA-01035: ORACLE only available to users with RESTRICTED SESSION privilege ORA-01037: maximum cursor memory exceeded ORA-01046: cannot acquire space to extend context area ORA-01073: fatal connection error: unrecognized call type ORA-01089: immediate shutdown in progress - no operations are permitted ORA-01090: shutdown in progress - connection is not permitted ORA-01092: ORACLE instance terminated. Disconnection forced ORA-01094: ALTER DATABASE CLOSE in progress. Connections not permitted ORA-01109: database not open ORA-01147: SYSTEM tablespace file %s is offline ORA-01154: database busy. Open, close, mount, and dismount not allowed now ORA-01155: the database is being opened, closed, mounted or dismounted ORA-01219: database not open: queries allowed on fixed tables/views only ORA-01237: cannot extend datafile %s ORA-01456: may not perform insert/delete/update operation inside a READ ONLY transaction ORA-01536: space quota exceeded for tablespace '%s' ORA-01539: tablespace '%s' is not online ORA-01542: tablespace '%s' is offline, cannot allocate space in it ORA-01562: failed to extend rollback segment number %s ORA-01573: shutting down instance, no further change allowed ORA-01628: max # extents (%s) reached for rollback segment %s ORA-01629: max # extents (%s) reached saving undo for tablespace %s ORA-01630: max # extents (%s) reached in temp segment in tablespace %s ORA-01631: max # extents (%s) reached in table %s.%s ORA-01632: max # extents (%s) reached in index %s.%s ORA-01650: unable to extend rollback segment %s by %s in tablespace %s ORA-01651: unable to extend save undo segment by %s for tablespace %s ORA-01652: unable to extend temp segment by %s in tablespace %s ORA-01653: unable to extend table %s.%s by %s in tablespace %s ORA-01654: unable to extend index %s.%s by %s in tablespace %s ORA-01655: unable to extend cluster %s.%s by %s in tablespace %s ORA-01656: max # extents (%s) reached in cluster %s.%s ORA-01658: unable to create INITIAL extent for segment in tablespace %s ORA-01659: unable to allocate MINEXTENTS beyond %s in tablespace %s ORA-01680: unable to extend LOB segment by %s in tablespace %s ORA-01681: max # extents (%s) reached in LOB segment in tablespace %s ORA-01683: unable to extend index %s.%s partition %s by %s in tablespace %s ORA-01684: max # extents (%s) reached in table %s.%s partition %s ORA-01685: max # extents (%s) reached in index %s.%s partition %s ORA-01686: max # files (%s) reached for the tablespace %s ORA-01688: unable to extend table %s.%s partition %s by %s in tablespace %s ORA-01691: unable to extend lob segment %s.%s by %s in tablespace %s ORA-01692: unable to extend lob segment %s.%s partition %s by %s in tablespace %s ORA-01693: max # extents (%s) reached in lob segment %s.%s ORA-01694: max # extents (%s) reached in lob segment %s.%s partition %s ORA-03113: end-of-file on communication channel ORA-03114: not connected to ORACLE ORA-03134: Connections to this server version are no longer supported. ORA-03135: connection lost contact ORA-03136: inbound connection timed out ORA-03232: unable to allocate an extent of %s blocks from tablespace %s ORA-03233: unable to extend table %s.%s subpartition %s by %s in tablespace %s ORA-03234: unable to extend index %s.%s subpartition %s by %s in tablespace %s ORA-03235: max # extents (%s) reached in table %s.%s subpartition %s ORA-03236: max # extents (%s) reached in index %s.%s subpartition %s ORA-03237: Initial Extent of specified size cannot be allocated ORA-03238: unable to extend LOB segment %s.%s subpartition %s by %s in tablespace %s ORA-03239: maxextents (%s) reached in LOB segment %s.%s subpartition %s ORA-04020: deadlock detected while trying to lock object %s%s%s%s%s ORA-06019: NETASY: invalid login (connect) string ORA-06021: NETASY: connect failed ORA-06030: NETDNT: connect failed, unrecognized node name ORA-06031: NETDNT: connect failed, unrecognized object name ORA-06032: NETDNT: connect failed, access control data rejected ORA-06033: NETDNT: connect failed, partner rejected connection ORA-06034: NETDNT: connect failed, partner exited unexpectedly ORA-06035: NETDNT: connect failed, insufficient resources ORA-06036: NETDNT: connect failed, no response from object ORA-06037: NETDNT: connect failed, node unreachable ORA-06039: NETDNT: connect failed ORA-06040: NETDNT: invalid login (connect) string ORA-06108: NETTCP: connect to host failed ORA-06113: NETTCP: Too many connections ORA-06114: NETTCP: SID lookup failure ORA-06143: NETTCP: maximum connections exceeded ORA-06315: IPA: Invalid connect string ORA-06316: IPA: Invalid database SID ORA-06317: IPA: Local maximum number of users exceeded ORA-06318: IPA: Local maximum number of connections exceeded ORA-06319: IPA: Remote maximum number of users exceeded ORA-06320: IPA: Remote maximum number of connections exceeded ORA-06404: NETCMN: invalid login (connect) string ORA-06413: Connection not open. ORA-10435: enable tracing of global enqueue service deadlock detetction ORA-10626: specify timeout for online index rebuild to wait for DML ORA-10906: Unable to extend segment after insert direct load ORA-12150: TNS:unable to send data ORA-12151: TNS:received bad packet type from network layer ORA-12152: TNS:unable to send break message ORA-12153: TNS:not connected ORA-12154: TNS:could not resolve service name ORA-12155: TNS:received bad datatype in NSWMARKER packet ORA-12156: TNS:tried to reset line from incorrect state ORA-12157: TNS:internal network communication error ORA-12158: TNS:could not initialize parameter subsystem ORA-12159: TNS:trace file not writeable ORA-12160: TNS:internal error: Bad error number ORA-12161: TNS:internal error: partial data received ORA-12162: TNS:service name is incorrectly specified ORA-12163: TNS:connect descriptor is too long ORA-12166: TNS:Client can not connect to HO agent. ORA-12168: TNS:Unable to contact Directory Server. ORA-12169: TNS:Net service name given as connect identifier is too long ORA-12170: TNS:Connect timeout occurred ORA-12171: TNS:could not resolve connect identifier: %s ORA-12196: TNS:received an error from TNS ORA-12197: TNS:keyword-value resolution error ORA-12198: TNS:could not find path to destination ORA-12200: TNS:could not allocate memory ORA-12201: TNS:encountered too small a connection buffer ORA-12202: TNS:internal navigation error ORA-12203: TNS:unable to connect to destination ORA-12204: TNS:received data refused from an application ORA-12205: TNS:could not get failed addresses ORA-12206: TNS:received a TNS error during navigation ORA-12207: TNS:unable to perform navigation ORA-12208: TNS:could not find the TNSNAV.ORA file ORA-12209: TNS:encountered uninitialized global ORA-12210: TNS:error in finding Navigator data ORA-12211: TNS:needs PREFERRED_CMANAGERS entry in TNSNAV.ORA ORA-12212: TNS:incomplete PREFERRED_CMANAGERS binding in TNSNAV.ORA ORA-12213: TNS:incomplete PREFERRED_CMANAGERS binding in TNSNAV.ORA ORA-12214: TNS:missing local communities entry in TNSNAV.ORA ORA-12215: TNS:poorly formed PREFERRED_NAVIGATORS Addresses in TNSNAV.ORA ORA-12216: TNS:poorly formed PREFERRED_CMANAGERS addresses in TNSNAV.ORA ORA-12217: TNS:could not contact PREFERRED_CMANAGERS in TNSNAV.ORA ORA-12218: TNS:unacceptable network configuration data ORA-12219: TNS:missing community name from address in ADDRESS_LIST ORA-12221: TNS:illegal ADDRESS parameters ORA-12222: TNS:no such protocol adapter ORA-12223: TNS:internal limit restriction exceeded ORA-12224: TNS:no listener ORA-12225: TNS:destination host unreachable ORA-12226: TNS:operating system resource quota exceeded ORA-12227: TNS:syntax error ORA-12228: TNS:protocol adapter not loadable ORA-12229: TNS:Interchange has no more free connections ORA-12230: TNS:Severe Network error ocurred in making this connection ORA-12231: TNS:No connection possible to destination ORA-12232: TNS:No path available to destination ORA-12233: TNS:Failure to accept a connection ORA-12235: TNS:Failure to redirect to destination ORA-12236: TNS:protocol adapter not loaded ORA-12316: syntax error in database link's connect string ORA-12326: database %s is closing immediately; no operations are permitted ORA-12329: database %s is closed; no operations are permitted ORA-12500: TNS:listener failed to start a dedicated server process ORA-12501: TNS:listener failed to spawn process ORA-12502: TNS:listener received no CONNECT_DATA from client ORA-12504: TNS:listener was not given the SID in CONNECT_DATA ORA-12505: TNS:listener could not resolve SID given in connect descriptor ORA-12506: TNS:listener was not given the ALIAS in CONNECT_DATA ORA-12507: TNS:listener could not resolve ALIAS given ORA-12508: TNS:listener could not resolve the COMMAND given ORA-12509: TNS:listener failed to redirect client to service handler ORA-12510: TNS:database temporarily lacks resources to handle the request ORA-12511: TNS:service handler found but it is not accepting connections ORA-12512: TNS:service handler found but it has not registered a redirect address ORA-12513: TNS:service handler found but it has registered for a different protocol ORA-12514: TNS:listener could not resolve SERVICE_NAME given in connect descriptor ORA-12515: TNS:listener could not find a handler for this presentation ORA-12516: TNS:listener could not find available handler with matching protocol stack ORA-12517: TNS:listener could not find service handler supporting direct handoff ORA-12518: TNS:listener could not hand off client connection ORA-12519: TNS:no appropriate service handler found ORA-12520: TNS:listener could not find available handler for requested type of server ORA-12521: TNS:listener could not resolve INSTANCE_NAME given in connect descriptor ORA-12522: TNS:listener could not find available instance with given INSTANCE_ROLE ORA-12523: TNS:listener could not find instance appropriate for the client connection ORA-12524: TNS:listener could not resolve HANDLER_NAME given in connect descriptor ORA-12525: TNS:listener has not received client's request in time allowed ORA-12526: TNS:listener: all appropriate instances are in restricted mode ORA-12527: TNS:listener: all instances are in restricted mode or blocking new connections ORA-12528: TNS:listener: all appropriate instances are blocking new connections ORA-12529: TNS:connect request rejected based on current filtering rules ORA-12531: TNS:cannot allocate memory ORA-12532: TNS:invalid argument ORA-12533: TNS:illegal ADDRESS parameters ORA-12534: TNS:operation not supported ORA-12535: TNS:operation timed out ORA-12536: TNS:operation would block ORA-12537: TNS:connection closed ORA-12538: TNS:no such protocol adapter ORA-12539: TNS:buffer over- or under-flow ORA-12540: TNS:internal limit restriction exceeded ORA-12541: TNS:no listener ORA-12542: TNS:address already in use ORA-12543: TNS:destination host unreachable ORA-12544: TNS:contexts have different wait/test functions ORA-12545: Connect failed because target host or object does not exist ORA-12546: TNS:permission denied ORA-12547: TNS:lost contact ORA-12549: TNS:operating system resource quota exceeded ORA-12550: TNS:syntax error ORA-12551: TNS:missing keyword ORA-12552: TNS:operation was interrupted ORA-12554: TNS:current operation is still in progress ORA-12555: TNS:permission denied ORA-12556: TNS:no caller ORA-12557: TNS:protocol adapter not loadable ORA-12558: TNS:protocol adapter not loaded ORA-12560: TNS:protocol adapter error ORA-12561: TNS:unknown error ORA-12562: TNS:bad global handle ORA-12564: TNS:connection refused ORA-12566: TNS:protocol error ORA-12569: TNS:packet checksum failure ORA-12570: TNS:packet reader failure ORA-12571: TNS:packet writer failure ORA-12574: TNS:redirection denied ORA-12582: TNS:invalid operation ORA-12583: TNS:no reader ORA-12585: TNS:data truncation ORA-12589: TNS:connection not bequeathable ORA-12590: TNS:no I/O buffer ORA-12591: TNS:event signal failure ORA-12592: TNS:bad packet ORA-12593: TNS:no registered connection ORA-12595: TNS:no confirmation ORA-12596: TNS:internal inconsistency ORA-12600: TNS: string open failed ORA-12602: TNS: Connection Pooling limit reached ORA-12606: TNS: Application timeout occurred ORA-12607: TNS: Connect timeout occurred ORA-12608: TNS: Send timeout occurred ORA-12609: TNS: Receive timeout occurred ORA-12612: TNS:connection is busy ORA-12615: TNS:preempt error ORA-12623: TNS:operation is illegal in this state ORA-12624: TNS:connection is already registered ORA-12636: Packet send failed ORA-12637: Packet receive failed ORA-12829: Deadlock - itls occupied by siblings at block %s of file %s ORA-12993: tablespace '%s' is offline, cannot drop column ORA-14117: partition resides in offlined tablespace ORA-14268: subpartition '%s' of the partition resides in offlined tablespace ORA-16000: database open for read-only access ORA-16003: standby database is restricted to read-only access ORA-16403: shutdown in progress - remote connection is not permitted ORA-16724: the intended state for resource has been set to OFFLINE ORA-16903: Unable to connect to database ORA-16914: Missing connect string. Try \"help\" ORA-18014: deadlock detected while waiting for resource %s ORA-21521: exceeded maximum number of connections in OCI (object mode only) ORA-21522: attempted to use an invalid connection in OCI (object mode only) ORA-23317: a communication failure has occurred ORA-24401: cannot open further connections ORA-24418: Cannot open further sessions. ORA-24778: cannot open connections ORA-25400: must replay fetch ORA-25401: can not continue fetches ORA-25402: transaction must roll back ORA-25403: could not reconnect ORA-25405: transaction status unknown ORA-25407: connection terminated ORA-25408: can not safely replay call ORA-25409: failover happened during the network operation,cannot continue ORA-25425: connection lost during rollback ORA-29306: datafile %s is not online ORA-30006: resource busy; acquire with WAIT timeout expired ORA-30036: unable to extend segment by %s in undo tablespace '%s' ORA-30040: Undo tablespace is offline ORA-31443: deadlock detected while acquiring lock on %s ORA-37013: (XSACQUIRE_DEADLOCK) Cannot wait to acquire object %j since doing so would cause a deadlock. ORA-44317: database open read-only