27 Application Continuity for Java

The outages of the underlying software, hardware, communications, and storage layers can cause application execution to fail. In the worst cases, the middle-tier servers may need to be restarted to deal with the logon stormsFoot 1 . To overcome such problems, Oracle Database 12c Release 1 (12.1) introduces the Application Continuity feature that masks database outages to the application and end users are not exposed to such outages.

Note:

Application Continuity is a feature of the Oracle JDBC Thin driver and is not supported by JDBC OCI driver.

Application Continuity provides a general purpose, application-independent solution that enables recovery of work from an application perspective, after the occurrence of a planned or unplanned outage. The outage can be related to system, communication, or hardware following a repair, a configuration change, or a patch application.

See Also:

This chapter discusses the JDBC aspect of Application Continuity in the following sections:

Configuring Oracle JDBC for Application Continuity for Java

You must use either the oracle.jdbc.replay.OracleDataSourceImpl or oracle.jdbc.replay.OracleConnectionPoolDataSourceImpl data source to obtain JDBC connections. They are new Oracle JDBC data sources, and work similarly to the existing non-XA data sources, such as oracle.jdbc.pool.OracleDataSource. You can use both in a standalone manner, or configure them as connection factories for a connection pool, such as Universal Connection Pool (UCP), or Oracle WebLogic Server connection pool.

The following code snippet illustrates their usage in a standalone JDBC application:

import java.sql.Connection;
import javax.sql.PooledConnection;
import oracle.jdbc.OracleConnection;
import oracle.jdbc.replay.OracleDataSourceFactory;
import oracle.jdbc.replay.OracleDataSource;
import oracle.jdbc.replay.OracleConnectionPoolDataSource;
 
...
{
......
OracleDataSource rds = OracleDataSourceFactory.getOracleDataSource();
rds.setUser(user);
rds.setPassword(passwd);
rds.setURL(url);
......  // Other data source configuration like callback, timeouts, etc.
 
Connection conn = rds.getConnection();
((OracleConnection) conn).beginRequest();  // Explicit request begin
......  // JDBC calls protected by Application Continuity
((OracleConnection) conn).endRequest();  // Explicit request end
conn.close();
 
OracleConnectionPoolDataSource rcpds = OracleDataSourceFactory.getOracleConnectionPoolDataSource();
rcpds.setUser(user);
rcpds.setPassword(passwd);
rcpds.setURL(url);
......  // other data source configuration like callback, timeouts, and so on
 
PooledConnection pc = rcpds.getPooledConnection();
Connection conn2 = pc.getConnection();  // Implicit request begin
......  // JDBC calls protected by Application Continuity
conn2.close();  // Implicit request end
......

See Also:

"Data Sources and URLs" for more information about Oracle JDBC data sources

You must remember the following points while using the connection URL:

  • Always use the thin driver in the connection URL.

  • Always connect to a service. Never use instance_name or SID because these do not direct to known good instances and SID is deprecated.

  • If the addresses in the ADDRESS_LIST at the client does not match the REMOTE_LISTENER setting for the database, then it does not connect showing services cannot be found. So, the addresses in the ADDRESS_LIST at the client must match the REMOTE_LISTENER setting for the database:

    • If REMOTE_LISTENER is set to the SCAN_VIP, then the ADDRESS_LIST uses SCAN_VIP

    • If REMOTE_LISTENER is set to the host VIPs, then the ADDRESS_LIST uses the same host VIPs

    • If REMOTE_LISTENER is set to both SCAN_VIP and host VIPs, then the ADDRESS_LIST uses SCAN_VIP and the same host VIPs

      Note:

      For Oracle clients prior to release 11.2, the ADDRESS_LIST must be upgraded to use SCAN, which means expanding the ADDRESS_LIST to three ADDRESS entries corresponding to the three SCAN IP addresses.

      If such clients connect to a database that is upgraded from an earlier release through Database Upgrade Assistant, then you must retain the ADDRESS_LIST of these clients set to the HOST VIPs. However, if REMOTE_LISTENER is changed to ONLY SCAN, or the clients are moved to a newly installed Oracle Database 12c Release 1, where REMOTE_LISTENER is ONLY SCAN, then they do not get a complete service map, and may not always be able to connect.

  • Set RETRY_COUNT, CONNECT_TIMEOUT, and TRANSPORT_CONNECT_TIMEOUT parameters in the connection string. This is a general recommendation for configuring the JDBC thin driver connections, starting from Oracle Database Release 11.2.0.2. These settings improve acquiring new connections at runtime, at replay, and during work drains for planned outages.

    The CONNECT_TIMEOUT parameter is equivalent to the SQLNET.OUTBOUND_CONNECT_TIMEOUT parameter in the sqlnet.ora file and applies to the full connection. The TRANSPORT_CONNECT_TIMEOUT parameter applies as per the ADDRESS parameter. If the service is not registered for a failover or restart, then retrying is important when you use SCAN. For example, for using remote listeners pointing to SCAN addresses, you should use the following settings:

    jdbc:oracle:thin:@(DESCRIPTION = 
     (TRANSPORT_CONNECT_TIMEOUT=3)
     (RETRY_COUNT=20)(FAILOVER=ON)
     (ADDRESS_LIST =(ADDRESS=(PROTOCOL=tcp)
      (HOST=CLOUD-SCANVIP.example.com)(PORT=5221)) 
     (CONNECT_DATA=(SERVICE_NAME=orcl))) 
     
    REMOTE_LISTENERS=CLOUD-SCANVIP.example.com:5221 
    

    Similarly, for using remote listeners pointing to VIPs at the database, you should use the following settings:

    jdbc:oracle:thin:@(DESCRIPTION = 
    (TRANSPORT_CONNECT_TIMEOUT=3)
    (CONNECT_TIMEOUT=60)(RETRY_COUNT=20)(FAILOVER=ON)
     (ADDRESS_LIST=
     (ADDRESS=(PROTOCOL=tcp)(HOST=CLOUD-VIP1.example.com)(PORT=5221) )
    (ADDRESS=(PROTOCOL=tcp)(HOST=CLOUD-VIP2.example.com)(PORT=5221) )
     (ADDRESS=(PROTOCOL=tcp)(HOST=CLOUD-VIP3.example.com)(PORT=5221) )) 
    (CONNECT_DATA=(SERVICE_NAME=orcl)))
    
    REMOTE_LISTENERS=CLOUD-VIP1.example.com:5221 
    

Configuring Oracle Database for Application Continuity for Java

You must have the following configuration for Oracle Database to use Application Continuity for Java:

  • Use Oracle Database 12c Release 1 (12.1)

  • If you are using Oracle Real Application Clusters (Oracle RAC) or Oracle Data Guard, then ensure that FAN is configured with Oracle Notification System (ONS) to communicate with Oracle WebLogic Server or the Universal Connection Pool (UCP)

  • Use an application service for all database work. To create the service you must:

    • Run the SRVCTL command if you are using Oracle RAC

    • Use the DBMS_SERVICE package if you are not using Oracle RAC

  • Set the required properties on the service for replay and load balancing. For example, set:

    • FAILOVER_TYPE = TRANSACTION for using Application Continuity

    • COMMIT_OUTCOME = TRUE for enabling Transaction Guard

    • REPLAY_INITIATION_TIMEOUT = 900 for setting the duration in seconds for which replay will occur

    • FAILOVER_RETRIES = 30 for specifying the number of connection retries for each replay

    • FAILOVER_DELAY = 10 for specifying the delay in seconds between connection retries

    • GOAL = SERVICE_TIME, if you are using Oracle RAC, then this is a recommended setting

    • CLB_GOAL = LONG, if you are using Oracle RAC, then this is a recommended setting

  • Do not use the database service, that is, the default service corresponding to the DB_NAME or DB_UNIQUE_NAME. This service is reserved for Oracle Enterprise Manager and for DBAs. Oracle does not recommend the use of the database service for high availability because this service cannot be:

    • Enabled and disabled

    • Relocated on Oracle RAC

    • Switched over to Oracle Data Guard

See Also:

Oracle Database Development Guide for more information on the operation and usage of Application Continuity.

Identifying Request Boundaries in Application Continuity for Java

A Request is a unit of work on a physical connection to Oracle Database that is protected by Application Continuity. Request demarcation varies with specific use-case scenarios. A request begins when a connection is borrowed from the Universal Connection Pool (UCP) or WebLogic Server connection pool, and ends when this connection is returned to the connection pool.

Note:

You cannot borrow a connection from the Database Resident Connection Pool (DRCP) because DRCP does not work with Application Continuity. Refer to Chapter 23, "Database Resident Connection Pooling" for more information about DRCP.

The JDBC driver provides explicit request boundary declaration APIs beginRequest and endRequest in the oracle.jdbc.OracleConnection interface. These APIs enable applications, frameworks, and connection pools to indicate to the JDBC Replay Driver about demarcation points, where it is safe to release the call history, and to enable replay if it had been disabled by a prior request. At the end of the request, the JDBC Replay Driver purges the recorded history on the connection, where the API is called. This helps to further conserve memory consumption for applications that use the same connections for an extended period of time without returning them to the pool.

You do not need to make any changes to your application for identifying request boundaries, if the application uses connections from these Oracle connection pools, or from the oracle.jdbc.replay.OracleConnectionPoolDataSourceImpl data source. However, for the connection pool to work, the application must get connections when needed, and release connections when not in use. This scales better and provides request boundaries transparently.

Applications that do not borrow and return connections from the Oracle connection pools should explicitly mark request boundaries. For example, if your application is using custom JDBC pools, then the beginRequest method should be called at check-out and the endRequest method should be called at check-in. These methods can also be used for standalone JDBC applications without a connection pool.

The APIs have no impact on the applications other than improving resource consumption, recovery, and load balancing performance. These APIs do not involve altering a connection state by calling any JDBC method, SQL, or PL/SQL. An error is returned if an attempt is made to begin or end a request while a local transaction is open.

Registering a Connection Initialization Callback in Application Continuity for Java (optional)

Non-transactional session state (NTSS) is state of a database session that exists outside database transactions and is not protected by recovery. For applications that use stateful requests, the non-transactional state is re-established as the rebuilt session.

For applications that set state only at the beginning of a request, or for stateful applications that gain performance benefits from using connections with a preset state, one among the following callback options are provided:

No Callback

In this scenario, the application builds up its own state during each request.

Connection Labeling

This scenario is applicable only to Universal Connection Pool (UCP) and Oracle WebLogic server. The application can be modified to take advantage of the preset state on connections. Connection Labeling APIs determine how well a connection matches, and use a callback to populate the gap when a connection is borrowed. All applications cannot use Connection Labeling because it requires re-coding to some extent.

Connection Initialization Callback

In this scenario, the replay driver uses an application callback to set the initial state of the session during runtime and replay. The JDBC replay driver provides an optional connection initialization callback interface as well as methods for registering and unregistering such callbacks.

When registered, the initialization callback is executed at each successful reconnection following a recoverable error. An application is responsible for ensuring that the initialization actions are the same as that on the original connection before failover. If the callback invocation fails, then replay is disabled on that connection.

This section discusses initialization callbacks in the following sections:

Creating an Initialization Callback

To create a JDBC connection initialization callback, an application implements the oracle.jdbc.replay.ConnectionInitializationCallback interface. One callback is allowed for every instance of the oracle.jdbc.replay.OracleDataSource interface.

Note:

This callback is only invoked during failover, after a successful reconnection.

Example

The following example demonstrates a simple initialization callback implementation:

import oracle.jdbc.replay.ConnectionInitializationCallback;
class MyConnectionInitializationCallback implements ConnectionInitializationCallback
{
    public MyConnectionInitializationCallback()  
    {
        ...
    }
    public void initialize(java.sql.Connection connection) throws SQLException
    {
        // Reset the state for the connection, if necessary (like ALTER SESSION)
        ...
    }
 }

Registering an Initialization Callback

The JDBC Replay Driver provides the registerConnectionInitializationCallback(ConnectionInitializationCallback cbk) method in the oracle.jdbc.replay.OracleDataSource interface for registering a connection initialization callback. One callback is allowed for every instance of the OracleDataSource interface.

Removing or Unregistering an Initialization Callback

The JDBC Replay Driver provides the unregisterConnectionInitializationCallback(ConnectionInitializationCallback cbk) method in the oracle.jdbc.replay.OracleDataSource interface for unregistering a connection initialization callback.

Delaying the Reconnection in Application Continuity for Java

By default, when JDBC Replay Driver initiates a failover, the driver attempts to recover the in-flight work at an instance where the service is available. For doing this, the driver must first reestablish a good connection to a working instance. This reconnection can take some time if the database or the instance needs to be restarted before the service is relocated and published. So, the failover should be delayed until the service is available from another instance or database.

You need to use the FAILOVER_RETRIES and FAILOVER_DELAY parameters to manage reconnecting. These parameters can work well in conjunction with a planned outage, for example, an outage that may make a service unavailable for several minutes. While setting the FAILOVER_DELAY and FAILOVER_RETRIES parameters, check the value of the REPLAY_INITIAITION_TIMEOUT parameter first. The default value for this parameter is 900 seconds. A high value for the FAILOVER_DELAY parameter can cause replay to be canceled.

Parameter Name Possible Value Default Value
FAILOVER_RETRIES Positive integer zero or above 30
FAILOVER_DELAY Time in seconds 10

Examples

This section provides configuration examples for service creation and modification in the following subsections:

Creating Services on Oracle RAC

If you are using Oracle RAC or Oracle RAC One, then use the SRVCTL command to create and modify services in the following way:

For Policy-Managed Environments

srvctl add service -d codedb -s GOLD -g Srvpool -j SHORT -B SERVICE_TIME –z 30 –w 10 -commit_outcome
 TRUE -e TRANSACTION -replay_init_time 1800 -retention 86400 -notification TRUE

For Administrator-Managed Environments

srvctl add service -d codedb -s GOLD -r serv1 -a serv2  -j SHORT -B SERVICE_TIME –z 30 –w 10 -commit_outcome
 TRUE -e TRANSACTION -replay_init_time 1800 -retention 86400 -notification TRUE

Modifying Services on Single-Instance Databases

If you are using a single-instance database, then use the DBMS_SERVICE package to modify services in the following way:

declare
params dbms_service.svc_parameter_array;
begin
params('FAILOVER_TYPE'):='TRANSACTION';
params('REPLAY_INITIATION_TIMEOUT'):=1800;
params('RETENTION_TIMEOUT'):=604800;
params('FAILOVER_DELAY'):=10;
params('FAILOVER_RETRIES'):=30;
params('commit_outcome'):='true';
params('aq_ha_notifications'):='true';
dbms_service.modify_service('[your service]',params);
end;
/

Retaining Mutable Values in Application Continuity for Java

A mutable object is a variable, function return value, or other structure that returns a different value each time that it is called. For example, Sequence.NextVal, SYSDATE, SYSTIMESTAMP, and SYS_GUID. To retain the function results for named functions at replay, the DBA must grant KEEP privileges to the user who invokes the function. This security restriction is imposed to ensure that it is valid for replay to save and restore function results for code that is not owned by that user.

Grant and Revoke Interface

You can work with mutables values by using the standard GRANT and REVOKE interfaces in the following way:

Dates and SYS_GUID Syntax

The DATE_TIME and SYS_GUID syntax is as follows:

GRANT [KEEP DATE_TIME | KEEP SYS_GUID].. [to  USER]
REVOKE [KEEP DATE_TIME | KEEP SYS_GUID] … [from USER]

For example, for EBS standard usage with original dates

Grant KEEP DATE_TIME,  KEEP SYS_GUID  to [custom user];
Grant KEEP DATE_TIME,  KEEP SYS_GUID  to [apps user];

Sequence Syntax

The Sequence syntax can be of the following types:

Owned Sequence Syntax

ALTER SEQUENCE [sequence object] [KEEP|NOKEEP];

This command retains the original values of sequence.nextval for replaying, so that the keys match after replay. Most applications need to retain the sequence values at replay. The ALTER SYNTAX is only for owned sequences.

Others Sequence Syntax

GRANT KEEP SEQUENCES.. [to  USER]  on [sequence object];
REVOKE  KEEP SEQUENCES … [from USER]  on [sequence object];

For example, use the following command for EBS standard usage with original sequence values:

Grant KEEP SEQUENCES  to [apps user] on [sequence object]; 
Grant KEEP SEQUENCES  to [custom user] on [sequence object];

GRANT ALL Statement

The GRANT ALL statement grants KEEP privilege on all the objects of a user. However, it excludes mutable values, that is, mutable values require explicit grants.

Rules for Grants on Mutable Values

Follow these rules while granting privileges on mutable objects:

  • If a user has KEEP privilege granted on mutables values, then the objects inherit mutable access when the SYS_GUID, SYSDATE, and SYSTIMESTAMP functions are called.

  • If the KEEP privilege on mutable values on a sequence object is revoked, then SQL or PL/SQL blocks using that object will not allow mutable collection or application for that sequence.

  • If granted privileges are revoked between runtime and failover, then the mutable values that are collected are not applied for replay.

  • If new privileges are granted between runtime and failover, mutable values are not collected and these values are not applied for replay.

Disabling Replay in Application Continuity for Java

This section describes the following concepts:

How to Disable Replay

If any application module uses a design that is unsuitable for replay, then the disable replay API disables replay on a per request basis. Disabling replay can be added to the callback or to the main code by using the disableReplay method of the oracle.jdbc.replay.ReplayableConnection interface. For example:

if (connection instanceof oracle.jdbc.replay.ReplayableConnection)
{ 
    (( oracle.jdbc.replay.ReplayableConnection)connection).disableReplay(); 
    
}

Disabling replay does not alter the connection state by reexecuting any JDBC method, SQL or PL/SQL. When replay is disabled using the disable replay API, both recording and replay are disabled until that request ends. There is no API to reenable replay because it is invalid to reestablish the database session with time gaps in a replayed request. This ensures that replay runs only if a complete history of needed calls has been recorded.

When to Disable Replay

By default, the JDBC replay driver replays following a recoverable error. The disable replay API can be used in the entry point of application modules that are unable to lose the database sessions and recover. For example, if the application uses the UTL_SMTP package and does not want messages to be repeated, then the disableReplay API affects only the request that needs to be disabled. All other requests continue to be replayed.

The following are scenarios to consider before configuring an application for replay:

Application Calls External PL/SQL Actions that Should not Be Repeated

During replay, autonomous transactions and external PL/SQL calls can have side effects that are separate from the main transaction. These side effects are replayed unless you specify otherwise and leave persistent results behind. These side effects include writing to an external table, sending email, forking sessions out of PL/SQL or Java, transferring files, accessing external URLs, and so on. For example, in case of PL/SQL messaging, suppose, you walk away in-between some work without committing and the session times out. Now, if you issue a Ctrl+C command, then the foreground of a component fails. When you resubmit the work, then this side effect can also be repeated.

See Also:

Oracle Database Development Guide for more information about potential side effects of Application Continuity

You must make a conscious decision about whether to enable replay for external actions or not. For example, you can consider the following situations where this decision is important:

  • Using the UTL_HTTP package to issue a SOA call

  • Using the UTL_SMTP package to send a message

  • Using the UTL_URL package to access a web site

Use the disableReplay API if you do not want such external actions to be replayed.

Application Synchronizes Independent Sessions

You can configure an application for replay if the application synchronizes independent sessions using volatile entities that are held until commit, rollback, or session loss. In this case, the application synchronizes multiple sessions connected to several data sources that are otherwise inter-dependent using resources such as a database lock. This synchronization may be fine if the application is only serializing these sessions and understands that any session may fail. However, if the application assumes that a lock or any other volatile resource held by one data source implies exclusive access to data on the same or a separate data source from other connections, then this assumption may be invalidated when replaying.

During replay, the driver is not aware that the sessions are dependent on one session holding a lock or other volatile resource. You can also use pipes, buffered queues, stored procedures taking a resource (such as a semaphore, device, or socket) to implement the synchronization that are lost by failures.

Note:

The DBMS_LOCK does not replay in the restricted version.

Application Uses Time at the Middle-tier in the Execution Logic

In this case, the application uses the wall clock at the middle-tier as part of the execution logic. The JDBC replay driver does not repeat the middle-tier time logic, but uses the database calls that execute as part of this logic. For example, an application using middle-tier time may assume that a statement executed at Time T1 is not reexecuted at Time T2, unless the application explicitly does so.

Application assumes that ROWIds do not change

If an application caches ROWIDs, then access to these ROWIDs may be invalidated due to database changes. Although a ROWID uniquely identifies a row in a table, a ROWID may change its value in the following situations:

  • The underlying table is reorganized

  • An index is created on the table

  • The underlying table is partitioned

  • The underlying table is migrated

  • The underlying table is exported and imported using EXP/IMP/DUL

  • The underlying table is rebuilt using Golden Gate or Logical Standby or other replication technology

  • The database of the underlying table is flashed back or restored

It is bad practice for an application to store ROWIDs for later use as the corresponding row may either not exist or contain completely different data.

Application Assumes that Side Effects Execute Once

In this case, the following are replayed during a replay:

  • Autonomous transactions

  • Opening of back channels separate to the main transaction side effects

Examples of back channels separate to the main transaction include writing to an external table, sending email, forking sessions out of PL/SQL or Java, writing to output files, transferring files, and writing exception files. Any of these actions leave persistent side effects in the absence of replay. Back channels can leave persistent results behind. For example, if a user leaves a transaction midway without committing and the session times out, then the user presses Ctrl+C, the foreground or any component fails. If the user resubmits work, then the side effects can be repeated.

Application Assumes that Location Values Do not Change

SYSCONTEXT options comprise a location-independent set such as National Language Support (NLS) settings, ISDBA, CLIENT_IDENTIFIER, MODULE, and ACTION, and a location-dependent set that uses physical locators. Typically, an application does not use the physical identifier, except in testing environments. If physical locators are used in mainline code, then the replay finds the mismatch and rejects it. However, it is fine to use physical locators in callbacks.

Example

select 
    sys_context('USERENV','DB_NAME') 
    ,sys_context('USERENV','HOST') 
    ,sys_context('USERENV','INSTANCE') 
    ,sys_context('USERENV','IP_ADDRESS') 
    ,sys_context('USERENV','ISDBA')  
    ,sys_context('USERENV','SESSIONID') 
    ,sys_context('USERENV','TERMINAL') 
    ,sys_context('USERENV',ID') 
from dual

Diagnostics and Tracing

The JDBC Replay driver supports standard JDK logging. Logging is enabled using the Java command-line -Djava.util.logging.config.file=<file> option. Log level is controlled with the oracle.jdbc.internal.replay.level attribute in the log configuration file. For example:

oracle.jdbc.internal.replay.level = FINER|FINEST

where, FINER produces external APIs and FINEST produces large volumes of trace. You must use FINEST only under supervision.

If you use the java.util.logging.XMLFormatter class to format a log record, then the logs are more readable but larger. If you are using replay with FAN enabled on UCP or WebLogic Server, then you should also enable FAN-processing logging.

Writing Replay Trace to Console

Following is the example of a configuration file for logging configuration.

oracle.jdbc.internal.replay.level = FINER
handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = ALL
java.util.logging.ConsoleHandler.formatter = java.util.logging.XMLFormatter

Example: Writing Replay Trace to a File

Following is the example of a properties file for logging configuration.

oracle.jdbc.internal.replay.level = FINEST
# Output File Format (size, number and style)
# count: Number of output files to cycle through, by appending an integer to the base file name:
# limit: Limiting size of output file in bytes
handlers = java.util.logging.FileHandler
java.util.logging.FileHandler.pattern = [file location]/replay_%U.trc
java.util.logging.FileHandler.limit = 500000000
java.util.logging.FileHandler.count = 1000
java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter


Footnote Legend

Footnote 1: "A Logon storm is a sudden increase in the number of client connection requests."