My expdp job failed with the error:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit
Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
ORA-31693: Table data object "ADP_SERVICES"."XDATA_MESSAGE" failed to load/unload and is being skipped due to error:
ORA-29913: error in executing ODCIEXTTABLEPOPULATE callout
ORA-01591: lock held by in-doubt distributed transaction 65.9.335625
Master table "SYSTEM"."SYS_EXPORT_TABLE_01" successfully loaded/unloaded
| Cause: | Trying to access resource that is locked by a dead two-phase commit transaction that is in prepared state. |
| Action: | DBA should query the pending_trans$ and related tables, and attempt to repair network connection(s) to coordinator and commit point. If timely repair is not possible, DBA should contact DBA at commit point if known or end user for correct outcome, or use heuristic default if given to issue a heuristic commit or abort command to finalize the local portion of the distributed transaction. |
select local_tran_id,state,fail_time,retry_time,os_user,commit# from dba_2pc_pending;
You can manually force the local portion of the in-doubt transaction by following the steps:
NOTE: Reads are blocked because, until the transaction is resolved, Oracle does not assume which version of the data to display for a query user.
- This node is not the Global Coordinator as the local_tran_id is different than the last portion of the Global_Tran_ID; ie the distributed transaction did not originate at this node.
- GLOBAL_TRANSACTION_ID represents the common transaction ID that will be the same on the every node for a distributed transaction. It is of the form global_database_name.hhhhhhhh.local_transaction_id where global_database_name is the database name of the global coordinator and hhhhhhhh represents in hexadecimal the internal database identifier of the global coordinator.
- The transaction in this node is in PREPARED State
- Look at the ADVICE and TRAN_COMMENT columns for information about this transaction, if any of those columns are populated, they could help you decide whether the local portion of the transaction should be rolled back or committed. (COMMIT COMMENT... or SET TRANSACTION... NAME populates tran_comment column)
We climb the session tree so that we find coordinators, until we eventually reach the global coordinator. Along the way, you might find a coordinator that has resolved the transaction. If not, you can eventually work your way to the commit point site, which will have always have resolved the in-doubt transaction.
To trace the session tree, query DBA_2PC_NEIGHBORS on each node.
To trace the session tree, query DBA_2PC_NEIGHBORS on each node.
- DBA_2PC_NEIGHBORS provides information about connections associated with an in-doubt transaction. Information for each connection is different, based on whether the connection is inbound (IN_OUT = in) or outbound (IN_OUT = out).
- In our case we see that IN_OUT = in, indicating that our db (db102c) is a server of the DB102D client as specified in the database column.
- DBUSER_OWNER shows the connecting user to our db102c database
- INTERFACE = N indicates that neither DB102C nor any of its dependent sites is the commit point site.
COMMIT FORCE 'DB102D.UK.ORACLE.COM.ea27958e.9.46.338','xxx';
-OR-
COMMIT FORCE '65.9.335625','xxx';
The COMMIT FORCE clause lets you manually commit an in-doubt distributed transaction. The transaction is identified by the 'string' containing its local or global transaction ID. You can use integer to specifically assign the transaction a system change number (SCN). If you omit integer, then the transaction is committed using the current SCN.
In order to ensure global integrity, we use the commit# of the commit point site (highest global commit#)
By specifying the SCN for the transaction when forcing a transaction to commit, the in-doubt transaction is committed with the SCN assigned when it was committed at other nodes.
In this way, we can maintain the synchronized commit time of the distributed transaction even if there is a failure.
We only specify an SCN only when we can determine the SCN of the same transaction already committed at another node.
In cases where the in-doubt transaction is to be ROLLBACK you would use syntax ROLLBACK FORCE <local_txn_ID> to set the state to forced rollback.
Purging the Views
DBMS_TRANSACTION.PURGE_LOST_DB_ENTRY is used to manually purge the details from the views once the in-doubt transaction has been resolved:execute DBMS_TRANSACTION.PURGE_LOST_DB_ENTRY ('65.9.335625');

