with an explicit deny in a VPC endpoint policy. | Ensure that the required VPC endpoints exist and the policies attached to them aren't restricting the listed required permissions and resources. |
### Networking issues
In addition to reviewing [Step 6: Configure your VPC for RDS Custom for Oracle](custom-setup-orcl.md#custom-setup-orc.vpc-config), verify that the following are configured correctly and not restricting access to the required AWS services:
**Security group attached to the Amazon EC2 instance**
Ensure that the security group allows all necessary inbound and outbound traffic for RDS Custom operations.
**Security group attached to your VPC**
Verify that VPC security groups permit traffic to and from the required AWS services.
**VPC endpoints**
Confirm that all required VPC endpoints are properly configured and accessible.
**Networking access control lists**
Check that network ACLs aren't blocking traffic needed for RDS Custom functionality.
## Troubleshooting custom engine version creation for RDS Custom for Oracle
When CEV creation fails, RDS Custom issues `RDS-EVENT-0198` with the message `Creation failed for custom engine version major-engine-version.cev_name`, and includes details about the failure. For example, the event prints missing files.
CEV creation might fail because of the following issues:
+ The Amazon S3 bucket containing your installation files isn't in the same AWS Region as your CEV.
+ When you request CEV creation in an AWS Region for the first time, RDS Custom creates an S3 bucket for storing RDS Custom resources (such as CEV artifacts, AWS CloudTrail logs, and transaction logs).
CEV creation fails if RDS Custom can't create the S3 bucket. Either the caller doesn't have S3 permissions as described in [Step 5: Grant required permissions to your IAM user or role](custom-setup-orcl.md#custom-setup-orcl.iam-user), or the number of S3 buckets has reached the limit.
+ The caller doesn't have permissions to get files from your S3 bucket that contains the installation media files. These permissions are described in [Step 7: Add necessary IAM permissions](custom-cev.preparing.md#custom-cev.preparing.iam).
+ Your IAM policy has an `aws:SourceIp` condition. Make sure to follow the recommendations in [AWS Denies access to AWS based on the source IP](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_aws_deny-ip.html) in the *AWS Identity and Access Management User Guide*. Also make sure that the caller has the S3 permissions described in [Step 5: Grant required permissions to your IAM user or role](custom-setup-orcl.md#custom-setup-orcl.iam-user).
+ Installation media files listed in the CEV manifest aren't in your S3 bucket.
+ The SHA-256 checksums of the installation files are unknown to RDS Custom.
Confirm that the SHA-256 checksums of the provided files match the SHA-256 checksum on the Oracle website. If the checksums match, contact [AWS Support](https://aws.amazon.com/premiumsupport) and provide the failed CEV name, file name, and checksum.
+ The OPatch version is incompatible with your patch files. You might get the following message: `OPatch is lower than minimum required version. Check that the version meets the requirements for all patches, and try again`. To apply an Oracle patch, you must use a compatible version of the OPatch utility. You can find the required version of the Opatch utility in the readme file for the patch. Download the most recent OPatch utility from My Oracle Support, and try creating your CEV again.
+ The patches specified in the CEV manifest are in the wrong order.
You can view RDS events either on the RDS console (in the navigation pane, choose **Events**) or by using the `describe-events` AWS CLI command. The default duration is 60 minutes. If no events are returned, specify a longer duration, as shown in the following example.
```
aws rds describe-events --duration 360
```
Currently, the MediaImport service that imports files from Amazon S3 to create CEVs isn't integrated with AWS CloudTrail. Therefore, if you turn on data logging for Amazon RDS in CloudTrail, calls to the MediaImport service such as the `CreateCustomDbEngineVersion` event aren't logged.
However, you might see calls from the API gateway that accesses your Amazon S3 bucket. These calls come from the MediaImport service for the `CreateCustomDbEngineVersion` event.
## Fixing unsupported configurations in RDS Custom for Oracle
In the shared responsibility model, it's your responsibility to fix configuration issues that put your RDS Custom for Oracle DB instance into the `unsupported-configuration` state. If the issue is with the AWS infrastructure, use the console or the AWS CLI to fix it. If the issue is with the operating system or the database configuration, log in to the host to fix it.
**Note**
This section explains how to fix unsupported configurations in RDS Custom for Oracle. For information about RDS Custom for SQL Server, see [Fixing unsupported configurations in RDS Custom for SQL Server](custom-troubleshooting-sqlserver.md#custom-troubleshooting-sqlserver.fix-unsupported).
The following tables includes descriptions of the notifications and events that the support perimeter sends and how to fix them. These notifications and the support perimeter are subject to change. For background on the support perimeter, see [RDS Custom support perimeter](custom-concept.md#custom-troubleshooting.support-perimeter). For event descriptions, see [Amazon RDS event categories and event messages](USER_Events.Messages.md).
| Event ID | Configuration | RDS event message | Action |
| --- | --- | --- | --- |
| `SP-O0000` | Manual unsupported configuration | The RDS Custom DB instance status is set to [Unsupported configuration] because of: *reason*. | To resolve this issue, create an Support case. |
**AWS resources (infrastructure)**
| Event ID | Configuration | RDS event message | Action |
| --- | --- | --- | --- |
| SP-O1001 | Amazon Elastic Block Store (Amazon EBS) volumes | The following EBS volumes were added to EC2 instance *ec2\$1id*: *volume\$1id*. To resolve the issue, detach the specified volumes from the instance. | RDS Custom creates two types of EBS volume, besides the root volume created from the Amazon Machine Image (AMI), and associates them with the EC2 instance: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) When you create your DB instance, the storage configurations that you specify configure the data volumes. The support perimeter monitors the following: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) Use the following CLI command to compare the volume type of the EBS volume details and the RDS Custom for Oracle DB instance details: aws rds describe-db-instances \
--db-instance-identifier db-instance-name | grep StorageType
|
| SP-O1002 | Amazon Elastic Block Store (Amazon EBS) volumes | EBS volume *volume\$1id* has been detached from EC2 instance [*ec2\$1id*]. You can't detach the original volume from this instance. To resolve the issue, re-attach *volume\$1id* to *ec2\$1id*. | RDS Custom creates two types of EBS volume, besides the root volume created from the Amazon Machine Image (AMI), and associates them with the EC2 instance: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) When you create your DB instance, the storage configurations that you specify configure the data volumes. The support perimeter monitors the following: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) Use the following CLI command to compare the volume type of the EBS volume details and the RDS Custom for Oracle DB instance details: aws rds describe-db-instances \
--db-instance-identifier db-instance-name | grep StorageType
|
| SP-O1003 | Amazon Elastic Block Store (Amazon EBS) volumes | The original EBS volume *volume\$1id* attached to EC2 instance *ec2\$1id* has been modified as follows: size [*X*] to [*Y*], type [*N*] to [*M*], or IOPS [*J*] to [*K*]. To resolve the issue, revert the modification. | RDS Custom creates two types of EBS volume, besides the root volume created from the Amazon Machine Image (AMI), and associates them with the EC2 instance: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) When you create your DB instance, the storage configurations that you specify configure the data volumes. The support perimeter monitors the following: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) Use the following CLI command to compare the volume type of the EBS volume details and the RDS Custom for Oracle DB instance details: aws rds describe-db-instances \
--db-instance-identifier db-instance-name | grep StorageType
|
| SP-O1004 | Amazon EC2 instance state | Automated recovery left EC2 instance [*ec2\$1id*] in an impaired state. To resolve the issue, see [Troubleshooting instance recovery failures](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-recover.html#TroubleshootingInstanceRecovery). | To check the status of a DB instance, use the console or run the following AWS CLI command: aws rds describe-db-instances \
--db-instance-identifier db-instance-name |grep DBInstanceStatus
|
| SP-O1005 | Amazon EC2 instance attributes | EC2 instance [*ec2\$1id*] was modified as follows: attribute [*att1*] changed from [*val-old*] to [*val-new*], attribute [*att2*] changed from [*val-old*] to [*val-new*]. To resolve the issue, revert to the original value. | |
| SP-O1006 | Amazon EC2 instance state | EC2 instance [*ec2\$1id*] was terminated or can't be found. To resolve the issue, delete the RDS Custom DB instance. | The support perimeter monitors EC2 instance state-change notifications. The EC2 instance must always be running. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) |
| SP-O1007 | Amazon EC2 instance state | EC2 instance [*ec2\$1id*] was stopped. To resolve the issue, start the instance. | The support perimeter monitors EC2 instance state-change notifications. The EC2 instance must always be running. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) |
| SP-1008 | Amazon SQS permission | Permissions are missing for Amazon SQS. Check the permissions for the IAM instance profile, VPC endpoint policy, and dependent service connections, and then try again. | You can resolve this by making sure the IAM profile associated with the host has the following permissions: "SQS:SendMessage"
"SQS:ReceiveMessage"
"SQS:DeleteMessage"
"SQS:GetQueueUrl"
|
| SP-1009 | Amazon Simple Queue Service (Amazon SQS) | The SQS queue [%s] was deleted and couldn't be recovered. To resolve this issue, recreate the queue. | Recreate the Amazon SQS queue. |
**Operating system**
| Event ID | Configuration | RDS event message | Action |
| --- | --- | --- | --- |
| SP-O2001 | RDS Custom agent status | The RDS Custom agent isn't running on EC2 instance [*ec2\$1id*]. Make sure the agent is running on [*ec2\$1id*]. | On RDS Custom for Oracle, the DB instance goes outside the support perimeter if the RDS Custom agent stops. The agent publishes the `IamAlive` metric to Amazon CloudWatch every 30 seconds. An alarm is triggered if the metric hasn't been published for 30 seconds. The support perimeter also monitors the RDS Custom agent process state on the host every 30 minutes. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) When the RDS Custom agent is running again, the `IamAlive` metric is published to Amazon CloudWatch, and the alarm switches to the `OK` state. This switch notifies the support perimeter that the agent is running. |
| `SP-O2002` | AWS Systems Manager agent (SSM agent) status | The Systems Manager agent on EC2 instance [*ec2\$1id*] is unreachable. Make sure that you that have correctly configured the network, agent, and IAM permissions. | SSM Agent must always be running. The RDS Custom agent is responsible for making sure that the Systems Manager agent is running. If SSM Agent was terminated and then restarted, the RDS Custom agent publishes the metric `SSM_Agent_Restarted_Or_NotFound` to CloudWatch. The RDS Custom agent has an alarm on the metric `do-not-delete-rds-custom-ssm-agent-restarted-or-notfound-ec2-id` configured to trigger when there has been a restart in each of the previous three minutes. The support perimeter also monitors the process state of SSM Agent on the host every 30 minutes. For more information, see [Troubleshooting SSM Agent](https://docs.aws.amazon.com/systems-manager/latest/userguide/troubleshooting-ssm-agent.html). |
| `SP-O2003` | AWS Systems Manager agent (SSM agent) status | The Systems Manager agent on EC2 instance [*ec2\$1id*] crashed multiple times. For more information, see the SSM Agent troubleshooting documentation. | For more information, see [Troubleshooting SSM Agent](https://docs.aws.amazon.com/systems-manager/latest/userguide/troubleshooting-ssm-agent.html). |
| SP-O2004 | OS time zone | The time zone on EC2 instance [*ec2\$1id*] was changed. To resolve this issue, revert the timezone to the previous setting of [*previous-time-zone*]. Then use an RDS options group to change the time zone. | RDS automation detected that the time zone on the host was changed without the use of an option group. This host-level change can cause RDS automation failures, so the EC2 instance is placed in the `unsupported-configuration` state. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) Your DB instance becomes available within 30 minutes. To prevent moving out of perimeter in the future, modify your timezone through an options group. For more information, see [Oracle time zone](custom-managing.timezone.md). |
| SP-O2005 | `sudo` configurations | The sudo configurations on EC2 instance [*ec2\$1id*] lack necessary permissions. To resolve this issue, revert the recent changes to the sudo configurations. | The support perimeter verifies that certain OS users are allowed to run certain commands on the host. It monitors `sudo` configurations and compares them to the supported state. If the `sudo` configurations aren't supported, RDS Custom tries to overwrite them and return to the previous supported state. If the attempt is successful, RDS Custom sends the following notification: RDS Custom successfully overwrote your configuration. If the overwrite isn't successful, your DB instance remains in the unsupported configuration state. To resolve this problem, either revert the changes within the `sudoers.d/` file or fix the permissions. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) After the support perimeter determines that the `sudo` configurations are supported, your RDS Custom for Oracle DB instance becomes available within 30 minutes. |
| SP-O2006 | S3 bucket accessibility | RDS Custom automation can't download files from the S3 bucket on EC2 instance [*ec2\$1id*]. Check your networking configuration and make sure the instance allows connections to and from S3. | |
| SP-2007 | High Availability Software Solution Version | The HA solution of your instance differs from the expected version. To resolve this issue, create an AWS Support case. | Create an AWS Support case. |
**Database**
| Event ID | Configuration | RDS event message | Action |
| --- | --- | --- | --- |
| SP-O3001 | Database archive lag target | The ARCHIVE\$1LAG\$1TARGET parameter on EC2 instance [*ec2\$1id*] is out of the recommended range *value\$1range*. To resolve the issue, set the parameter to a value within value\$1range. | The support perimeter monitors the `ARCHIVE_LAG_TARGET` database parameter to verify that the latest restorable time of the DB instance is within reasonable bounds. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) Your DB instance becomes available within 30 minutes. |
| SP-O3002 | Oracle Data Guard role | The database role [*role\$1name*] isn't supported for Oracle Data Guard on EC2 instance [*ec2\$1id*]. To resolve the issue, set the DATABASE\$1ROLE parameter to either PRIMARY or PHYSICAL STANDBY. | The support perimeter monitors the current database role every 15 seconds and sends a CloudWatch notification if the database role has changed. The Oracle Data Guard `DATABASE_ROLE` parameter must be either `PRIMARY` or `PHYSICAL STANDBY`. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) After the support perimeter determines that the database role is supported, your RDS Custom for Oracle DB instance becomes available within 15 seconds. |
| SP-O3003 | Database health | The SMON process of the Oracle database is in a zombie state. To resolve the issue, manually recover the database on EC2 instance [*ec2\$1id*], open the database, and then immediately back it up. For more help, contact Support. | The support perimeter monitors the DB instance state. It also monitors how many restarts occurred during the previous hour and day. You're notified when the instance is in a state where it still exists, but you can't interact with it. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) After your DB instance restarts, the RDS Custom agent detects that your DB instance is no longer in an unresponsive state. It then notifies the support perimeter to reevaluate your DB instance state. |
| SP-O3004 | Database log mode | The database log mode on EC2 instance [*ec2\$1id*] was changed to [*value\$1b*]. To resolve the issue, set the log mode to [*value\$1a*]. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) The RDS Custom agent automatically restarts your DB instance and sets the log mode to `ARCHIVELOG`. Your DB instance becomes available within 30 minutes. |
| SP-O3005 | Oracle home path | The Oracle home on EC2 instance [*ec2\$1id*] was changed to *new\$1path*. To resolve the issue, revert the setting to *old\$1path*. | |
| SP-O3006 | Database unique name | The database unique name on EC2 instance [*ec2\$1id*] was changed to *new\$1value*. To resolve the issue, revert the name to *old\$1value*. | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/custom-troubleshooting.html) The RDS Custom agent automatically restarts your DB instance and sets the log mode to `ARCHIVELOG`. Your DB instance becomes available within 30 minutes. |
## Troubleshooting upgrades for RDS Custom for Oracle
Your upgrade of an RDS Custom for Oracle instance might fail. Following, you can find techniques that you can use during upgrades of RDS Custom DB for Oracle DB instances:
+ Examine the upgrade output log files in the `/tmp` directory on your DB instance. The names of the logs depend on your DB engine version. For example, you might see logs that contain the strings `catupgrd` or `catup`.
+ Examine the `alert.log` file located in the `/rdsdbdata/log/trace` directory.
+ Run the following `grep` command in the `root` directory to track the upgrade OS process. This command shows where the log files are being written and determine the state of the upgrade process.
```
ps -aux | grep upg
```
The following shows sample output.
```
root 18884 0.0 0.0 235428 8172 ? S< 17:03 0:00 /usr/bin/sudo -u rdsdb /rdsdbbin/scripts/oracle-control ORCL op_apply_upgrade_sh RDS-UPGRADE/2.upgrade.sh
rdsdb 18886 0.0 0.0 153968 12164 ? S< 17:03 0:00 /usr/bin/perl -T -w /rdsdbbin/scripts/oracle-control ORCL op_apply_upgrade_sh RDS-UPGRADE/2.upgrade.sh
rdsdb 18887 0.0 0.0 113196 3032 ? S< 17:03 0:00 /bin/sh /rdsdbbin/oracle/rdbms/admin/RDS-UPGRADE/2.upgrade.sh
rdsdb 18900 0.0 0.0 113196 1812 ? S< 17:03 0:00 /bin/sh /rdsdbbin/oracle/rdbms/admin/RDS-UPGRADE/2.upgrade.sh
rdsdb 18901 0.1 0.0 167652 20620 ? S< 17:03 0:07 /rdsdbbin/oracle/perl/bin/perl catctl.pl -n 4 -d /rdsdbbin/oracle/rdbms/admin -l /tmp catupgrd.sql
root 29944 0.0 0.0 112724 2316 pts/0 S+ 18:43 0:00 grep --color=auto upg
```
+ Run the following SQL query to verify the current state of the components to find the database version and the options installed on the DB instance.
```
SET LINESIZE 180
COLUMN COMP_ID FORMAT A15
COLUMN COMP_NAME FORMAT A40 TRUNC
COLUMN STATUS FORMAT A15 TRUNC
SELECT COMP_ID, COMP_NAME, VERSION, STATUS FROM DBA_REGISTRY ORDER BY 1;
```
The output resembles the following.
```
COMP_NAME STATUS PROCEDURE
---------------------------------------- -------------------- --------------------------------------------------
Oracle Database Catalog Views VALID DBMS_REGISTRY_SYS.VALIDATE_CATALOG
Oracle Database Packages and Types VALID DBMS_REGISTRY_SYS.VALIDATE_CATPROC
Oracle Text VALID VALIDATE_CONTEXT
Oracle XML Database VALID DBMS_REGXDB.VALIDATEXDB
4 rows selected.
```
+ Run the following SQL query to check for invalid objects that might interfere with the upgrade process.
```
SET PAGES 1000 LINES 2000
COL OBJECT FOR A40
SELECT SUBSTR(OWNER,1,12) OWNER,
SUBSTR(OBJECT_NAME,1,30) OBJECT,
SUBSTR(OBJECT_TYPE,1,30) TYPE, STATUS,
CREATED
FROM DBA_OBJECTS
WHERE STATUS <>'VALID'
AND OWNER IN ('SYS','SYSTEM','RDSADMIN','XDB');
```
## Troubleshooting replica promotion for RDS Custom for Oracle
You can promote managed Oracle replicas in RDS Custom for Oracle using the console, `promote-read-replica` AWS CLI command, or `PromoteReadReplica` API. If you delete your primary DB instance, and all replicas are healthy, RDS Custom for Oracle promotes your managed replicas to standalone instances automatically. If a replica has paused automation or is outside the support perimeter, you must fix the replica before RDS Custom can promote it automatically. For more information, see [Promoting an RDS Custom for Oracle replica to a standalone DB instance](custom-rr.promoting.md).
The replica promotion workflow might become stuck in the following situation:
+ The primary DB instance is in the state `STORAGE_FULL`.
+ The primary database can't archive all of its online redo logs.
+ A gap exists between the archived redo log files on your Oracle replica and the primary database.
**To respond to the stuck workflow**
1. Synchronize the redo log gap on your Oracle replica DB instance.
1. Force the promotion of your read replica to the latest applied redo log. Run the following commands in SQL\$1Plus:
```
ALTER DATABASE ACTIVATE STANDBY DATABASE;
SHUTDOWN IMMEDIATE
STARTUP
```
1. Contact Support and ask them to move your DB instance to `available` status.