Exporting data to cloud storage


Configure your target subscription to export data directly to your Amazon Simple Storage Service (Amazon S3™) private cloud storage.

Overview of tasks

To export data to cloud storage, the following tasks must be completed:

Amazon Web Services

  • Create an Amazon S3 bucket.

  • Create an IAM role.

  • Set a policy for the Amazon S3 bucket and attach it to the IAM role.

Veeva Network

  • Create an external credential for Amazon S3.

  • Configure the target configuration to export to the Amazon S3 bucket.

Configure Amazon Web Services

Create an Amazon S3 bucket and IAM role so you can export data directly from a target subscription.

For detailed information about these steps, see the Amazon Simple Storage Service User Guide: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html.

Complete the following steps in your AWS account:

  1. Create an Amazon S3 bucket. This will be the target S3 bucket for Network to write to.

    On the Create bucket page, type a Bucket name, choose the AWS Region and ensure that Block all public access is selected.

  2. In the IAM Management Console, create an IAM Role. Network will use this to connect to the Amazon S3 bucket and load data.

    You can either copy the generated Trust Relationship Statement into the IAM role or use Amazon's IAM role setup. The trust relationship statement is generated when you create the Amazon S3 credential in Network (Settings > External Credentials).

    Example Generated Trust Relationship Statement

    {
       "Effect": "Allow",
       "Principal": {
         "AWS": "arn:aws:iam::271184731614:root"
       },
       "Action": "sts:AssumeRole",
       "Condition": {
         "StringEquals": {
           "sts:ExternalId": "verteo.veevanetwork.com:instance_307"
         }
       }
    }
    

    If you use Amazon's IAM role setup, the following settings are required:

    • The Trusted entity type is Another AWS account.

    • The Account ID is 271184731614 (the Network AWS Account ID).

    • The External ID is required by Network and is considered a best practice for third-party access. For example, verteo.veevanetwork.com:instance_307.

    Note: The External ID varies for each Network instance. Refer to the Generated Trust Relationship Statement for the information in your Network instance.

  3. In the IAM Management Console, create a policy for the Amazon S3 bucket. This is used by the IAM role to load data to the Amazon S3 bucket.

    The AWS permissions must enable the following actions:

    • List - ListBucket

    • Write - PutObject

    In the Resources section, define the Amazon Resource Name (ARN) for the bucket and object.

    The ARN format uses the following naming convention:

    arn:aws:s3:::<S3 bucket name>

    Example policy

    {
        "Version": "2020-09-01",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "s3:PutObject",
                    "s3:ListBucket"
                ],
                "Resource": [
                    "arn:aws:s3:::verteo-bucket/*",
                    "arn:aws:s3:::verteo-bucket"
                ]
            }
        ]
    }
  4. In the IAM Management Console, attach the new policy to the IAM role that you created.

Create an Amazon S3 credential

Create the credential in your Network instance so you can connect to your Amazon S3 bucket.

  1. In the Admin console, click Settings > External Credentials.

  2. Click Add Credentials.

  3. On the New External Credential dialog, choose Amazon S3 from the list.

  4. Click Continue.

  5. On the New External Credential page, if you haven't already set up your IAM role, click Generate Trust Relationship Statement to get the JSON for the IAM Role trust relationship.

  6. Provide the following information for the S3 credential:

  7. In the Name field, type a name for the credential.

  8. In the Role ARN field, type the Amazon Resource Name for the IAM Role.
  9. Type the S3 URI. The s3:// prefix is added to the URI by default. If you have folders in your bucket, specify the folders in the path.
  10. Expand the S3 Region list to select the AWS region where the S3 bucket is located; for example, us-west-2.

    To review the list of regions, see the AWS General Reference Guide: https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region.

  11. Click Test Connection. Network validates the credentials and ensures that the S3 bucket exists.

  12. Save your changes.

The Amazon S3 credential is complete. It is now added to the list on the External Credentials page.

Configure the target subscription

Set the target subscription to export files directly to the Amazon S3 bucket.

To configure the subscription:

  1. Create a target subscription or open an existing subscription (System Interfaces > Target Subscriptions).

  2. In the General Export Options section, define the settings in the File Format section as usual.

  3. In the Export Locations section, select Export to Cloud Storage.

    The Network FTP Path remains selected because a copy is always exported there also.

  4. The External Credential Location setting displays. Expand the list and choose your Amazon S3 credential.

    Click Test Connection to validate that credential is correct.

  5. Save your changes.

After you run the source subscription, the Job Details page includes the export path in the Job Result Summary section.

Export errors

If there was a problem exporting to the cloud storage, the following error will be reported in the Job Error Log on the Job Details page:

Unexpected file system error encountered, please check your configuration <external credential>. If the problem persists contact your Veeva Network administrators."

Exported files

The subscription files will be exported to your Amazon S3 bucket using the following path format: <S3 URI>/<export folder>.

Example

These example paths are based on the following settings:

  • External Credential Location (S3 URI) - s3://verteo-bucket

  • Compression format - Zip

  • Export File/Folder Name - Default. For example, exp_000000A3.zip.

Compression Format Cloud Storage Path
Compressed single file s3://verteo-bucket/exp_000000A3.zip
Compressed individual files s3://verteo-bucket/exp_000000A3/hcp.zip
s3://verteo-bucket/exp_000000A3/hco.zip
s3://verteo-bucket/exp_000000A3/address.zip
Uncompressed s3://verteo-bucket/exp_000000A3/hcp.csv
s3://verteo-bucket/exp_000000A3/hco.csv
s3://verteo-bucket/exp_000000A3/address.csv

The folders are created in the Amazon S3 bucket when the files are exported.

Configuration management considerations

When you include a target subscription in an export package, the Amazon S3 credentials are not migrated to the target environment. The Export to Cloud Storage option in the target subscription configuration is also disabled in the target environment.