How to Automate Uploading Backups to Amazon S3

Data Protection Fundamentals: How to Backup an Amazon S3 Bucket

Amazon S3 is reliable cloud storage provided by Amazon Web Services (AWS). Files are stored as objects in Amazon S3 buckets. This storage is widely used to store data backups due to the high reliability of Amazon S3. Unlike Amazon Elastic Block Storage (EBS), where redundant data is stored in i availability zone, in Amazon S3, redundant data is distributed across multiple availability zones. If a data center in one zone becomes unavailable, yous tin can access data in some other zone. In some cases, you may need to support data stored in Amazon S3 buckets to avoid information loss caused by human error or software failure. Data can exist deleted or corrupted if a user who has access to an S3 bucket deletes data or corrupts data by writing unwanted changes. Software failure can crusade similar results.

If y'all want to create an AWS S3 backup, you tin can utilize one of these methods:

• Enable AWS S3 versioning to preserve older versions of files that tin can be restored
• Configure AWS S3 replication from i S3 bucket to some other
• Use the sync tool in AWS command-line interface (CLI) to copy files from AWS S3 to an EC2 instance
• Utilize s3cmd or s4cmd to download files from a bucket to a local file organisation on a Linux automobile

This blog mail explains how you can back up S3 buckets to take a backup copy of data stored in Amazon S3 by using Amazon S3 versioning and AWS CLI.

Amazon S3 Versioning

Object versioning is an effective feature in Amazon S3 that protects your information in a bucket against corruption, writing unwanted changes, and deletion. When changes are fabricated to a file (that is stored as an object in S3), then a new version of the object is created. Multiple versions of the same object are stored in a bucket. You tin admission and restore previous versions of the object. If the object is deleted, the "deletion marker" is applied to the object, but yous can reverse this activeness and open a previous version of the object before deletion. Amazon S3 versioning tin exist used without additional S3 backup software.

You can use the lifecycle policy to define how long versions should be stored in an S3 saucepan to have a class of Amazon S3 backup. Extra costs for storing additional versions should not be high if you configure the lifecycle policy properly, and new versions replace the oldest ones. The sometime versions can be deleted or moved to more than cost-effective storage (for example, cold storage) to optimize costs.

What is an S3 bucket? Read the blog mail to larn more than well-nigh Amazon S3.

How to enable AWS S3 versioning

Sign in to AWS Management Panel by using the account with enough permissions. Click Services and and then select S3 in the Storage category.

In the navigation pane, click Buckets and select the needed S3 bucket you want to enable versioning for. In this example, I select the bucket with the proper name blog-bucket01. Click the bucket name to open up bucket details.

Open up the Properties tab for the selected bucket.

In the Bucket Versioning section, click Edit.

Bucket versioning is disabled past default.

Click Enable to turn on bucket versioning.

Click Save Changes.

A tip is displayed that you might need to update your lifecycle rules. This is the adjacent step.

The bulletin is displayed at the top of the page if configuration changes have been applied: Successfully edited bucket versioning.

Lifecycle rules

In order to configure lifecycle rules for Amazon S3 versioning, go to the Management tab on the page of the selected bucket. In the Lifecycle rules section, click Create lifecycle rule.

The Create lifecycle dominion page is opened.

Lifecycle Rule configuration. Enter the lifecycle rule proper noun, for example, Blog lifecycle 01.

Choose the dominion scope. You can utilize filters to utilize lifecycle rules to specific objects or apply the dominion to all objects in the bucket.

Define object tags to point objects for which lifecycle actions must exist applied. Enter a cardinal and value in the advisable fields and click the Add tag push button to add together the tag or the Remove button to remove the tag.

Lifecycle rule actions. Choose the deportment you want this rule to perform:

• Transition current versions of objects between storage classes
• Transition previous versions of objects between storage classes
• Expire electric current versions of objects
• Permanently delete previous versions of objects
• Delete expired delete markers or incomplete multipart uploads

Transition noncurrent versions of objects between storage classes.

Select storage course transitions and the number of days afterward which objects become noncurrent.

In my example, the objects are moved from the current S3 storage class to Standard-IA after 35 days.

Permanently delete previous versions of objects.

Enter the number of days afterward which previous versions must exist deleted. The value must be higher than the number of days after which objects become noncurrent. In my example, objects are permanently deleted after 40 days.

Click Create Rule to create a lifecycle rule.

Replicating the Bucket

As an alternative to Amazon S3 automated fill-in, yous can replicate the bucket beyond regions. You lot need to create a second bucket that is the destination bucket in another region and create a replication dominion. After creating the replication rule, all changes made in the source saucepan are automatically reflected in the destination bucket.

Locate the Replication rules section in the Management tab for your source bucket and click Create replication rule.

The Create replication rule page opens.

Enter a replication rule proper noun, for example, Blog S3 bucket replication.

Define the rule status when the rule is created (enabled or disabled).

Source bucket. The source bucket has been selected already (weblog-bucket01).

Choose a rule scope. You can utilize the replication rule for all objects in the saucepan or configure filters and employ the dominion to custom objects.

Destination. Enter the destination bucket name or click Browse S3 and select a bucket from the list. You can select a saucepan in this account or in another account. If AWS S3 versioning is enabled for the source bucket, object versioning must also be enabled for the destination bucket. A destination region is displayed for the selected destination bucket.

Configure the identity and access management (IAM) role, then select a storage class and additional replication options. Click Save to save the configuration and create a replication rule for the saucepan.

AWS S3 Backup in the CLI

AWS CLI is the powerful command-line interface to work with dissimilar Amazon services, including Amazon S3. There is a useful sync control that allows you to support Amazon S3 buckets to a Linux motorcar by copying files from the saucepan to a local directory in Linux running on an EC2 instance.

A feature of the sync command in AWS CLI is that files in a local file arrangement (Amazon S3 backup destination) are not deleted if these files are missing in the source S3 saucepan and vice versa. This is important for AWS S3 backup because if some files were accidentally deleted in the S3 saucepan, the existing files are not deleted in the local directory of a Linux machine after synchronization.

Advantages:

• Back up of large S3 buckets and scalability
• Multiple threads are supported during synchronization
• The power to synchronize only new and updated files
• Loftier synchronization speed due to smart algorithms

Disadvantages:

• Linux running on an EC2 instance consumes the storage infinite of EBS volumes. Storage costs for EBS volumes are higher than for S3 buckets.

Commands for Ubuntu Server are used in this tutorial.

Showtime, y'all demand to install AWS CLI.

Update the tree of repositories:

sudo apt-get update

Install AWS CLI:

sudo apt install awscli

or

Install unzip:

sudo apt install unzip

ringlet "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.nix" -o "awscliv2.zip"

unzip awscli-exe-linux-x86_64.nil

sudo ./aws/install

.

Check the AWS credentials in Linux running on your EC2 case.

aws configure list

.

Add together credentials to access AWS with AWS CLI from the Linux instance if credentials have not been set:

aws configure

Enter the post-obit parameters:

AWS Access Key ID

AWS Secret Access Key

Default region name

Default output format

.

Create a directory to store your Amazon S3 fill-in. In my example, I create the ~/s3/ directory to store S3 backups and a subdirectory with a name identical to the bucket name. Files stored in the S3 bucket should exist copied to this local directory on the Linux machine. ~ is the home directory of a user, which is /dwelling/ubuntu in my case.

mkdir -p ~/s3/your_bucket_name

Replace your_bucket_name with the proper noun of your saucepan (weblog-bucket01 in our instance).

mkdir -p ~/s3/blog-bucket01

.

Synchronize the bucket contents with your local directory on the EC2 case running Linux:

aws s3 sync s3:// web log-bucket01 /domicile/ubuntu/s3/ weblog-bucket01/

.

If the credential configuration, saucepan name, and destination path are correct, data should outset downloading from the S3 bucket. The fourth dimension needed to end the operation depends on the size of the files in the bucket and your internet connexion speed.

Amazon S3 automated backup

You can configure Amazon S3 automatic backup jobs with AWS CLI sync. Create a sync.sh script file to run AWS S3 fill-in (synchronize files from an S3 bucket to a local directory on your Linux instance) then run this script on schedule.

nano /habitation/ubuntu/s3/sync.sh

.

#!/bin/sh

# Brandish the current date and time

repeat '-----------------------------'

engagement

repeat '-----------------------------'

echo ''

# Display the script initialization message

echo 'Syncing remote S3 bucket...'

# Running the sync command

/usr/bin/aws s3 sync s3://{BUCKET_NAME} /home/ubuntu/s3/{BUCKET_NAME}/

# Echo "Script execution is completed"

repeat 'Sync complete'

.

Supercede the {BUCKET_NAME} with the name of the S3 bucket you lot desire to back upwardly.

The complete path to aws (AWS CLI binary) is defined to make crontab execute the aws application correctly in the shell environment used past crontab.

Make the script executable:

sudo chmod +ten /domicile/ubuntu/s3/sync.sh

.

Run the script to bank check whether the script works:

/home/ubuntu/s3/sync.sh

.

Edit crontab (a scheduler in Linux) of the electric current user to schedule Amazon S3 backup script execution.

crontab -east

You may need to select a text editor to edit crontab configuration.

.

The format of crontab for scheduling tasks is the following:

g h dom mon dow command

Where: k – minutes; h – hours; dom – day of calendar month; dow – twenty-four hours of calendar week.

Permit's add a configuration line for the task to run synchronization every hr and relieve AWS S3 backup results to the log file. Add this line to the bottom of the crontab configuration.

0 * * * * /home/ubuntu/s3/sync.sh > /home/ubuntu/s3/sync.log

Amazon S3 automatic backup is configured. The log file tin be used to bank check execution of synchronization tasks.

Conclusion

At that place are multiple methods to perform Amazon S3 backup and two of them take been covered in this blog mail service. You can enable object versioning for a bucket to preserve previous versions of objects, which allows you to get files if unwanted changes were written to the files. Amazon S3 replication is some other native tool to make a copy of files stored in an Amazon S3 bucket as objects. In this case, objects are replicated from i bucket to another. You can besides create a backup of an Amazon S3 bucket by using the sync tool in AWS CLI, which allows you to synchronize files in a bucket with a local directory of a Linux car running on an EC2 example. Amazon S3 automated backup tin can be scheduled past using a script and crontab.

In general, Amazon S3 deject storage is very reliable and fill-in to Amazon S3 is a common exercise. If y'all have a strong data protection strategy and AWS backup strategy, you should have a backup copy. In this case, it is recommended that you back up data to Amazon S3 and some other destination location. Use NAKIVO Backup & Replication to protect your data on physical and virtual machines. NAKIVO Backup & Replication supports backup of Amazon EC2 instances, VMware vSphere VMs, Hyper-V VMs, and concrete machines. The backup destination can exist an Amazon S3 bucket, EC2 instance with the attached EBS volume, local directory on a physical server, virtual machine, and network-attached storage (NAS). NAKIVO Backup & Replication likewise supports backup copy to tape. You can back up data to Amazon S3 and transport a fill-in copy to other supported locations. Schedule a demo to encounter the power of features included in NAKIVO Fill-in & Replication.

cheynebefurely.blogspot.com

Source: https://www.nakivo.com/blog/how-to-backup-an-amazon-s3-bucket/

Share this post