Introduction
NOTE: This guide requires the use of WOlastic 20090708 or later!
One of the major "gripes" about Amazon EC2 instances is that by default, do not handle persistent data. In other words, if you ever decide to "power-off" an EC2 instance, all data stored on the instance storage, including any changes to the 10GB root partition from your AMI, is lost. (Note: However, rebooting an EC2 instance, no data loss occurs.)
Amazon recently began offering as part of their Amazon AWS service, Amazon EBS. EBS, which stands for "Elastic Block Store", handles persistence for when "issues" arises with an EC2 instance. You can view EBS, basically, as a hard-drive that is an accessible, mountable and formattable block-device for your EC2 instance. Sizes begin as small as 1GB up to 1TB (1,000GB).
Pricing structure is relatively simple with Amazon EBS. Pricing for storage is $0.10 USD per allocated GB per month. Meaning a 100GB EBS volume would cost $10 USD/month. However on top of that, Amazon has I/O charges on top of it at $0.10 USD per 1-million I/O. Amazon uses an example of a medium-sized database which does about 100 I/Os per second over the course of a month which equates to approximately an additional $16 USD/month. So your total bill for your 100GB EBS volume would be approximately $26 USD/month.
For all of the complete features of Amazon EBS, please visit:
Creating an Amazon EBS Volume
1. The assumption of this guide is you've already launched and are familiar with the WOlastic instance environment.
2. Log into your Amazon AWS account's Console at the following URL:
3. Find the instance you wish to add the EBS volume to. Under the Navigation pane on the left, select "Instances" and find your running WOlastic instance. Select & highlight the instance so it shows you the instance details on the bottom frame. You will see something similar to this:
In this example, our WOlastic instance is running on us-east-1b.
4. Now, under the Navigation pane on the left, select "Volumes" under the "Elastic Block Store" header.
5. If you do not have any existing "Volumes", your only option will be "Create Volume", otherwise it will list your existing EBS volumes that are under your account. Click on "Create Volume".
6. You will see a pop-out window asking for the following information:
7. In our example, we will be creating a 10GB EBS volume. You can create any size between 1GB and 1,000GB (1TB).
8. For "Availability Zone", select the zone in which your WOlastic instance is running on. In our example, it would be us-east-1b.
9. "Snapshot" isn't relevant to this guide, but basically you can restore "snapshots" of volumes which are stored on Amazon S3 (i.e. backups, pre-loaded data, etc.) Since in our case we're just creating an EBS volume to store our PostgreSQL or MySQL database, we leave this blank.
10. Click "Create" after you've filled out the relevant information.
11. Amazon AWS will at this point will create your EBS volume. Under "Status", it should show "creating". You can manually click "Refresh" to see the status of the EBS volume creation process. Once the "Status" shows available, we're ready to mount this volume under your WOlastic instance.
Mounting/Attaching an Amazon EBS Volume to your WOlastic Instance
1. At this point, we need to attach this volume to your WOlastic Instance. Select the EBS volume you just created in the AWS Management Console and click on "Attach Volume". A pop-up will occur asking you the following information:
2. "Volume" will automatically be filled in for you. "Instances" will be a pulldown of which instances are available to have the EBS volume attach to. If you have more than one instance running in an availability zone, please double-check you're attaching to the correct instance. If it's your only instance, then the pre-filled instance ID is the correct one.
3. "Device" should default to /dev/sdf. Leave it be. Now click "Attach".
4. At this point, the EBS volume will attach itself to the WOlastic instance. Hit "Refresh" a couple times and make sure under the "Attachment Information" column that the instance ID and associated mount point shows "attached".
5. Log into your WOlastic instance via SSH, if you haven't already, as the "appserver" user. At the shell prompt, type "dmesg". The last line you see should be similar to this:
6. This correctly means your Amazon EBS volume is now attached to your WOlastic instance.
7. Now, we must prepare the volume to store data so we must format a file-system. We will be using XFS. From the shell prompt, run the following command:
8. You might be prompted for a password. Enter the password for "appserver". If everything goes correctly, you should see something similar to this:
9. Your EBS is now ready to be used! All of the PostgreSQL & MySQL data are stored in /vol. We will now "move" the existing PostgreSQL & MySQL data off the 10GB root-partition in /vol to the new XFS-formatted EBS volume.
Migrating PostgreSQL & MySQL data to your EBS Volume
1. Run the following command to mount the new EBS volume under /temp:
2. You can run a "df" at the shell prompt to make sure it shows up mounted as /temp.
3. If you have existing data that is important, BACKUP your databases NOW! There are many guides on how to backup PostgreSQL & MySQL databases. Google it!
4. Now, shutoff all running instances of PostgreSQL & MySQL, if running. Run the following commands at the shell prompt:
5. If you see "FAILED", that means the service wasn't running in the first place. If it is running, you should see "OK".
6. Now we will move the data. Run the following commands (you have backups if you need them, right?) from the shell prompt:
7. Verify that the data has moved by running an "ls -l /temp". You should see both a mysql and pgsql directory.
8. Now, we will umount the Amazon EBS volume from /temp and remount it as /vol by running the following commands:
9. If you receive the following error message when you run the umount:
It means you've changed active directory to /temp with your shell account. Run "cd" and then the commands again.
10. If everything went smoothly, you should be able to run a "df" and verify that /dev/sdf is now mounted as /vol.
11. At this point, you can restart PostgreSQL and MySQL:
12. One last thing we need to do is add the proper entries to /etc/fstab to the WOlastic instance if you ever need to "restart" the instance and have the volume properly mount.
13. From the shell prompt, run the following command:
14. You're done! Now all of your PostgreSQL & MySQL data are persistent on your WOlastic instance on Amazon EC2! You can create "snapshot" backups using Amazon S3 of your EBS volume. Just make sure PostgreSQL & MySQL aren't running at the time of the backup. In the future, I will add some 3rd party contributed XFS & Amazon EBS/AWS utilities which allows you to take "live" snapshots without shutting down PostgreSQL or MySQL by "freezing" the XFS filesystem for a snapshot.
NOTE: This guide requires the use of WOlastic 20090708 or later!
One of the major "gripes" about Amazon EC2 instances is that by default, do not handle persistent data. In other words, if you ever decide to "power-off" an EC2 instance, all data stored on the instance storage, including any changes to the 10GB root partition from your AMI, is lost. (Note: However, rebooting an EC2 instance, no data loss occurs.)
Amazon recently began offering as part of their Amazon AWS service, Amazon EBS. EBS, which stands for "Elastic Block Store", handles persistence for when "issues" arises with an EC2 instance. You can view EBS, basically, as a hard-drive that is an accessible, mountable and formattable block-device for your EC2 instance. Sizes begin as small as 1GB up to 1TB (1,000GB).
Pricing structure is relatively simple with Amazon EBS. Pricing for storage is $0.10 USD per allocated GB per month. Meaning a 100GB EBS volume would cost $10 USD/month. However on top of that, Amazon has I/O charges on top of it at $0.10 USD per 1-million I/O. Amazon uses an example of a medium-sized database which does about 100 I/Os per second over the course of a month which equates to approximately an additional $16 USD/month. So your total bill for your 100GB EBS volume would be approximately $26 USD/month.
For all of the complete features of Amazon EBS, please visit:
Creating an Amazon EBS Volume
1. The assumption of this guide is you've already launched and are familiar with the WOlastic instance environment.
2. Log into your Amazon AWS account's Console at the following URL:
3. Find the instance you wish to add the EBS volume to. Under the Navigation pane on the left, select "Instances" and find your running WOlastic instance. Select & highlight the instance so it shows you the instance details on the bottom frame. You will see something similar to this:
In this example, our WOlastic instance is running on us-east-1b.
4. Now, under the Navigation pane on the left, select "Volumes" under the "Elastic Block Store" header.
5. If you do not have any existing "Volumes", your only option will be "Create Volume", otherwise it will list your existing EBS volumes that are under your account. Click on "Create Volume".6. You will see a pop-out window asking for the following information:
- Size:
- Availability Zone:
- Snapshot:
7. In our example, we will be creating a 10GB EBS volume. You can create any size between 1GB and 1,000GB (1TB).
8. For "Availability Zone", select the zone in which your WOlastic instance is running on. In our example, it would be us-east-1b.
9. "Snapshot" isn't relevant to this guide, but basically you can restore "snapshots" of volumes which are stored on Amazon S3 (i.e. backups, pre-loaded data, etc.) Since in our case we're just creating an EBS volume to store our PostgreSQL or MySQL database, we leave this blank.
10. Click "Create" after you've filled out the relevant information.
11. Amazon AWS will at this point will create your EBS volume. Under "Status", it should show "creating". You can manually click "Refresh" to see the status of the EBS volume creation process. Once the "Status" shows available, we're ready to mount this volume under your WOlastic instance.
Mounting/Attaching an Amazon EBS Volume to your WOlastic Instance
1. At this point, we need to attach this volume to your WOlastic Instance. Select the EBS volume you just created in the AWS Management Console and click on "Attach Volume". A pop-up will occur asking you the following information:
- Volume:
- Instances:
- Device:
2. "Volume" will automatically be filled in for you. "Instances" will be a pulldown of which instances are available to have the EBS volume attach to. If you have more than one instance running in an availability zone, please double-check you're attaching to the correct instance. If it's your only instance, then the pre-filled instance ID is the correct one.
3. "Device" should default to /dev/sdf. Leave it be. Now click "Attach".
4. At this point, the EBS volume will attach itself to the WOlastic instance. Hit "Refresh" a couple times and make sure under the "Attachment Information" column that the instance ID and associated mount point shows "attached".
5. Log into your WOlastic instance via SSH, if you haven't already, as the "appserver" user. At the shell prompt, type "dmesg". The last line you see should be similar to this:
[9508565.493572] sdf: unknown partition table
6. This correctly means your Amazon EBS volume is now attached to your WOlastic instance.
7. Now, we must prepare the volume to store data so we must format a file-system. We will be using XFS. From the shell prompt, run the following command:
sudo mkfs.xfs /dev/sdf
8. You might be prompted for a password. Enter the password for "appserver". If everything goes correctly, you should see something similar to this:
meta-data=/dev/sdf isize=256 agcount=16, agsize=163840 blks
= sectsz=512 attr=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal log bsize=4096 blocks=2560, version=1
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
9. Your EBS is now ready to be used! All of the PostgreSQL & MySQL data are stored in /vol. We will now "move" the existing PostgreSQL & MySQL data off the 10GB root-partition in /vol to the new XFS-formatted EBS volume.
Migrating PostgreSQL & MySQL data to your EBS Volume
1. Run the following command to mount the new EBS volume under /temp:
sudo mount /dev/sdf /temp
2. You can run a "df" at the shell prompt to make sure it shows up mounted as /temp.
3. If you have existing data that is important, BACKUP your databases NOW! There are many guides on how to backup PostgreSQL & MySQL databases. Google it!
4. Now, shutoff all running instances of PostgreSQL & MySQL, if running. Run the following commands at the shell prompt:
sudo service postgresql stop
sudo service mysqld stop
5. If you see "FAILED", that means the service wasn't running in the first place. If it is running, you should see "OK".
6. Now we will move the data. Run the following commands (you have backups if you need them, right?) from the shell prompt:
sudo mv /vol/pgsql /temp/pgsql
sudo mv /vol/mysql /temp/mysql
7. Verify that the data has moved by running an "ls -l /temp". You should see both a mysql and pgsql directory.
8. Now, we will umount the Amazon EBS volume from /temp and remount it as /vol by running the following commands:
sudo umount /temp
sudo mount /dev/sdf /vol
9. If you receive the following error message when you run the umount:
umount: /temp: device is busy
It means you've changed active directory to /temp with your shell account. Run "cd" and then the commands again.
10. If everything went smoothly, you should be able to run a "df" and verify that /dev/sdf is now mounted as /vol.
11. At this point, you can restart PostgreSQL and MySQL:
sudo service postgresql start
sudo service mysqld start
12. One last thing we need to do is add the proper entries to /etc/fstab to the WOlastic instance if you ever need to "restart" the instance and have the volume properly mount.
13. From the shell prompt, run the following command:
sudo cp /etc/fstab.vol /etc/fstab
14. You're done! Now all of your PostgreSQL & MySQL data are persistent on your WOlastic instance on Amazon EC2! You can create "snapshot" backups using Amazon S3 of your EBS volume. Just make sure PostgreSQL & MySQL aren't running at the time of the backup. In the future, I will add some 3rd party contributed XFS & Amazon EBS/AWS utilities which allows you to take "live" snapshots without shutting down PostgreSQL or MySQL by "freezing" the XFS filesystem for a snapshot.
Leave a comment