Storage and Permissions

In this lab you can gain knowledge and practice the following topics:

Creating and using a Storage Bucket - Cloud Storage Bucket.
Versioning & Lifecycle.
IAM - Users and Permissions.
Service Accounts.
Mounting a Storage Bucket as a Drive on a Linux Server.

Storage Bucket:

One of the most common storage configurations in the cloud is file storage in a bucket.

Traditionally, storage was a disk attached to a server/computer, and to use the disk you need some kind of file system.

A storage bucket is called that because it does not use a file system. All files go in as binary files in bulk, and management is done through an index.

The human user can manage their files and the bucket in a folder structure, but these are not folders like in a file system. In terms of storage, the folders are just an addition to a string in the file’s Metadata. When a user requests to view files in a specific folder, the index finds files whose Metadata contains the “folder” name as part of the string.

Creating a Bucket:

In the GCP management menu (hamburger icon) go to Cloud Storage, Buckets. On the page that opens click CREATE at the top.

Give it a unique name - say, yos-storage. Continue. Since no two buckets in the entire global GCP infrastructure can have the same name. Each bucket requires a new name that no one in the world has yet taken.

Select Region - me-west1. CREATE.

A confirmation dialog will pop up, confirm without changing anything. The bucket is created and its page opens.

Right now we see the OBJECTS tab view which manages the files themselves. You can see that there are additional tabs related to bucket management.

Using the Bucket:

Start by uploading a simple file.

Create a file on the desktop of your computer, named doc.txt. Inside the file write the word test, save it, and upload the file to the bucket.

Now we have one file in the bucket.

If you click on the file to get its details, you will see that the file has a URL through which you can get the file content via HTTPS, or a path that can be used in the CLI.

Click the back arrow at the top to return to the bucket, and switch to the PROTECTION tab.

You can see that OBJECT VERSIONING is OFF. Clicking the blue line will pop up a confirmation dialog, click CONFIRM.

In many cases a window will pop up for configuring Versioning, where you can choose how many versions to keep, and how long to keep older versions.

If it doesn’t pop up and just returns to the PROTECTION page, you can see that OBJECT VERSIONING is ON, but below there is a message saying there are no rules on the versioning. Which means that every time a file is uploaded or deleted, the file from the previous version will remain forever.

This way everything accumulates and takes up storage costs.

So we will define 2 rules for older versions.

The First Rule:

Click MANAGE RULES.
Click ADD A RULE.
Select Delete object, then CONTINUE.
Now under Set Conditions select Number of newer versions.
In the text field enter 3.
Click CREATE at the bottom.

The Second Rule:

Same as the first rule, except in step 4 the condition to select will be Days since becoming noncurrent, in the text field enter 7 days and then create the rule.

Lifecycle and Storage Classes:

Storage buckets offer several storage classes, where the standard class applies to frequently used files - files that a user or software is constantly uploading/downloading/deleting. But there are files for long-term storage that are used only occasionally.

On one hand - the less frequently a file is used, the cheaper the storage class it can be stored in.

On the other hand - the cheaper the storage class a file is kept in, the higher the retrieval cost on those rare occasions when access is required.

Therefore to achieve cost-effective storage, you need to calculate the frequency of file access and find an appropriate storage class accordingly.

Lifecycle rules allow managing components within a storage bucket.

You can define several rules with different conditions, and thereby determine that certain files will be deleted after a period of time, while other files will be moved to archive storage in a cheaper class after a month or more.

While we were working on file Versioning we created a rule that deletes older versions. Now we will create a rule that moves files three months old and older to archive.

Defining a Lifecycle Rule:

Enter the bucket page in the management console - Cloud Storage >> Buckets >> yos-storage
Switch to the LIFECYCLE tab
Click ADD A RULE
Select the option Set storage class to coldline
Click CONTINUE
Check Age and in the field that opens enter the number 90
Check Storage class matches and in the options that open check Standard
Click CREATE at the bottom

Now files that remain in the bucket after 90 days move from standard storage to a cheaper storage class.

All of this assumes that access to such older files happens only rarely. If there is still frequent access to these files, it will not be cost-effective to move them to cheaper storage because there are retrieval costs every time you access a file.

Working with Versions and gsutil:

Until now we have used gcloud commands, now we will get acquainted with gsutil commands which are used for accessing storage.

To display the bucket contents in the CLI open PowerShell and enter the command:

gsutil ls gs://yos-storage

Or display the file content:

gsutil cat gs://yos-storage/doc.txt

Open the file we created earlier on the desktop and add another line to it with the text “test”. Save and close.

Note that the gsutil tool uses Linux commands for file handling. So if we want to use it to upload a file to the bucket, the command looks like this:

gsutil cp $env:userprofile\desktop\doc.txt gs://yos-storage

Now refresh the bucket page, and we will see that there is still one file there, but in the version history column we can see that there is one older version.

Clicking on the file and switching to the VERSION HISTORY tab will show us both versions of the file, and the option to restore the older version that was overwritten by the new one.

You can also copy the older version, download it, or move it to another location.

Using the Bucket from a Linux Machine:

Start instance-1 (as we created in lab number 1) and connect to it via SSH. Then enter the following command:

gsutil ls gs://yos-storage

You can see the file that was uploaded earlier.

Now we need to upload a new file. First create it:

echo "test2" >> doc2.txt

This command adds the text “test2” to the file doc2.txt. If the file does not exist, the command creates it.

Enter the ls command to display the files in this location, and we will see that the file was indeed created.

The command cat doc2.txt will display the file’s content and this way we verify that what we wanted is in the file.

And now copy the file to the bucket:

gsutil cp doc2.txt gs://yos-storage

… The command returns HTTP code 403 saying there are no permissions.

Permissions and Roles in GCP:

Why did it work from the home computer in PowerShell earlier but not now?

Use the following command to display the accounts activated on this machine for authenticating against GCP:

gcloud auth list

And there we see there is only one authenticated account - the project’s built-in service account for running virtual machines (Compute Engine default service account).

If we enter the same command in PowerShell, we will get our Google account as the result, the one we used to register for GCP.

In lab number 1 we performed an authentication process against GCP to authenticate the user for PowerShell on the home computer. Therefore commands from PowerShell receive our permissions. Since we created the projects, we have Owner permissions - which is the highest permission. Under our permission, any action can be performed. But the service account running the Linux machine does not come with permissions for storage buckets. We need to grant it those permissions, or do something else.

Granting permissions to the Compute Engine default service account grants those same permissions to every machine running under that account. Therefore it is more correct to create a dedicated service account for machines that need the specific permission, and run the machine that needs the permission using the dedicated account.

Creating a Service Account:

In the main menu go to IAM & Admin >> Service Accounts.
Click CREATE SERVICE ACCOUNT at the top.
Give it a name - vm-access-bucket, and click CREATE AND CONTINUE.
Now we need to grant the account permissions. Click on the menu where it says Select a role.
In the Filter line at the top, type Compute Viewer and from among the results find and select Compute Viewer. Click CONTINUE.
Now this is the last step, and it is less relevant for us.

In this step you can add names of groups or users who have permission to run actions through this account, or to manage it. Managing it means issuing authentication keys for it, etc.

We will skip this and perform a similar action later.

Click DONE.

Connecting a Machine to a Service Account:

Enter the machine’s page in Compute Engine >> VM instances.
Shut down the machine (if it is running).
Enter the machine’s settings for editing by clicking EDIT at the top.
After the disk settings, there is Security and access. There one of the settings is called Identity and API access. You can see that Compute Engine default service account is configured there. Click the menu arrow on the right, and select the service account we created earlier - vm-access-bucket.
Click SAVE.
Start the machine.

Granting Permissions on the Storage Bucket:

Return to the bucket we created earlier. Using my example, go to Cloud Storage >> Buckets >> yos-storage.
Switch to the PERMISSIONS tab.

You can see here default settings, which all come inherited from the project - those with Editor or Owner permissions in the project receive similar permissions on the bucket. We will do something more targeted following the Principle of Least Privilege.
Click GRANT ACCESS.
In the field that says New principals write the full address of the service account we created - vm-access-bucket. You can find this under IAM & Admin >> Service Accounts. Hovering the mouse over the account’s “email” displays a copy button on the side. Paste that into the field above.

You will immediately see a selection option pop up with the account’s full name. Select it and you will see the email marked as a tag in the text field.
Click with the mouse on the field that says Select a role, and a panel with a list of Roles to choose from will pop up. In the Filter field type Storage Legacy Bucket Writer.

You can also choose Storage Object Admin but that is a slightly stronger permission, and we will choose the more limited one.
- You can see next to each Role in the list a brief description of the permissions it grants.
Click SAVE.

Testing:

Now connect again to Instance-1, find the doc2.txt file and prepare to upload it to the bucket.

First, verify the permissions we are working with:

gcloud auth list

This command displays the accounts that have been authenticated on this machine, and which one is currently active. We will immediately see that the active account on this machine is the service account we created and granted permissions on the bucket.

Now we can give the command again to upload the file to the bucket:

gsutil cp doc2.txt gs://yos-storage

And we can see that the command passed successfully.

We can verify and display the files in the bucket using the command:

gsutil ls gs://yos-storage

And this is what it looks like:

Displaying files from a bucket

Users and IAM:

Earlier we saw how permissions at the project level filter down to permissions at the level of a specific resource, and we also learned how to give permissions to a specific user on that specific resource.

Principles of Least Privilege favor granting permission at the resource level - unless it is a user (or group of users) whose role is to access all resources of this type in the same project.

To maintain order and prevent unnecessary complications, permissions are usually given to groups rather than to a specific user. But groups are not available for a single project, only for an organization, so we will not touch on that.

Now we will see how to grant permissions to the service account on all buckets in the project:

Go to the general users page in IAM - IAM & ADMIN >> IAM
See in the list the vm-access-bucket account, and also see that it has the Role given to it when the account was created - Compute Viewer
Click the pencil on the right side of the row, and a page will open for editing permissions
Click ADD ANOTHER ROLE
Hovering the mouse over the field that opens will pop up a Roles search panel
Now we can grant the account any permission at the project level. We could give it Storage Object Admin permission (the Legacy permission we gave at the bucket level is not available at the project level), or give it other permissions on VM, on networks and firewall, and more.
Click CANCEL to close the permissions window and return to the users list.
Click GRANT ACCESS at the top
In the field under Add principals you can put any email address that Google recognizes. Including another Gmail address we have, a sibling’s or friend’s address, or anything else.
In the Select a role field you can choose and grant that Google-recognized account the permission, as we saw earlier.

Mounting a Bucket to a Machine as a Storage Drive:

You can mount a storage bucket directly to a Linux machine, just as you would connect a disk. This solution only works on newer versions, such as Ubuntu 18.04 and above, or CentOS 7 and above.

Google has not yet provided an officially supported equivalent solution for Windows machines, so a workaround is needed.

Since instance-1 runs on a Debian operating system, we will use commands appropriate for this Linux distribution. In other cases you can look in Google’s Docs or in other articles online to get different installation instructions.

Installing GCSFuse:

Installing the basic Fuse component:

sudo apt-get install fuse

Adding the GCSFuse installation and update repositories to the operating system’s software sources:

export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb https://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

Updating the system with the values added, and installing the GCSFuse component:

sudo apt-get update
sudo apt-get install gcsfuse

Mounting the Bucket as a Drive on the Linux System:

First you need to create a directory where we can mount the bucket:

mkdir bucket

Then mount the bucket there:

gcsfuse yos-storage bucket

The command is built in three/four parts (the fourth part is optional):

The command name - gcsfuse.
Optional - Flags, flags/properties you can specify for the mount (we will see this later).
The bucket name (no prefix like gs:// needed).
The directory where the bucket is mounted.

Now in the bucket directory you can see the bucket’s contents. This is what it looks like:

Mounting a bucket as a storage drive

You can now create folders and files within them:

Creating files inside a bucket/storage drive

Persistent Mounting of the Bucket as a Drive on the Linux System:

The gcsfuse command we just ran to mount the bucket to the bucket directory is the equivalent of the mount command used in Linux to place drives and partitions at specific locations in the system. Both commands are valid as long as volatile memory is active, and once it is cleared everything resets.

So if we restart instance-1 we will see that after the restart the mount is gone.

This is how you can ensure persistent mounting of the bucket in a certain directory, by adding the following line to the fstab file:

yos-storage /home/yosyos/bucket gcsfuse user,rw,allow_other,_netdev,file_mode=777,dir_mode=777,exec

Information about fstab, persistent mounting of storage media on a Linux server, and the syntax of the Flags that make up the command, appears in the appendix.

Appendix - Persistent Mounting of a Bucket as a Drive on a Linux System:

Persistent mounting of various storage media - ranging from partitions on physical (or virtual) disks connected directly to the system, to storage media accessible over the network such as shares through Samba or NFS and more, is done through the fstab file.

This file sits in the etc directory and contains mount commands. When a Linux system boots, the system uses the file to connect to drives. Therefore corrupting this file may cause the system not to boot the next time the machine is powered on/restarted, because the system will fail to connect to the partitions where it stores operating system files.

Proceed carefully through the steps, and verify everything is correct before allowing yourself to restart.

Those who are still inexperienced should take a snapshot before making the changes, in case the machine crashes. Still, anyone who carefully follows the instructions will be fine.

Anyone who has not restarted instance-1 needs to first unmount the current mount to see that the following operations apply changes:

umount bucket

Now you can verify that the mount was removed. Either by reviewing the system’s file systems and mounts with the command:

df- h

Or simply review the bucket directory contents and see that there is nothing there:

cd ~
ls bucket

The bucket directory is empty. You can mount again, this time persistently.

Open the fstab file for editing:

sudo nano /etc/fstab

Go down to an empty line in the file and enter the following:

yos-storage /home/yosyos/bucket gcsfuse defaults 0 0

Close and save the file using ctrl + x, then y.

We don’t want to restart the system right away, because perhaps we made a mistake writing the command. In that case the system might not boot. We want to check everything is correct first.

With the following command we will run the fstab file proactively, and this way we will see if there are any problems:

sudo mount -a

But now if we try to access the bucket directory contents - using the ls bucket command, we will get a message that we do not have permission to access it.

What happened?

Let’s go over the command format in the fstab file:

First put the location that Linux uses to find the file system in question. This can be a physical location inside the machine, such as a disk or partition - identified by partition name or by UUID.
Then put the location where you want to mount the file system.
Then put the type of file system being accessed, so Linux knows how to address it and how to load it.
Then there is space for the mount configuration. This is where the Flags go. In many cases you can leave defaults, which is what we have done until now.
Here there is a numeric value that was historically used for backup configuration. Today this is not used and the value is always 0.
Here there is another numeric value that refers to the priority order for mounting disks. For example - the system partition for operating system files should get the number 1, and disks or partitions that come afterward get consecutive numbers. The number 0 indicates no special order, which is how most cases are handled.

Notes:

An error in the numbering of item 6 will cause an error when loading the operating system. Notice in the file that only the system partition is numbered with 1.
Other errors can also cause operating system failures.

Values - Flags:

Now we address item 4, which in the command we entered received default values. One of them says that only whoever mounted the file system gets access to it. Since the file system was mounted via fstab, the one who mounted it was effectively the root user. So our user has no access to it.

So we will edit the mount command in the file, and this time instead of default values we will enter some different Flags:

yos-storage /home/yosyos/bucket gcsfuse user,rw,allow_other,_netdev

We omitted the numeric values at the end.

Now you can view files in the directory, but not create files or folders. By adding more values to the command you can do everything:

yos-storage /home/yosyos/bucket gcsfuse user,rw,allow_other,_netdev,file_mode=777,dir_mode=777,exec

The Values Entered:

user - allows a regular user to mount the file system.

rw - allows read and write to the mounted file system. As opposed to ro which allows read only.

allow_other - allows a regular user to access a file system that someone else mounted.

_netdev - marks that the file system being mounted is only accessible over the network, and therefore should be mounted only after the network card is available. If you do not mark this when needed, the operating system will fail to boot. Relevant when mounting various types of network shares - as in our case.

file_mode - sets the permission on files within the file system. In our case we set 777, which gives access to the files to every user connected to the machine.

dir_mode - sets the permission on directories within the file system. In our case we set 777, so new files can be created in directories by any user connected to the machine.

Exec - sets that files from the mounted file system can be executed.

Notes:

The order of Flags matters, because some Flags when placed, set additional Flags as defaults. Therefore if you want to choose a different option, the Flags that change the default values must come afterward.
For example - user sets noexec as its default, so in the command we used, we placed the user Flag first and then the Flag that says exec, to ensure there are permissions to execute files from within the bucket.
Because of this, in cases where you specify a set of Flags and one or more are not working, check in the Linux mount command Docs whether one of the later Flags does not conflict with one of the earlier ones. In such a case you need to change the order in which you place the Flags in the command, until you get the result you want.