Blob Storage to PowerBI part 2: Uploading data to Blob storage

Now that we have created our Blob storage account it is now time to upload some data to it. I will show two ways on how to do this, one way will be using the Azure portal and the other one will be using a free tool provided by Microsoft called Azure Storage Explorer.

First lets make sure we have the following items:

1. Blob storage account

If you are missing the above please read over my previous post Blob Storage to PowerBI part1: Creation of Blob Storage. In that post I outlined the steps needed to create a Blob Storage account.

The other thing that I needed to mention is that if you are looking to change the theme of your portal from white and blue you can click on the little gear in the top right corner. That will bring up the settings and you can choose from the different themes that Azure offers.

Hello darkness my old friend.

With that out of the way lets continue.

So first step is to click on the storage account that you have already created. This can be done either through the Storage account icon from the home page or by using search. Once you are there click on the Blob storage account (dataengineerslifeblob in our example).

The storage account menu where you will be creating your first container.

Once you see the window pane on the right pop up you will be presented with a couple of options. It would take several blog posts to go through all of the options but what we will be focusing on is the "Containers" icon, which we will be using to create our first Blob Storage container. That container will then house our Blob files and folders.

If you want to learn more about the other different storage types that Storage account offers I highly recommend the free Azure course called: Survey the services on the Azure Data platform. The section called "Understand data storage in Azure Storage" offers a good overview of all the options available within a storage account.

Once you click on containers you will be brought to another screen where you can create them:

Click on the container button. In that menu enter your container name and leave the public access level Private (which is the default setting).

Almost there

You should now see your container in a couple of seconds:

Click on the container name and now we will start uploading data into it. Click on the Upload button and then select the file that you want to upload. After a few seconds you will see that file in the Blob Container:

Step 1

Success!!!

One of the things you will notice is that when uploading data directly though the portal is that you don't have the ability to create folders. The way Containers work is that the container itself should be the folder. But have no fear, if you still wish to organize your data within a container based on folders then you would need to install a free tool called "Azure Storage Explorer".

Azure Storage explorer is a data management tool that allows you to view multiple storage accounts. Click on the link, select your operating system and then click on the download button. Follow the installation instructions and the install should not take more than a couple of minutes.

Once installed open Azure Storage Explorer and we will then proceed to add the Storage account that you have just created. Luckily that option is the very first thing that you see when you open it.

First pop up you will see first time you open Azure Storage Explorer

Select Azure and then click Next. You will then be greeted with the Azure authentication screen where you will have to enter your userid and password that you used to log into the Azure Portal.

We are almost done. I promise.

After that your id will be authenticated and this can take a few seconds up to a couple of minutes.

You will then see your Subscription group and resource group in the top left corner. This is what you want to add so click on Apply.

Once your subscription group is added you will now be able to see all of the storage accounts that are linked to it. If you are in an organization with multiple subscription groups then all you would need to do is be added to them by your infrastructure or admin team. 

In any case for the purposes of this POC find your subscription group and then click on the drop down arrow. This will bring up all of the storage accounts that are linked to this group. I then find my "dataengineerslifeblob" account and click the drop down arrow. Follow this up by also clicking the drop down button for the Blob Container and then you will see all of the containers that you have created. Select one and you should see all the files pop up on the right hand side.

We are in the final round of creating a folder in a blob container.

Now since we are using Storage Explorer you can click on the "New Folder" button to create a folder. However, folders in Blob containers work differently where they are not "created" until there is a file uploaded to them. Thus if you want to create a folder, but don't have the files to upload to it, you will need to put in a dummy text file (or any kind of file) as a placeholder. As soon as all files are deleted from that folder then the folder will disappear. 

One of the cool things that you can do is that you can upload the entire folder into Blob storage account instead of doing it one file at a time. To do this click on "Upload" and then select "Upload a folder". This will bring up the upload menu.

As you can see I first select the folder that I want to upload. That folder can have multiple folders under it, or just a single file. I then leave Blob type as default. For the Destination directory I leave that default as well since we don't have any nested folder structure yet. I would only update this if you wanted to upload this folder under a different folder that has already been uploaded.

Click on the Upload button to start the process. Depending on the size and number of files this can take a few seconds or a couple of hours. The sample size that I have is very small so mine finishes in just a couple of seconds.

Finally!! The files have been copied to our Blob storage account.

If you look at the bottom half of Azure Storage Explorer you will see a tab called "Activities". When the upload process first begins you will see a message of "Transferring" and it will point to the directory where the files are coming from. Since my files are very small, it took no time at all for it to show a green check mark to signify that that files were copied over without any issues. If there was a much larger number of files you could click on the drop down arrow next to it and see which files are currently being copied and if there were any errors during the copy process.

And that is it. If you click in the folder you should see the files that have been copied. 

All the files that were in the folder were copied. If there were additional folders then their hierarchy would have been copied in addition to the files that they contained.

Now that we have created our source and uploaded our files to it, I will show you how to set up an Azure Database. This will set up where we want our data to go.