Data Engineers Life

What is SQL Server Management Studio (SSMS) In our previous blog post we looked at how to easily create a table using Azure Data Studio. After a quick installation and set up, we were able to run SQL queries against our Azure SQL server and create the Staging, Archive, and View in our SQL Database. If you missed that post please follow this link to review it:  Creating a table in Azure Database using Azure Data Studio In this blog post I will show you how to quickly set up SQL Server Management Studio (I will refer to it from now on as SSMS), configure it to point it to our Azure SQL server, and then run the CREATE TABLE and CREATE VIEW scripts that we did through Azure Data Studio.

What is Azure Data Studio In our previous post we create a table using the Query Editor, which is a tool provided within the Azure portal. Today I wanted to add a short post on how to do the same thing while using Azure Data Studio. Azure Data Studio is a free tool that is provided by Microsoft and it is an editor that allows you to connect to a number of different sources and run your queries against them. All you would need to do is install the tool and then set up the connection to your Azure Database (or Azure Date Warehouse).

In our previous post we have gone through the steps of creating an Azure SQL Server and attaching an Azure SQL database to it. If you have not done so please refer back to my previous post " Blob Storage to Power BI part 3: Creating an Azure Database ". So now that we have a database running in our Azure environment it is time to start using it. This post will focus on how to create Database tables through the Azure portal. I will then create two supplemental posts on how to create tables in both Azure Data Studio and SQL Server Management Studio (SSMS). This is to show you how each one feels and let you choose which one you prefer. Overview So before we begin let me summarize the three options you have to create your tables: Query Editor (available in Azure portal) Azure Data Studio SQL Server Management Studio (SSMS) All three will do the same thing, but each one is a little different. That is the beauty of it, you can use which ever one you would prefe

Welcome back to my blog post and today we will go through and create the target system where the files in our Blob storage reside in will be sent to. What we are going to do is go back into Azure portal and from there I will show you how to spin up two things: Azure SQL Server Azure SQL Database We will have to spin up the SQL server first, and then we will create and attach a database to it. One thing to keep in mind is that Databases that are attached to a different SQL server will not be able to communicate with each other. So if you are going to be attaching other Databases that you want to keep separate, but in the future will want to share data between each other, then you will have to create them under one SQL Server.  There is no right approach since it depends on your circumstances. For our purposes we will only be creating one SQL server and one SQL Database in Azure. So without further delay lets get started. So first things first, let us go back to ou

Now that we have created our Blob storage account it is now time to upload some data to it. I will show two ways on how to do this, one way will be using the Azure portal and the other one will be using a free tool provided by Microsoft called Azure Storage Explorer . First lets make sure we have the following items: Blob storage account If you are missing the above please read over my previous post Blob Storage to PowerBI part1: Creation of Blob Storage . In that post I outlined the steps needed to create a Blob Storage account. The other thing that I needed to mention is that if you are looking to change the theme of your portal from white and blue you can click on the little gear in the top right corner. That will bring up the settings and you can choose from the different themes that Azure offers. Hello darkness my old friend. With that out of the way lets continue. So first step is to click on the storage account that you have already created.

To start our Azure journey the first step that we would need to do is create a Blob storage account. You can think of Blob storage as a massive repository that will house any number of files. Microsoft provides the following definition: Microsoft definition for Azure Blob storage For a more detailed answer please check out the Introduction to Azure Storage document written by Microsoft. For our purposes we can think of Blob storage as the account that will store the files that our business users will create and will also be used as a source for our Azure Data Factory pipeline. So lets get started!!! First log into your Azure portal. You should see the following screen: First page you will see when logging into Azure After that you can do one of two things; either click on Storage Accounts or type in storage accounts in the search box at the top of the screen. Both will lead you to the same place, but search is useful if you are looking for a specific service tha

One of the things that I came across while working in Azure cloud space is that projects do not need to be overly complex to be useful to a business organization. In fact, by completing small projects a Data Engineer can build confidence with different business units, which in turn will lead to more projects in the future as they get more and more comfortable with using different Azure services. This is why I wanted to document a small Use Case that I had worked on. While this project is simple (and involves minimal transformations) it does illustrate that even simple projects will contain a number of tools that need to work in sync. This Use Case came from a user that wanted to see if they could create a pipeline that would take different csv files, and move them into a single database table, which will then be fed into a PowerBI report. At the time each department had a csv file which they managed and edited, then emailed. This email was then opened by the PowerBI develop