Showing posts with label Azure SQL Database. Show all posts
Showing posts with label Azure SQL Database. Show all posts

Thursday, February 13, 2025

Resizing operations - getting notifications on Managed Instance & regular SQL DB

How to monitor the progress of a resizing operation. (see previous post for Elastic Pool changes)

Managed Instance:

  •  Get-AzSqlInstanceOperation -ResourceGroupName "myrgnamehere" -managedinstancename "myservernamehere"
Azure SQL Database, not elastic pool. 
  •  get-azsqldatabaseactivity -ResourceGroupName "myrgnamehere" -servername "myservernamehere" -databasename "dbnamehere"

Note that instead of IN_PROGRESS, for both of these the state is "InProgress". Sloppy, Microsoft, sloppy.

Oh, but at least they start reporting info shortly after it starts!


$activity = (the above)

$latest_activity = $activity|sort -Property StartTime -Descending |select -first 1 #unsure if I need this, but what the heck, still works

if ($latest_activity.state -ne "INPROGRESS" -and $latest_activity.StartTime -gt [datetime]"2025/02/13 19:00:00")

{

#notification here
}

Tuesday, February 4, 2025

Azure SQL Database Elastic Pools - fun monitoring the resize.

Turns out the Cloud is, as previously surmised, is other people's computers. And they want their two dollars. 


So let's resize! And then get a notification when it's done!


The annoying part, though, is that for the FIRST change, it doesn't tell you ANYthing until you're done. Run the command, get back an empty prompt. Seriously it was a "WTF". 

AFTER it's done, the SECOND change you can monitor. 


This code will do both. Warning that it will Just Keep Notifying you because I am lazy and just wanted SOMETHING to work. 


And the part with writehost? here's what it looks like in my job history:


Oh, and percent done appears to START at 50%. Like I started a resize on a >1tb SQL DB, and under 2 minutes it says 50%. Sure, sure. *pat DB on head* who's a good little monitor?


```

Connect-AzAccount 

$activity = @()

$activity = Get-AzSqlElasticPoolActivity -ResourceGroupName "yourgroupnamehere" -servername "yourservernamehere" -elasticpoolname "yourelasticpoolnamehere"

$latest_activity = $activity|sort -Property StartTime -Descending |select -first 1

write-host "$($latest_activity.state) is the state of the operation started at $($latest_activity.starttime) and pct is $($latest_activity.PercentComplete)"

#that is so that when I run it with sql server (VM), I can get back the results of the powershell script. 


if ($latest_activity.state -ne "IN_PROGRESS" -and $latest_activity.StartTime -gt [datetime]"2025/02/04 21:13:00")

{

notificationcodehere
}

```

Friday, October 20, 2023

Azure SQL Database - ports you didn't realize you needed for your firewall request.

 Azure is fun. FUN, I say!


Ports: 

1433 - standard SQL port

11000-11999 - ports if you're using Proxy instead of Redirect

14000-14999 (!) - ports for the DAC (Dedicated Admin Connection)

Tuesday, October 17, 2023

Azure Data Factory - how to ACTUALLY use storage event triggers to copy data from Blob Storage to Azure SQL Database (Azure SQL DB)

 This post is in response to an incredibly frustrating interaction with Azure.

I figured I would do something incredibly simple - import a parquet file from blob storage into Azure SQL DB. When the file is written to Blob Storage (using CETAS, fwiw), it should activate the storage trigger, which should kick off the pipeline. 


The problem is that it won't work as the tutorial (https://learn.microsoft.com/en-us/azure/data-factory/tutorial-copy-data-portal) shows. It says it can't find the file. Why not? Well, it wasn't firewalls or anything else. It's that a linked service dataset, in "Sink" requires a container name - and the trigger sends the container name also. So I kept getting messages that it couldn't find \mycontainer\mycontainer\myfolder\myfile.parquet . I found a single message on Stack Overflow (https://stackoverflow.com/questions/67294521/azure-data-factory-how-to-get-new-file-location-from-triggerbody-folderpath/74738786) on how to solve it. So, here we go: 

package parameters:





Source:


@pipeline().parameters.trigger_folder_path

@pipeline().parameters.trigger_file_name


My Source Dataset:







Sink:







And the last part in the trigger that makes it all work:

Here's what to put for trigger_folder_path, as per Steve Johnson on SO. No, you don't have to change the YAML, you can modify the values above: 
@substring(triggerBody().folderPath,add(indexof(triggerBody().folderPath,'/'),1),sub(length(triggerBody().folderPath),add(indexof(triggerBody().folderPath,'/'),1)))


Why it's that hard to find/use, I have no idea. 
All Hail Steve Johnson!

Friday, October 28, 2022

Azure SQL Database - something like sp_send_dbmail

 I needed the ability to send emails from Azure SQL DB, which doesn't support it. The long-term goal is to use LogicApps or SendGrid or something like that, but while I wait for permissions and the like, let me share this instead. It acts like sp_send_dbmail, and will even run queries in the database (or other databases on that server!) and send them from your on-prem server.

This is based on others. They had the awesome parts, my parts are the crappy glue making it work. : )


The emailqueue table looks like this:

CREATE TABLE [dbo].[EmailQueue](
[Id] [INT] IDENTITY(1,1) NOT NULL,
[Recipients] [VARCHAR](250) NOT NULL,
[Cc_recipients] [VARCHAR](250) NULL,
[Email_Subject] [VARCHAR](250) NOT NULL,
[Email_body] [VARCHAR](MAX) NULL,
[Email_body_format] [VARCHAR](10) NULL,
[Query] [NVARCHAR](MAX) NULL,
[profile_name] [VARCHAR](250) NULL,
[QueueTime] [DATETIME2](7) NOT NULL,
[SentTime] [DATETIME2](7) NULL,
[Importance] [VARCHAR](10) NULL,
[attach_query_result_as_file] [BIT] NULL,
[query_attachment_filename] [VARCHAR](200) NULL,
[query_result_header] [BIT] NULL,
[query_result_width] [INT] NULL,
[query_result_separator] [VARCHAR](2) NULL,
[query_result_no_padding] [BIT] NULL,
[bcc_recipients] [VARCHAR](255) NULL,
[execute_query_database] [VARCHAR](255) NULL,
[from_address] [VARCHAR](MAX) NULL,
PRIMARY KEY CLUSTERED 
(
[Id] ASC
)WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, OPTIMIZE_FOR_SEQUENTIAL_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
ALTER TABLE [dbo].[EmailQueue] ADD  CONSTRAINT [Co_Importance]  DEFAULT ('Normal') FOR [Importance]
GO
ALTER TABLE [dbo].[EmailQueue] ADD  DEFAULT ((0)) FOR [attach_query_result_as_file]
GO
ALTER TABLE [dbo].[EmailQueue] ADD  DEFAULT ((1)) FOR [query_result_header]
GO
ALTER TABLE [dbo].[EmailQueue] ADD  DEFAULT ((0)) FOR [query_result_no_padding]
GO


And the SP: 

CREATE PROCEDURE [dbo].[log_dbmail] 

   @profile_name               sysname    = NULL,        

   @recipients                 VARCHAR(MAX)  = NULL, 

   @copy_recipients            VARCHAR(MAX)  = NULL,

   @blind_copy_recipients      VARCHAR(MAX)  = NULL,

   @subject                    NVARCHAR(255) = NULL,

   @body                       NVARCHAR(MAX) = NULL, 

   @body_format                VARCHAR(20)   = NULL, 

   @importance                 VARCHAR(6)    = 'NORMAL',

   @sensitivity                VARCHAR(12)   = 'NORMAL',

   @file_attachments           NVARCHAR(MAX) = NULL,  

   @query                      NVARCHAR(MAX) = NULL,

   @execute_query_database     sysname       = NULL,  

   @attach_query_result_as_file BIT          = 0,

   @query_attachment_filename  NVARCHAR(260) = NULL,  

   @query_result_header        BIT           = 1,

   @query_result_width         INT           = 256,            

   @query_result_separator     VARCHAR(2)       = ' ',

   @exclude_query_output       BIT           = 0,

   @append_query_error         BIT           = 0,

   @query_no_truncate          BIT           = 0,

   @query_result_no_padding    BIT           = 0,

   @mailitem_id               INT            = NULL OUTPUT,

   @from_address               VARCHAR(MAX)  = NULL,

   @reply_to                   VARCHAR(MAX)  = NULL

AS 

INSERT INTO dbo.EmailQueue

(

    Recipients,

    Cc_recipients,

Bcc_recipients,

    Email_Subject,

    Email_body,

    Email_body_format,

Query,

Execute_Query_Database,

attach_query_result_as_file,

query_attachment_filename,

query_result_header,

query_result_width,

query_result_separator,

query_result_no_padding,

    profile_name,

    QueueTime,

    SentTime,

Importance

)

VALUES

(   @recipients,            -- Recipients - varchar(250)

    @copy_recipients,          -- Cc_recipients - varchar(250)

@blind_copy_recipients,

    @subject,            -- Email_Subject - varchar(250)

    @body,          -- Email_body - varchar(max)

    @body_format,          -- Email_body_format - varchar(10)

@query,

@execute_query_database,

@attach_query_result_as_file,

@query_attachment_filename,

@query_result_header,

@query_result_width,

@query_result_separator,

@query_result_no_padding,

@profile_name,          -- profile_name - varchar(250)

    CAST(SYSDATETIMEOFFSET() AT TIME ZONE 'Central Standard Time' AS DATETIME), -- QueueTime - datetime2

    NULL,           -- SentTime - datetime2

@importance

    )



The powershell that pulls it all together: 

Wednesday, April 20, 2022

Xquery - a surprising difference between Azure SQL Database and Azure Managed Instance with Quotes

I'm moving processes into Azure.  Given that it's me, probably poorly. But I'm using this as an opportunity to improve things where possible, specifically looking at moving some processes to Azure SQL Database. 
Why? I love them for several reasons: 

  • Simpler
  • Cheaper
  • Quick to create
  • Quick to modify the instance (seconds, not hours)


Yes, there are still gotchas (Elastic Jobs, no cross-db queries, no CLR, can't replicate from it), but overall? Soooo nice.

---

I was trying to get some Xquery parsing working, and ran into a weird error:
"CONDITIONAL failed because the following SET options have incorrect settings: 'QUOTED_IDENTIFIER'"

Weird.  Doubly so because this code has worked on-prem for a decade, and the Managed Instance is running it now.  And because the SP specifically has a SET QUOTED_IDENTIFIER ON; 

Okay, let's take alook online.


Yeah, that's the problem.  And indeed, my code has similar xquery to the example:
set @myval = @xml.[value]('(*/A[B="C"]/D[E="F" or G="H"]/I)[1]', 'varchar (35)');

What's the fix? Use two single-quotes instead of the double quote.
set @myval = @xml.[value]('(*/A[B=''C'']/D[E=''F'' or G=''H'']/I)[1]', 'varchar (35)');