Showing posts with label sql server. Show all posts
Showing posts with label sql server. Show all posts

Friday, June 4, 2021

EXECUTE AT against Athena via Linked Server and "does not support the required transaction interface"

 TLDR - using EXECUTE AT with Athena via linked server.  I used https://www.mssqltips.com/sqlservertip/6191/query-aws-athena-data-from-sql-server to get it set up and it works!  But there are some issues.



1) Partition your tables in Athena.  Better performance if you have a decent amount of data

2) This means you can't just do "select from <linkedserver>...tablename", as that doesn't tell Athena to do partition elimination

3) Once you're using OPENQUERY, if you run into issues because it doesn't know what to expect, you can use CAST or CONVERT to force metadata (aka put "cast(fielda as varchar(500))" if it thinks the field is only 255 characters.

4) That doesn't help with over varchar(8000).  You need to move to EXECUTE AT instead, but that seems better then OPENQUERY as it appears to just pass the data back - I got rid of my CASTs in the athena query. Plus, dynamic SQL is easier! You will have to enable RPC OUT for the Linked Server.

5) I couldn't do INSERT INTO EXECUTE AT until I disabled Distributed Transactions in that Linked Server.



Update 2021/06/04 - and there's something screwy with EXECUTE AT.  It runs without the CAST statements and doesn't complain , but doesn't appear to save everything - I have a handful of rows (out of 100k, mind you) that don't match up - one of the fields is lopped off. It's the most annoying thing, since my other option is SQL Workbench which WORKS but isn't easily automated.  After futzing with it for a bit, I wound up moving over to a pure Powershell solution - I'll post that in a few.

Wednesday, October 17, 2012

Master of all I survey - using Event Notifications to track code changes and more across multiple SQL servers

Please check here if running my code.  I will post updates as I find/need them.  Google docs is current.
Latest version: 11/14/2012.

(last update 2013/07/31 - looks like it's alive and well in 2014!)

----------------------------------------------

Howdy!

I specifically went with a grandiose title to get people interested in this - it's (Event Notifications) a wonderful feature that very few people use (or even know about!), and now that I use it I find it absolutely vital.  I hope I get across how great this feature is, especially since it works from SQL Server 2005 - 2014.

Below is a link to my slide deck, demos, scripts and reports.
https://docs.google.com/folder/d/0B3a287PS_UJIcnY3Q1pvX3p1eEE/edit

The end result is a table that keeps track of code changes across potentially hundreds of servers.  I've got it running on more than a dozen 50 servers, and am slowly adding them to every server in our environment.

One of the takeaways: once you have it running, you can run the following report in SSMS - right-click on an object, choose the RDL above via "Custom Report...", and it'll show details.  Changes, index maint (if a table), etc.  Works on SP/fxns, tables, databases, and servers - click on an object, it'll provide details on that object.  Click on a DB, it'll give you all changes for that DB.  Click on Server...same thing.
This works in SSMS 2005/2008/2012.  And let me state for the record, it sucks that you have to develop it in BIDS 2005 in order for SSMS 2008 to view it properly.

I'll be tweaking this report; it's bare-bones and not particularly pretty, but is a really handy way to track down changes.

A couple examples: (apologies for the size, but otherwise you can't easily see the commandtext which shows the actual commands executed - click to enlarge)















In order for the reports to work in SSMS, you will need to create a linked server on each monitored server, pointing at the server that holds EventNotificationRec, with the name ENRepository (collation compatible true and RPC OUT on).  This way when you right-click on an object it will send the server/database/objectname across on each server - this is needed because SSMS does not let you use a different data source.

The version of the report shown this evening is EN_ReportSP.  I will be tweaking it, as well as the other code.  Watch this space.

Please feel free to use, share, improve.  Please contact me for commercial use (hey, a guy can hope).

Wednesday, September 12, 2012

[Powershell] Building a dm_os_wait_stats repository

(update 2013/06/13: I've superseded this with a new framework that will run almost any piece of code, powershell or sql, saving the results to a table, and running with multiple threads. Please give it a look!
http://thebakingdba.blogspot.com/2013/04/servers-extensible-powershell.html )

Inspired by Paul Randall's talk at PluralSight on Waits & Queues (highly recommended), I've built a repository of dm_os_wait_stats by cobbling together some other people's code.

Lots of code here, but for you it's a matter of copy & pasting 4 files, a table, and a job.  Call this my 1.0 release. For 1.0, I'm only doing one server at a time; yes, I plan on multithreading it, but it does 80 servers in under 3 minutes.  And yes, if it can't reach a server it will throw a message (that I don't currently capture), but it does continue.

What we're doing:

  • grab a list of servers that you already have stored in a database somewhere
  • for each server
    • run Paul Randall's code that aggregates the overall wait stats
    • save results to a central server (probably where you have your server list)
Powershell code pilfered from Chad Miller, SQL code from Paul Randall.  
and

First, grab the scripts for invoke-sqlcmd2 (http://gallery.technet.microsoft.com/ScriptCenter/en-us/7985b7ef-ed89-4dfd-b02a-433cc4e30894) and write-datatable (http://gallery.technet.microsoft.com/ScriptCenter/en-us/2fdeaf8d-b164-411c-9483-99413d6053ae) and save to files named invoke-sqlcmd2.ps1 and write-datatable.ps1, respectively.  Everything goes in  c:\sql_scripts.

Next, the actual query from Paul Randall; save this as get_dm_os_wait_stats.ps1. This gets useful info from the DMV.

WITH Waits AS
    (SELECT
        wait_type,
        wait_time_ms / 1000.0 AS WaitS,
        (wait_time_ms - signal_wait_time_ms) / 1000.0 AS ResourceS,
        signal_wait_time_ms / 1000.0 AS SignalS,
        waiting_tasks_count AS WaitCount,
        100.0 * wait_time_ms / SUM (wait_time_ms) OVER() AS Percentage,
        ROW_NUMBER() OVER(ORDER BY wait_time_ms DESC) AS RowNum
    FROM sys.dm_os_wait_stats
    WHERE wait_type NOT IN (
        'CLR_SEMAPHORE', 'LAZYWRITER_SLEEP', 'RESOURCE_QUEUE', 'SLEEP_TASK',
        'SLEEP_SYSTEMTASK', 'SQLTRACE_BUFFER_FLUSH', 'WAITFOR', 'LOGMGR_QUEUE',
        'CHECKPOINT_QUEUE', 'REQUEST_FOR_DEADLOCK_SEARCH', 'XE_TIMER_EVENT', 'BROKER_TO_FLUSH',
        'BROKER_TASK_STOP', 'CLR_MANUAL_EVENT', 'CLR_AUTO_EVENT', 'DISPATCHER_QUEUE_SEMAPHORE',
        'FT_IFTS_SCHEDULER_IDLE_WAIT', 'XE_DISPATCHER_WAIT', 'XE_DISPATCHER_JOIN', 'BROKER_EVENTHANDLER',
        'TRACEWRITE', 'FT_IFTSHC_MUTEX', 'SQLTRACE_INCREMENTAL_FLUSH_SLEEP',
        'BROKER_RECEIVE_WAITFOR', 'ONDEMAND_TASK_QUEUE', 'DBMIRROR_EVENTS_QUEUE',
        'DBMIRRORING_CMD', 'BROKER_TRANSMITTER', 'SQLTRACE_WAIT_ENTRIES',
        'SLEEP_BPOOL_FLUSH', 'SQLTRACE_LOCK'
--mdb 2012/09/12 adding 2012-specific waits to ignore
,'DIRTY_PAGE_POLL','HADR_FILESTREAM_IOMGR_IOCOMPLETION'
)
    )
SELECT
    @@servername as server_name, getdate() as insert_datetime, W1.wait_type AS WaitType, 
    CAST (W1.WaitS AS DECIMAL(14, 2)) AS Wait_S,
    CAST (W1.ResourceS AS DECIMAL(14, 2)) AS Resource_S,
    CAST (W1.SignalS AS DECIMAL(14, 2)) AS Signal_S,
    W1.WaitCount AS WaitCount,
    CAST (W1.Percentage AS DECIMAL(4, 2)) AS Percentage,
    CAST ((W1.WaitS / W1.WaitCount) AS DECIMAL (14, 4)) AS AvgWait_S,
    CAST ((W1.ResourceS / W1.WaitCount) AS DECIMAL (14, 4)) AS AvgRes_S,
    CAST ((W1.SignalS / W1.WaitCount) AS DECIMAL (14, 4)) AS AvgSig_S
FROM Waits AS W1
    INNER JOIN Waits AS W2 ON W2.RowNum <= W1.RowNum
GROUP BY W1.RowNum, W1.wait_type, W1.WaitS, W1.ResourceS, W1.SignalS, W1.WaitCount, W1.Percentage
HAVING SUM (W2.Percentage) - W1.Percentage < 95; -- percentage threshold


Now, create the table for the results.  The identity column is at the end so that write-datatable doesn't balk.  Technically the ID is not needed, but I can more easily see how the process is doing.


CREATE TABLE [dbo].[dm_os_wait_stats_info](
      [server_name] [sysname] NOT NULL,
      [insert_datetime] [datetime] NOT NULL,
      [WaitType] [varchar](120) NOT NULL,
      [Wait_S] [decimal](14, 2) NULL,
      [Resource_S] [decimal](14, 2) NULL,
      [Signal_S] [decimal](14, 2) NULL,
      [WaitCount] [bigint] NULL,
      [Percentage] [decimal](4, 2) NULL,
      [AvgWait_S] [decimal](14, 4) NULL,
      [AvgRes_S] [decimal](14, 4) NULL,
      [AvgSig_S] [decimal](14, 4) NULL,   
--ID at the end, otherwise the write-datatable chokes
      [id] [bigint] IDENTITY(1,1) NOT NULL,
 CONSTRAINT [PK_dm_os_wait_stats_info] PRIMARY KEY CLUSTERED
(
      [server_name],
      [insert_datetime],
      [WaitType]
)
)



Next, save the following code to a file named get_dm_os_wait_stats.ps1.  I put it in c:\sql_scripts.  The first two lines "dot source" the scripts so that their functions can be called.  The third is the actual heavy lifter.

. c:\sql_scripts\invoke-sqlcmd2.ps1
. c:\sql_scripts\write-datatable.ps1
invoke-sqlcmd2 -serverinstance "serverwithserverlist" -query "SELECT server_names FROM yourdatabase.dbo.yourserverlist WHERE active = 1" | foreach-object {$dt = invoke-sqlcmd2 -erroraction silentlycontinue -serverinstance $_.server -inputfile c:\sql_scripts\dm_os_wait_stats_agg.sql -As 'Datatable'; write-datatable -serverinstance "serverwithserverlist" -DATABASE "targetdb" -tablename "dm_os_wait_stats_info" -DATA $dt}



Finally, create the job.  Only needs one job step, set as Operating system (CmdExec).  Schedule that however often you want - I'd say either hourly or daily.  You'll want another job to delete old records (I'll leave that as an exercise for the reader)
powershell "& c:\sql_scripts\get_dm_os_wait_stats.ps1"

And that's pretty much it!  

Thursday, January 27, 2011

[Replication] Orphaned distrubution agent after SQL Agent crash

I need to figure out where THIS particular error is stored in the logs, since none of my normal processes or alerts flagged this...

"Error messages:
The replication agent has not logged a progress message in 10 minutes. This might indicate an unresponsive agent or high system activity. Verify that records are being replicated to the destination and that connections to the Subscriber, Publisher, and Di"

How'd I find it? The big red X in the Replication Monitor, for the server whose SQL Agent crashed earlier today. Nothing in the Agent Log or the SQL Server Log. So it's buried somewhere in sysjobhistory

I believe the extant process was running properly, but why chance it?

Login to server, find "qrdrsvc.exe" in Task Manager. Kill process. Run the Queue Reader job. On my server, it's [MYServerName].6 (?), and was Between Retries. Also, the job category was: "REPL-QueueReader"

And you didn't think you'd learn anything new today...

[Replication] Orphaned agents after Agent crash.

Had SQL Server agent crash on me, and then my home-grown replication monitoring gave me the following error via email (interestingly enough, the Alerts we created didn't fire):


Agent 'MYSERVERNAME-MYDatabase-MyPublication-Servername-525' is
retrying after an error. 14 retries attempted. See agent
job history in the Jobs folder for more details.


Since that's the error you'd see in the Replication Monitor (start->run->"sqlmonitor"), I went and looked at the agent which showed our now-at-16 retries:

[...]
2011-01-27 22:46:17.596 Agent message code 21036. Another distribution agent for the subscription or subscriptions is running, or the server is working on a previous request by the same agent.


Interesting. My guess is that since the SQL Agent crashed, it left behind agents that are still running their original orders, blocking the new agents from starting. Well, let's look at what's connected.


SELECT * FROM master..sysprocesses
WHERE program_name LIKE @@servername + '%'
AND login_time < CONVERT(CHAR(8),GETDATE(),112)--before today
AND hostname = @@servername


Bingo. Two SPIDs, both with program names like 'MYSERVERNAME-MYDatabase-MyPublication-Servername' - and which match my original email above.

Kill the two SPIDs, and the next go-round the distribution agent spins up successfully.

Tuesday, January 11, 2011

CHAR() vs CHAR()

Another of the "I forget the tool so let's put it in the blog" posts.

Obviously there's the datatype. But you can also create high-ascii & low-ascii symbols that might show up as a tiny square, or (at least in SSMS) not show up at all.

A map:
http://web.cs.mun.ca/~michael/c/ascii-table.html
This also explains some of the other code you may have seen for BULK INSERT, such as 0x0a (which is "newline" or "linefeed", the Unix "return"; Windows uses CR/LF). It can also be used to clean up code. Instead of something like
+ ''' +

you could use
+ char(39) + 


The value in CHAR has to be a number. Which is, naturally, both good and bad.

more examples:
select char(20) --random low-ascii
select char(32) --space

Wednesday, June 11, 2008

[Linked Servers] Trick to making cross-server queries faster

Learned this years ago, and it's one of those nice tricks to keep in your cap.

So, what collation are you running? Do you know? Do you always use the default? Do you use linked servers and need a little more performance?

If you do, then you're in luck. If you have a linked server between two servers running the same collation, enable "collation compatible" and your queries will run faster.

Why? As I remember, if you don't have it enabled, then your query is sent across without the WHERE clause. Once it comes back, it's evaluated, ensuring that collation is properly dealt with. If you have collation compatible = true, then it sends over the whole query, including the WHERE clause. So, fewer results returned, lower I/O on the far-side, and no processing required locally.

One thing, though - make sure you're on the same collation. On 2005, the default is still the same (IIRC), but what it's called has changed.

Monday, April 7, 2008

[WAT] Weird conversion error in SQL Server

Here's a fun conversion that will bite you.

DECLARE @test VARCHAR(10)
SET @test = '50000'
SELECT convert(varchar(5), convert(int, @test))


You get back "50000".
Now do this:

DECLARE @test VARCHAR(10)
SET @test = '500000'
SELECT convert(varchar(5), convert(int, @test))

What do you get back? "*". Nifty.

Tuesday, January 29, 2008

[Logins] Resync logins after moving a database across servers

Not mine. This is courtesy of Sql-server-performance.com, which you should already be reading. Putting it here for my edification.

When you move a database from one server to another, the user IDs will no longer match. So you can't create a user with the same name in that database, and you can't fix the problem easily. This will sync it.


--Script to resync orphan SQL Server login IDs and database user IDs


USE database_name --Change to active database name
GO

DECLARE @UserName nvarchar(255)
DECLARE orphanuser_cur cursor for
SELECT UserName = name
FROM sysusers
WHERE issqluser = 1 and (sid is not null and sid <> 0x0) and
suser_sname(sid) is null
ORDER BY name

OPEN orphanuser_cur
FETCH NEXT FROM orphanuser_cur INTO @UserName

WHILE (@@fetch_status = 0)
BEGIN
PRINT @UserName + ' user name being resynced'

EXEC sp_change_users_login 'Update_one', @UserName, @UserName

FETCH NEXT FROM orphanuser_cur INTO @UserName
END

CLOSE orphanuser_cur
DEALLOCATE orphanuser_cur