Monday, July 29, 2013

[Event Notifications] Emailed report based off collected ERRORLOG messages

One of the cool side effects of Event Notifications is the easy collection of SQL Errorlog.  While it doesn't collect all messages, it collects enough for me to be aware of problems.  Alas, it does not collect enough for me to get an email when a server decides to shut down... the skipped messages are usually around startup/shutdown and error dumps.  See https://connect.microsoft.com/SQLServer/feedback/details/782901/event-notification-not-all-errorlog-notices-are-sent-to-service-or-queue#details for more info.


BUT... since it's EN, you get them in real-time, so you could theoretically do other things with them.  This report runs once a day, but I'm sure someone in the comments will come up with a new use I hadn't considered.  I use Alerts on the SQL Server, but I could think of other triggers (what if you suddenly started getting failed logins?)

Another advantage: this has more than the sql ERRORLOG file has - it's closer to what the Event Logs have.  Which means that you don't just got "query ran out of memory", you get the name of the login as well.  REALLY handy, that.

For this report, I've just built a simple 3-way email.  If there are severity errors, you get a block of those.  If there are non-severity errors, they go in a different block And finally, login emails get their own. 

What to do: 
 1) Set up EN to capture ERRORLOG.  See my "Master of All I Survey" post for more.  If you already have EN set up, you'll need to DROP/CREATE a new EN that includes it.  (No alters, sorry)
2) Create a table with the exclusion table.  This allows you to skip records you know about, without changing code.



 CREATE TABLE [dbo].[EN_sql_errors_exclusion_email]
(
[id] [int] NOT NULL IDENTITY(1, 1),
[Exclusions] [varchar] (250) COLLATE SQL_Latin1_General_CP1_CI_AS NULL
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[EN_sql_errors_exclusion_email] ADD CONSTRAINT [PK__EN_sql_errors__ID] PRIMARY KEY CLUSTERED  ([id]) ON [PRIMARY]
GO


3) Create a job.  Here's my description:
This job uses EventNotificationRec..ENAudit_Events and looks for all SQL errors since 1 day ago , split up by those that have 'severity' in the text data, those that don't, and failed logins.  You can exclude data by adding rows to table EN_sql_errors_exclusion_email (uses LIKE, so 'blah%blah' will work; code already includes % at beginning & end)

Enjoy.  Let me know if you have any questions or concerns.
SET NOCOUNT ON
SET ANSI_NULLS ON
SET QUOTED_IDENTIFIER ON
IF OBJECT_ID('tempdb.dbo.#errors_mail') IS NOT NULL
DROP TABLE #errors_mail;
WITH cte
AS (SELECT servername
,loginname
,PostTime
,C.value('ApplicationName[1]', 'varchar(200)') AS ApplicationName
,DatabaseName
,C.value('TextData[1]', 'varchar(max)') AS TextData
,C.value('HostName[1]', 'varchar(200)') AS HostName
FROM EventNotificationRec..ENAudit_Events
OUTER APPLY message_body_xml.nodes('EVENT_INSTANCE') AS T (C)
WHERE EventType = 'ERRORLOG'
)
SELECT cte.*
INTO #errors_mail
FROM cte
LEFT JOIN EventNotificationRec.dbo.EN_sql_errors_exclusion_email e WITH (NOLOCK)
ON cte.textdata LIKE '%' + e.exclusions + '%'
WHERE --textdata LIKE '%severity%' --AND textdata NOT LIKE '%Login failed for user%'
[PostTime] > getdate()-1 --DATEADD(dd, DATEDIFF(dd, 1, GETDATE()), 0)
AND e.exclusions IS NULL
ORDER BY posttime DESC
DECLARE @tableHTML NVARCHAR(MAX)
,@MailSubject VARCHAR(100)
SELECT @MailSubject = 'Event Notifications - ' + CONVERT(VARCHAR,COUNT(*)) + ' SQL Errors since ' + CONVERT(VARCHAR(20),GETDATE()-1)
FROM #errors_mail
SET @tableHTML = N'<H3>Severity Errors since ' + CONVERT(VARCHAR(20), GETDATE()-1) + '</H3>'
+ N'<table border="1">' + N'<tr><th> ServerName </th>' + N'<th> LoginName </th>'
+ N'<th> PostTime </th>' + N'<th>ApplicationName</th>' + N'<th>DatabaseName</th>' + N'<th>TextData</th>'
+ N'<th> HostName </th></tr>' + ISNULL(CAST((SELECT td = RTRIM(LTRIM(ISNULL(T.ServerName,'')))
,''
,td = RTRIM(LTRIM(ISNULL(T.LoginName,'')))
,''
,td = ISNULL(CONVERT(VARCHAR(16), T.PostTime, 120),'')
,''
,td = ISNULL(RTRIM(LTRIM(T.ApplicationName)),'')
,''
,td = ISNULL(RTRIM(LTRIM(T.DatabaseName)),'')
,''
,td = ISNULL(RTRIM(LTRIM(T.TextData)),'')
,''
,td = ISNULL(RTRIM(LTRIM(T.HostName)),'')
FROM #errors_mail T
WHERE textdata NOT LIKE '%Login failed for%'
AND textdata LIKE '%Severity%'
ORDER BY T.ServerName, T.PostTime DESC
FOR
XML PATH('tr')
,TYPE
) AS NVARCHAR(MAX)),'') + N'</table>';
--now add nonseverity errors
SET @tableHTML = ISNULL(@tableHTML,'') + '<br><br>' + N'<H3>non-Severity Errors since ' + CONVERT(VARCHAR(20), GETDATE()-1) + '</H3>'
+ N'<table border="1">' + N'<tr><th> ServerName </th>' + N'<th> LoginName </th>'
+ N'<th> PostTime </th>' + N'<th>ApplicationName</th>' + N'<th>DatabaseName</th>' + N'<th>TextData</th>'
+ N'<th> HostName </th></tr>' + ISNULL(CAST((SELECT td = RTRIM(LTRIM(ISNULL(T.ServerName,'')))
,''
,td = RTRIM(LTRIM(ISNULL(T.LoginName,'')))
,''
,td = ISNULL(CONVERT(VARCHAR(16), T.PostTime, 120),'')
,''
,td = ISNULL(RTRIM(LTRIM(T.ApplicationName)),'')
,''
,td = ISNULL(RTRIM(LTRIM(T.DatabaseName)),'')
,''
,td = ISNULL(RTRIM(LTRIM(T.TextData)),'')
,''
,td = ISNULL(RTRIM(LTRIM(T.HostName)),'')
FROM #errors_mail T
WHERE textdata NOT LIKE '%Severity%'
ORDER BY T.ServerName, T.PostTime DESC
FOR
XML PATH('tr')
,TYPE
) AS NVARCHAR(MAX)),'') + N'</table>';
SET @tableHTML = ISNULL(@tableHTML,'') + '<br><br>' + N'<H3>Login Errors since ' + CONVERT(VARCHAR(20), GETDATE()-1) + '</H3>'
+ N'<table border="1">' + N'<tr><th> ServerName </th>' + N'<th> TextData </th>'
+ N'<th> Count(*) </th>' + N'<th>Started</th>'
+ N'<th> Ended </th></tr>' + ISNULL(CAST((SELECT td = RTRIM(LTRIM(ISNULL(T.ServerName,'')))
,''
,td = ISNULL(RTRIM(LTRIM(SUBSTRING(textdata,CHARINDEX('Login failed for', textdata), 200))),'')
,''
,td = isnull(CONVERT(VARCHAR(25), NULLIF(COUNT(*),0)),'')
,''
,td = isnull(CONVERT(VARCHAR(16), MIN(posttime), 120),'')
,''
,td = isnull(CONVERT(VARCHAR(16), Max(posttime), 120),'')
FROM #errors_mail T
WHERE textdata LIKE '%Login failed for%'
GROUP BY t.servername, SUBSTRING(textdata,CHARINDEX('Login failed for', textdata), 200)
ORDER BY T.ServerName, SUBSTRING(textdata,CHARINDEX('Login failed for', textdata), 200) DESC
FOR
XML PATH('tr')
,TYPE
) AS NVARCHAR(MAX)),'') + N'</table>';
EXEC msdb.dbo.sp_send_dbmail @profile_name = 'yournamegoeshere', @recipients = 'tbd@dev.null',
@subject = @MailSubject, @body = @tableHTML, @body_format = 'HTML';

No comments: