Friday 19 February 2016

Q&A with Kevin Hill



Last year I decided to undertake some question and answer sessions with some of my fellow SQL professionals from around the world. 

Today I am posting the second session where I had the chance to pick the brains of Kevin Hill. Kevin has worked with SQL for a long time and gives a valuable insight into life as a DBA as he shared some of his views and opinions and experiences as a SQL professional. 

Incredibly insightful and extremely informative here we go, Q&A session #2, with Kevin Hill.

OK so first off, who are you and what do you?
I am Kevin Hill, a 17 year SQL Server veteran and Cycling enthusiast.  I am primarily a production admin, but am currently doing development in SQL and SSIS. I ran my own consulting firm for 7 years, and also did two contracts at Microsoft in their SQL Support group.

How did you start off working with SQL Server in particular?
I started with MS Access in 1994, and in 1999 I needed something to connect to an obscure Accounting package to pull reports. That something was SQL Server’s DTS functionality (v7.0)

So what was it that led you from DTS to SQL administration? Was it a planned move or just something you fell into?
Lol…DTS is/was sort of a piece of administration…just import/export.  But…that got my curiosity up so I went looking for what else SQL could do and found a whole career under the hood. I veered off into CRM and Sales force automation for a bit, but came back when that market cooled off.

And still doing SSIS now!
Yep…data import/export/manipulate is going to be around a long time. “ETL developer” is actually a job title that pays well.  Similar things.

If you had to define what it is about working with SQL that you enjoy the most what would it be?
I like being “the guy” that you call when the database is down, the website is down, the revenues are gone…so I can bring it back to life…quickly.

What was it like at Microsoft in the SQL support group, that must have been interesting?
Always busy, always issues to research when I was not on my 4 hours of phone time per day. I was on the non-core “Cluster/replication/DTS/Security” team, so most of my calls were more involved than installation and backup issues.  You can literally learn things walking down the halls of that place…hear an interesting conversation…walk up and listen.  Anywhere else, that is rude.

What in particular did you take away from your time at Microsoft?
10 years out from the last time I was there I would have to say process and approach more than anything else. Approach a situation with a down SQL Server or failing install with calmness, knowing the resources and answers you need are there. Follow the troubleshooting process and get to the root issue. I carried that successfully into the next gig, where I stayed for over 8 years, managing thousands of SQL Server instances with a team of 4 or less at any given time. 

You mentioned running your own consulting firm, how did that come about and what were the challenges?
I started out as an independent contractor on 1/1/2001 as the dot-com crash ramped up to full force. Lots of small customers, many of whom needed me to do things I could not do, so I brought several others on board as sub-contractors and paid them really well.  The single biggest challenge is bringing new business in the door.  I don’t have that skillset, and could not afford to hire anyone.  Eventually I took a great full time gig and shut down my firm.  I still do a little freelance from time to time.

So I know lots of people in DBA roles have contracting/consulting in mind at some point. So a couple of questions on that:

      What advice would you give people who are looking into contracting?
Proper form of business…sole prop, LLC, S-Corp. Decide all this beforehand, and figure in self-employment taxes, healthcare, etc. before you ever get started.  Otherwise go contract W-2 through another firm if the back office stuff scares you.
      What are the benefits and pitfalls?
Lots of each.  Biggest benefit is deciding where, how, and when you want to work.  Biggest pitfalls…legal/tax stuff, and having to find new contracts.  Sales is hard.

How important is it to be passionate about what you do?
Unlike a lot of folks, I think it is more important to be experienced and confident than passionate.  A passionate DBA with limited experience can still mess up a database.  The right mix of experience, passion and confidence can take you far

Thats a really good answer! So when you're interview a DBA, what things do you look for in an individual aside from technical skills?
Can I work beside this person at 3am, 30 hours into a crisis without wanting to throttle him (or him me…)? 
Can he communicate with people in non-SQL terms?
Can I make him panic in an interview? If yes, I don’t want him on the team.
If the candidate brings up personal info, hobbies, beliefs, etc.  I encourage that…
All of this applies equally to male or female, despite me saying “him”…I’ve met a ton of great women in technology

Haha, I now have great images now of how you make people panic in interviews! What do you ask more of, technical questions or non technical?
Mostly technical, then the generic “tell me about yourself” so they can open up with whatever they want to share.  On technical, with one exception I am looking for understanding of the idea, not syntax.  That one exception is “Give me the minimum T-SQL command to back up the master database to a local drive”   Miss that, and we’re done….but I’ll throw in some more questions just to make you feel better.
Panic-inducing: I hand you my production laptop and have you get the local SQL Server instance to start (which I have broken before the interview).

What are your favourite questions to ask?
      • Backups
      • Replication
      • Clustering
      • Patching a SQL 2005/8 cluster
      • Log shipping
      • Story time…what’s the worst thing you ever did, and how did you fix it?
      • How do you back up tempdb
      • Indexing strategies
      • All of these, only if appropriate to the position, and for DBA, not Dev.

If you had to ask one question that could determine if a candidate is a good DBA or not - what would it be?
Assuming I had resume and references that look good: “what do you want me to know about your database administration skills?”

And how important is certification?
For me, not very. SQL certification will not surpass the years of real-world experience I have.  And if you have no experience, the cert won’t help you much, other than knowledge gained from studies. One definite benefit is that certifications often get used to filter resumes, especially when the recruiter is non-technical.

Yes, completely agree. So what are your methods for staying up to date?
Read article, read another article, spin up a VM and try it myself. Occasional SQL Saturday events and user groups

What can people expect from the SQL events and groups and what do you get from them?
Knowledge, community, and here in Dallas – pizza.
For me, its mostly connections to other DBAs, unless I go specifically because of the topic.  Tonight’s topic is “Biml Baby Steps - Automation for Everyone”

What are your own personal favourite areas to work on within SQL Server?
Mostly administration and SSIS packages. I do T-SQL dev, but writing stored procs all day isn’t me.  I’m very good at logical design, and seeing the flaws in existing databases I inherit

Inheriting databases can be an interesting situation, what sort of things do you look for with regards to finding potential issues?
It’s a long and glorious list.  From a current server where I am now:
      • SQL 2008 R2, RTM
      • Incorrect Memory configuration
      • Full recovery model on data that only changes once a week at most
      • ZERO documentation
      • New data is imported to a new table and a UNION ALL is modified to add that table
      • ZERO documentation
      • Stored Procedures have no comments, poor formatting and developer names in the name of the sproc
      • Autogrow is 1MB, data imports are hundreds of MB each
      • Everyone is sysadmin, probably including you…
      • Change control process is intentionally shortcut and ignored on this internal production system
      • Ownership changed to me in December, then was yanked back 3 weeks later with developers overwriting my fixes in prod. 
That's an impressive list, reminds me of one or two environments! So what's the one thing people can do make your life easier when you inherit a system (I suspect documentation!)?
In person handoff/intro/walk-through. That is not always possible of course. If not, give me time to reverse-engineer from live activity. And documentation :-)

I've just wrote a piece about the essential items for DBA's, specifically scripts...are there any scripts that you use routinely?
Mostly my own homegrown stuff, but I have Adam Machanic’s sp_whoisactive, and a few performance scripts from David West.

How important is it to gain knowledge in other areas, inside SQL or out and is there anything particular you are looking into at the moment?
Extremely important from a career perspective.  If you are only an admin, learn some Dev or SSRS.  I was laid off after 8 years and found myself several versions behind, and knowing nothing about SSRS, SSAS/BI or AlwaysOn Availability Groups.  This stood out like a sore thumb on my resume.   Also, you must get familiar with at least one scripting language (I’m learning Python), and get very comfortable with PowerShell.

Very good point, a lot of people talk about keeping a very broad skillset and I was going to ask about non-SQL areas and you mentioned Python, are there any other areas that you are learning or have learnt about? 
I should be learning more about current OS and Storage, but I’m in a DEV role now so I’m busy refreshing myself there. I was a paper MCSE back in 1999 so I have just enough OS to know when to stop what I am doing and call the right team.
I knew “line number BASIC” back in the 8th grade, but have never been a talented programmer.  If you can develop and admin, you will do really well.

Speaking of development, how do you as a DBA promote a good relationship between yourself and dev teams? 
Like any other relationship, communication and courtesy.  I don’t get into the name-calling and insults that a lot of DBAs like to use for developers.  I teach them things that they need to know, and sometimes they teach me some Dev stuff.  Neither of us has a job without the other.   I’m constantly amazed at how people in IT consistently forget about “Please” and “Thank you.”  Treat people with respect and they work with you every time.

So SQL aside, should you learn about other databases, Oracle, NoSQL for example? 
Absolutely! In order:  SQL, Oracle, any NoSQL/BI/Big Data type, and maybe MySQL….list based on benefit to the resume.

I got asked this the other day so I'll do the same, describe being a DBA in three words
Security, Availability, Performance.  Or, More coffee please.

Haha, the latter especially! What advice would you give to someone who is looking to start working with SQL, either as a DBA, database developer?
SQL 101 – learn how to back up and restore a database…even if you are not an admin.  Read blogs, attend community events, follow SQL Server MVPs and other experts on Twitter and find a Senior SQL person to mentor you.

Are there any particular SQL Experts that you follow? 
 I’ve met almost all of this list:
  • Paul Randal – possibly one of the smartest people in the world on SQL Server internals
  • Brent Ozar – brilliant and funny.
  • Ryan Adams – local to me, great guy and very skilled in HA/DR.  Paid to go to his pre-conference at SQLSaturday Austin recently
  • Adam Machanic – wrote sp_whoisactive, which every DBA should be using
  • Thomas LaRock – Solarwinds and past president of PASS
  • Anyone that is or ever was a SQL Server MVP
Yes I 100% agree! All of the guys you mention there are incredibly talented (I'll be seeing Thomas LaRock at SQLBits soon); so what is it that makes people follow these guys in particular, expertise aside?
How well they communicate for the most part. A genius that can’t communicate isn’t all that helpful when he cannot touch the keyboard.  I met Joe Celko 2 weeks ago and was looking for something to have him autograph. Always pictured him as a grumpy old geezer (much like myself), but I was totally wrong. Incredibly nice, very engaging and the only dude at SQLSatAustin in a suit.

There's lots of people but Brent is the one that springs to mind who mastered the balance of having both the expertise and the ability to communicate to people whether complete novices or experienced SQL folk.

What are you views on the junior DBA role and do DBA's make good mentors?
Everyone has to start somewhere yes?  You need to make sure your Juniors are following processes and best practices to ensure you minimize the risk to production that ANY DBA can make.  Some DBAs are good mentors, some not…depends on the person and their desire to teach.
Have you been involved in mentoring and what did you take from it?  
I was never on the receiving end as a DBA, but I did mentor a friend for a few months. Very senior Windows/Network/Hardware guy looking to add to his basic skillset. We went through best practices, backups, DR techniques already in place at the company. Then he took a management job. Since then I received an email that he got and resolved a database mirroring ticket completely on his own at the company he now works for. I did not teach him mirroring :-)

How important has social media become to technology professionals?
Very. I hang out on Twitter every day. The #sqlhelp hashtag is the easiest way to get some of the best minds in the world looking at your issue, so you can be pointed in the right direction.

I think the strength is the number of people/experts giving up their time to help others through forums, #sqlhelp etc
Agreed – without that I would still be trying to figure out how to query an XML file!

SQL 2016 is just around the corner, any particular functionality you're looking forward to?
Availability groups, since I am a HA/DR freak. That is day one, conversation one at any new place. Yes, that is not a SQL 2016 new feature, but I’m still trying to get my customers off 2005/8!

Finally, if you weren't doing SQL what would you be up to?
If I still need income, probably I would go back into the Finance world.  There will always be data and money to manage. If I won the lottery, I would be a full-time cycling coach for kids.

I thoroughly enjoyed doing this Q&A, for me it was the perfect follow up to the first one I did with Sergey Smirnov - it was a bit like doing the bands tricky second album and genuinely I found the article a brilliant read and I really hope you find it as insightful as I did. You can find Kevin at the following:





Wednesday 17 February 2016

Essential items for the DBA's toolkit.

DBA's have to write a lot of scripts to perform a wide range of functions, from maintenance to monitoring we have to dig deep into various areas of SQL to get the information that we need to do our job. But I don't like re-inventing the wheel and thankfully for us some of the best scripts are already there! Created by leading experts the following scripts are those that I (and many others) use all of the time and consider to be an essential part of our DBA toolkit.


Ola Hallengrens Maintenance Solution is a comprehensive set of scripts for running backups, consistency checks and index maintenance. The award winning solution is ridiculously easy to set up and contains a huge amount configurable options. Needless to say its used by many organisations around the world.

Next up we have sp_whoisactive written by Adam Machanic which is one of the most used scripts in the universe. The script gives you an incredible amount of information  about what is happening within your SQL instance. I have lost count the number of times the script has helped me diagnose problems and like everything one this page is used by database professionals the world over.

Brent Ozar's collection of scripts provide you with some serious troubleshooting capabilities and Sp_blitz is one of my personal favourites. Let this script run and it will return with a wealth of information about your SQL instance, why it is configured correctly and how to rectify it.

Analysing cumulative wait statistics is a fundamental part of troubleshooting and this script by Paul Randal is the one I (and many others) use the most. Just to mention the script originally came from Glenn Berry and was modified by Paul quite a bit.

For me all of these go beyond scripts, they're actually solutions in their own right. Maintenance tasks, monitoring, checking configurations and performance are all covered. Maybe you know of some more, created your own perhaps - let me know!




Legacy Servers - Improve Long Running Backups

So this is the first post in a series where I am focusing on inheriting legacy SQL instances. Starting as a new DBA in a company isn't just about new builds and actually just as much of your time (if not more) is bringing the old legacy systems into line and perhaps giving them a new lease of life.

One area to focus on is backups, whilst we would seek to improve the performance of maintenance tasks as a general rule it is very typical to find long running backup jobs on legacy systems as the database has grown way beyond what it once was. This could mean maintenance tasks are now clashing with one another and overlapping, not to mention out of hour processes running into office hours.

Before we go into the backup task itself as specifically we're looking at old systems there's a fair chance that we're going to find a lot of objects that aren't actually required anymore.

Firstly, are all the databases active? It might take a bit of digging around but occasionally you might find databases prefixed with things like ‘TEST’ or ‘TEMP' that show the database was a one off and not actually removed afterwards (btw, I've seen this, many times!). They're just the obvious one's and in fact asking around might throw up some serious space saving opportunities and that means reducing your backup time too. 

We can take the same approach to look into the tables of some of the big databases (or all of them for that matter) and again look for temporary extracts or perhaps logging tables that are not being maintained and contain years and years of data that isn’t needed – and in our case, still being backed up (btw, I've seen this too!). Once we’ve done that we can now start looking to improve the backup time and one popular method is to stripe the backup across multiple files. To test this I’ve taken a database that has been taking around 55 minutes to backup on my test system (I'm using SQL2008 R2 on a pretty old setup).

BACKUP DATABASE [TEST_DB_1] TO 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_1.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_2.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_3.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_4.bak',
DISK = N'C:\Backup\StripeTest\TEST_DB_1_5.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_6.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_7.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_8.bak'

Processed 2953328 pages for database 'TEST_DB_1', file 'TEST_DB_1' on file 1.
Processed 117 pages for database 'TEST_DB_1', file 'TEST_DB_1_log' on file 1.
BACKUP DATABASE successfully processed 2953445 pages in 2003.321 seconds (11.517 MB/sec).

The time is now 00:33:24 - by striping the backup across 8 files I’ve now reduced the time by around 20 minutes.


Now what we can do to keep an eye on things whilst the backup task is running is to run this simple query on the sys.dm_exec_requests DMV  to show the percent completion of any active tasks; our backup will have a command of ‘BACKUP’ and the session_id > 50 clause is to ensure we are only looking at user processes and not system.


SELECT percent_complete, command
FROM sys.dm_exec_requests
WHERE session_id > 50

Now in the example I used 8 files and now we can experiment a little to find a sweet spot, so trying 12 files…

BACKUP DATABASE [TEST_DB_1] TO 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_1.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_2.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_3.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_4.bak',
DISK = N'C:\Backup\StripeTest\TEST_DB_1_5.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_6.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_7.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_8.bak',
DISK = N'C:\Backup\StripeTest\TEST_DB_1_9.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_10.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_11.bak', 
DISK = N'C:\Backup\StripeTest\TEST_DB_1_12.bak'


gives the following:
Processed 2954224 pages for database 'TEST_DB_1', file 'TEST_DB_1' on file 1.
Processed 432 pages for database 'TEST_DB_1', file 'TEST_DB_1_log' on file 1.
BACKUP DATABASE successfully processed 2954656 pages in 1421.776 seconds (16.235 MB/sec).

The time is now down to 00:23:42! Backup time has been halved using twelve files.

So to conclude striping backup files is a great way to improve the times of our backup processes. Prior to that though ensure that you aren't backing up objects that don't need to be, so look out for databases that aren't in use anymore and large tables containing temporary workloads or logging data that hasn't been purged.

Wednesday 3 February 2016

SQL Server Updates (January 2016)

A couple of releases have hit the virtual shelves over the last couple of days. Firstly we have the Community Technology Preview 3.3 of SQL Server 2016 which is available here and if you want to know what's in this release then check out the technical overview.

Secondly the January release of SQL Server Management Studio has been released as well that contains a handful of bug fixes, head over here for info and download.

The best way to become a DBA?

I often get asked what is the best way to become a DBA. In fact this post was inspired by someone over on LinkedIn asking the very same question.

People give different recommendations. Training and certification, maybe move into the role gradually through different areas of your company or perhaps just get thrown in at the deep end as the good old accidental DBA.

Unfortunately all of the above have drawbacks. Training and certification takes time and there is no guarantee that it will open a door for you (you'll need experience too). A move within your own company is quite common but it can be hard to juggle different activities, especially when you're learning the ropes of a new technology. 

And despite being one the most common starting places for many of us the accidental DBA role really isn't ideal because its missing one major component, there just isn't anyone there to guide you along. SQL Server can be a steep learning curve at the best of times and whilst there is a wealth of learning material out there it can be very difficult to filter through it, especially in the early days.

So what exactly is the best way? Well for someone who hasn't worked with SQL Server before or has limited experience then there is no better road to take than becoming a Junior DBA.

Becoming a junior DBA does not have a reliance on experience or certification. In fact most people that I talk to say they would look for a passion to learn SQL Server as being the biggest factor, check out my Q&A with Sergey Smirnov for example. The biggest benefit though is that working in a junior role normally means you can work alongside at least one DBA who will act as a mentor.

Having someone by your side who can share their knowledge and experience is extremely beneficial. For a junior DBA you get to work with a database professional creating real solutions to real problems and you should still have the time to learn and experiment. 
It is worth mentioning as well that mentoring isn't a one sided benefit either. Its an incredibly rich and rewarding experience which I would recommend to anyone.

But the one downside I see is that there aren't enough of the junior DBA roles available. I'm sure there used to be more and that now they appear to be on the decline. If this is the case, why is that? Do we even need the junior role? I'm really interested to hear peoples views on this. 

Monday 1 February 2016

Blog Update Feb 2016

I've had some amazing feedback regarding the 6 Mistakes That Developers Make in SQL post that I put up recently. Some interesting points have been made, some arguments have been had and also I've put a lot of thought into how the post was received overall.

One change that I would like to make to the article is to point out that it isn't aimed at seasoned SQL developers but is more so based on people perhaps starting out developing in SQL Server or perhaps thinking about it. However, if you've been developing for some time and are still doing these things, seriously...don't.

In addition I had some comments back how the post wasn't detailed enough. I have to say it was kept fairly high level deliberately with a view of expanding the points raised into posts of their own. So I have been working hard and I will be posting something in the near future about maintaining development environments (and who should do it) and will work through the other points in time.

Finally I owe everyone a huge "THANKS". The stats on this post have seriously exceeded my expectations! I have learnt a stack from writing it and I hope to keep on posting informative and interesting reads through 2016. Many thanks again.