Adding Capacity? Turn the Spigot Slowly.

Success does not consist in never making mistakes but in never making the same one a second time. - George Bernard Shaw

 I've seen this more than once on two different teams where I work:  on a network upgrade and on our BI Tools Team.  Capacity is identified as a problem which is causing incidents.  The solution is to submit a change to add horsepower.  Come Monday morning, new incidents arise.  To the clients the systems still aren't working, but to IS the problem has simply shifted.

When you add capacity to a system, you're fixing a bottleneck but are all of your downstream systems equipped to handle the deluge of new data coming their way?  You could be breaking South Fork Dam over Johnstown.  When you open the floodgates, are your systems ready?

In my case, an upgrade to SSRS server capacity threw more at the report source databases than they could handle.  After a day of intense investigation, we realized what needed to be done.  Adjust queue depths and indexes and upgrade disk frame technology and database server memory and CPU.  Additionally, the application support team took the most important and often overlooked step of report query tuning.  Soon everything was humming like a well oiled machine.

The lesson learned was that bottlenecks are systemic:  they don't go away they just move.  Before adding capacity, if you're lucky enough to be able to replicate Production in your Test environment, test the upgrade thoroughly.  If upgrading Test to match Prod is too expensive or wouldn't yield accurate results due to differences is load, then turn the upgrade spigot slowly.  The temptation to do everything possible all at once can be hard to resist, but changes are best done one at a time so the effects can be observed.  If multiple changes must go in at the same time, use administrative tools to restrict the effects, allowing you to turn the knob slowly and keep an eye on things.

Why BI is Primed to be BIg

The ideas behind Business Intelligence (BI) go back decades. Online Transaction Processing and Decision Support Systems are the predecessors of something whose time has come. BI is primed to be the single competitive advantage that makes or breaks companies. Here are five reasons this is true:

1. Simplifying the Environment - IT environments are way too complex. There are too many applications running on too many systems being supported by too many people. This house of cards is being held together by a patchwork of home grown solutions and over customized out of the box middleware. Sound familiar? Reducing complexity can seem daunting and this keeps many shops paralyzed. The good news is that executives are getting the memo. The tough economy has taught us that less is more. Plus, the more you simplify, the easier it becomes to simplify more. Be tough on taking things out of the environment. Don't let politics dictate which pet technologies get to stay.
2. Virtualization - Virtual machines have drastically shortened infrastructure planning. They also allow data warehouse environments to be set up quickly for project managers to see how technologies will play together for their pilot or proof of concept before full scale Production deployments. Computing resources are also allocated more efficiently, allowing unused memory and CPU to be assigned dynamically to whatever needs it. Cloud computing will only accelerate this trend.
3. Database abstraction - New tools and programming practices are insulating users from the database details, freeing IT to set up data in business terms one time rather than writing reports for each type of decision that needs to be made. This empowers business users to get the answers they seek, creating BI analysts in their respective business areas of expertise.
4. Social networking - There are new sources of data available directly from existing and potential customers. Business Intelligence systems can meld customer interests, friend networks, and buying behaviors from traditional point of sale sources to generate new leads, improve forecasting, and guide product development. These days, product life cycles are short, public sentiment can turn on a dime and competitors can come out of nowhere. To stay on top of things, intuition and fact based decision making need to be partners.
5. Mobile Computing - Today's business leaders and sales people want their information from anywhere on all of their devices. This creates a headache for information security, but real opportunities for BI since better data is created when it's entered in real time and tagged with location metadata.

These five things will enable BI to be big, but that won't necessarily make it big for you. For you it should start with goals (start with your mission statement and go from there). Once you know the goals of your organization, you're equipped to look for answers that can enable you to achieve those goals. Along your data mining journey, you will come across many interesting things, but if you don't use your goals as the foundation, these interesting nuggets can just as easily lead you astray, so stay focused and happy mining!