Showing posts with label autonomics. Show all posts
Showing posts with label autonomics. Show all posts

Tuesday, October 7, 2014

Starvation: Electronic books, DRM, the local library, and database locks

Over the past days I ran into an interesting database problem. It boils down to resource management and database locks. One of my sons is an avid reader and thus we have an ongoing flow of hardcopy and electronic books, most of them provided by the local public library (THANK YOU!). Recently, my son used the electronic library to place a reservation on a hard-to-get ebook. Yesterday, he received the email that the book was available exclusively to him (intention lock) and to be checked out within 48 hours (placing the exclusive lock). And so my problems began...
Trouble lending an ebook

There is a hard limit on the maximum number of checked out ebooks per account. All electronic books are lent for 14 days without a way to return them earlier because of Digital Rights Management (DRM). If the account is maxed out, lending a reserved book does not work. Pure (teenage) frustration. However, there is an exclusive lock on the book copy and nobody else can lend it either, making the book harder to get and (seemingly) even more popular. As consequence more reservation requests are placed, making the book even harder to lend. In database theory this is called starvation effect or resource starvation. My advise of "read something else" is not considered a solution.

How could this software problem be solved? A change to DRM to allow earlier returns seems to be too complex. As there is also a low limit for open reservation requests per account, temporarily bumping up the number of books that can be lent per account would both solve the starvation effect and enhance the usability. It would even increase the throughput (average books out to readers), would reduce lock waits (trying to read a certain book), and customer feedback.

BTW: The locklist configuration in DB2 (similar to the number of books lent per account) is adapted automatically by the Self Tuning Memory Manager (STMM), for easy of use, for great user/customer feedback.

Friday, September 13, 2013

Gradual Adjustments: Back to school, epilepsy medication, and DB2

This week school started again for the kids (and parents). It means to adjust to getting up early in the morning (where is the extra hour of sleep?), getting used to new and different schedules, and coping with homework. It is a process that isn't done in the blink of an eye, but which takes time. Most adjustments are "in the system" within a day or two, the rest is a matter of few weeks until everybody is really back in school mode/mood.

As I wrote before, one of my sons has/had epilepsy. Since middle of June we are happy to phase out his medication. Every 4-5 weeks we are reducing the dose of tablets he has to take, so that his body can adjust to the changed "chemical cocktail". After each change it takes a week to get used to it, then the remaining 3-4 weeks to really adapt to the new dose. Getting rid of medication is not an abrupt event, it takes a while.

Guess what DB2' autonomic features, especially the self-tuning memory manager (STMM) are doing? For a change in the workload characteristics STMM tries to adjust the configuration, the amount of available memory for different consumers. First it usually is a bigger adjustment, then it gradually moves to the final state. In a pureScale environment, previously a single member was in charge of choosing the memory configuration if STMM was active. Starting with DB2 10.5, you can specify whether each member is adjusting the memory configuration on its own, or a single member should do it and whether you pick that member or DB2 dynamically picks it.
So is it like "back to school"? I guess not, every family member has to adjust at the same pace and over night...

Tuesday, February 19, 2013

Some fun with DB2 Maintenance Policies and pureXML (Part 2)

In my post yesterday I started to look into retrieving and processing the automated maintenance policies using pureXML. The article ended with the conclusion to use the stored procedure AUTOMAINT_GET_POLICY to fetch the policy document. As stated, the SP returns the policy as BLOB in an output parameter. So, as first step, we need to convert the BLOB to XML and make it available for further processing. My choice is an SQL-based table function.

CREATE OR REPLACE FUNCTION MaintPolicy (poltype varchar(50))
    RETURNS TABLE(poltype varchar(50), poldoc xml)
    NO EXTERNAL ACTION
    MODIFIES SQL DATA
    BEGIN ATOMIC
    declare tpoldocblob blob(2m);
    call sysproc.automaint_get_policy(poltype,tpoldocblob);
    return values(poltype,xmlparse(document tpoldocblob));
END


To choose which kind of policy should be fetched and returned, the policy type is passed in. The prototype above doesn't check for valid parameters. It could be extended for error handling and returning all policy documents if null is passed in. The MODIFIES SQL DATA keyword is needed because the AUTOMAINT_GET_POLICY stored procedure seems to modify data and we ran into an error without it. Within the function we call the SP, fetch the policy as BLOB and then return a table row with the policy type and the policy document. The document is converted to the XML type using XMLPARSE.

By having a table function, we can now "simply" include the policy documents into regular SELECT statement.

select poltype, poldoc from table(MaintPolicy('MAINTENANCE_WINDOW'))

POLTYPE                          POLDOC

-------------------------------- -----------------------------------------------
MAINTENANCE_WINDOW
<DB2MaintenanceWindows xmlns="http://www.ibm.com/xmlns/prod/db2/autonomic/config">
<!-- Online Maintenance Window  -->
<OnlineWindow Occurrence="During" startTime="22:00:00" duration="07">
  <DaysOfWeek>Fri Sat</DaysOfWeek>
  <DaysOfMonth>All</DaysOfMonth>
  <MonthsOfYear>All</MonthsOfYear>
</OnlineWindow>
</DB2MaintenanceWindows>    

  1 record(s) selected.


The next step is to look into the XML document. This can be done using either XMLQUERY or XMLTABLE. We continue our quest with XMLTABLE, so that we can turn the individual pieces of information into relational columns. The reason is that most tools and administrators prefer relational data.

The obvious way for using XMLTABLE is the following, the result is not:

select t.* from table(MaintPolicy('MAINTENANCE_WINDOW')) as p, xmltable(XMLNAMESPACES( DEFAULT 'http://www.ibm.com/xmlns/prod/db2/autonomic/config'), '$POLDOC/DB2MaintenanceWindows' passing p.poldoc as "POLDOC" COLUMNS ONL_DAYS VARCHAR(30) PATH 'OnlineWindow/DaysOfWeek', ONL_DOM  VARCHAR(60) PATH 'OnlineWindow/DaysOfMonth', ONL_MOY  varchar(50) PATH 'OnlineWindow/MonthsOfYear' ) as t
 

SQL20267N  The function "HLOESER.MAINTPOLICY" (specific  "SQL130219093012500")
modifies SQL data and is invoked  in an illegal context. Reason code =  "1". 
SQLSTATE=429BL


The error SQL20267N indicates that our table function is not the last reference in the FROM clause which seems a limitation of calling functions that have MODIFIES SQL DATA. In the explanation and user response sections of the error message the advice is given to rewrite it using a common table expression. Let's try a CTE:

with p(polytpe,poldoc) as
     (select poltype, poldoc from table(MaintPolicy('MAINTENANCE_WINDOW'))) 
select t.* 
from p, xmltable(
   XMLNAMESPACES( DEFAULT 'http://www.ibm.com/xmlns/prod/db2/autonomic/config'),

   '$POLDOC/DB2MaintenanceWindows' passing p.poldoc as "POLDOC" 
   COLUMNS
     ONL_DAYS VARCHAR(30) PATH 'OnlineWindow/DaysOfWeek',
     ONL_DOM  VARCHAR(60) PATH 'OnlineWindow/DaysOfMonth',
     ONL_MOY  varchar(50) PATH 'OnlineWindow/MonthsOfYear' ) as t

ONL_DAYS             ONL_DOM                 ONL_MOY                           

-------------------- ----------------------- --------------------
Fri Sat              All                     All           

  1 record(s) selected.


Finally! The function and the query could now be extended and adapted. The WITH clause as a simple form of a common table expression retrieves the policy type and document using our own table function. In the main SELECT we first reference the data of the CTE, then call XMLTABLE for the XML processing. XMLNAMPESPACES is the first function parameter to set the default namespace, i.e., to declare the context for the XML processing. That way we can avoid further namespace declarations within the following clauses. Next, for all the column-related processing, we use "DB2MaintenanceWindows" as the starting point within the XML document. Three columns should be returned, online days, days of month and months of years. Because of the earlier namespace declaration all we need to do is go down to the respective elements.

What is left is to adapt our prototype to the individual requirements and extend it to other policy documents. Do you have questions or comments?

Monday, February 18, 2013

Some fun with DB2 Maintenance Policies and pureXML (Part 1)

To help reduce administrative costs, DB2 supports automatic collection of statistics, automatic database backups, automatic reorganization of tables and indexes, and some more. This so-called automatic maintenance is only an option, it is not necessary to use it with DB2. The specific autonomics can be configured on the database level. If automatic maintenance is switched on, it is a good idea to tell DB2 when it may perform those tasks. You don't want to take indexes, tables, or the entire database offline during peak hours or have I/O-intensive maintenance operations during the day. Your preferences are configured using automated maintenance policies as described in "Scheduled maintenance for high availability".

Policies are specified using XML documents (samples are provided as a starter), then registered with DB2 via either AUTOMAINT_SET_POLICY or AUTOMAINT_SET_POLICYFILE stored procedures. Now that the policies are set, how do you know which are implemented or active? Let's have some fun with DB2, policies, and XML functionality...

To retrieve the policy information, DB2 offers two stored procedures analogous to the SET procedures: AUTOMAINT_GET_POLICY and AUTOMAINT_GET_POLICYFILE. The former returns a BLOB which includes the XML document with the policy, the latter stores the policy document in the file system (only specify the filename and it is stored under ~/sqllib/tmp for most installations). The XML-based information somewhere in the file system is not what we wanted. Looking at AUTOMAINT_GET_POLICY doesn't make us happier either:

 db2 "call automaint_get_policy('MAINTENANCE_WINDOW',null)"

  Value of output parameters
  --------------------------
  Parameter Name  : POLICY
  Parameter Value : x'3C3F786D6C2076657273696F6E3D22312E302220656E636F64696E673D225554462D38223F3E200A3C4442324D61696E74656E616E636557696E646F7773200A786D6C6E733D22687474703A2F2F7777772E69626D2E636F6D2F786D6C6E732F70726F642F6462322F6175746F6E6F6D69632F636F6E66696722203E0A0A203C212D2D204F6E6C696E65204D61696E74656E616E63652057696E646F7720202D2D3E0A203C4F6E6C696E6557696E646F77204F6363757272656E63653D22447572696E672220737461727454696D653D2232323A30303A303022206475726174696F6E3D22303722203E0A20203C446179734F665765656B3E467269205361743C2F446179734F665765656B3E0A20203C446179734F664D6F6E74683E416C6C3C2F446179734F664D6F6E74683E0A20203C4D6F6E7468734F66596561723E416C6C3C2F4D6F6E7468734F66596561723E0A203C2F4F6E6C696E6557696E646F773E0A3C2F4442324D61696E74656E616E636557696E646F77733E200A'

  Return Status = 0


We cannot directly query the output parameter and it is a BLOB. How about the DB2 system catalog as source for lots of information? SYSTOOLS.POLICY looks promising and it shows up in the package cache with queries related to maintenance policies.

db2 describe table systools.policy

                      Data type                     Column
Column name           schema    Data type name      Length     Scale Nulls
--------------------- --------- ------------------- ---------- ----- ------
MED                   SYSIBM    VARCHAR                    128     0 No   
DECISION              SYSIBM    VARCHAR                    128     0 No   
NAME                  SYSIBM    VARCHAR                    128     0 No   
UPDATE_TIME           SYSIBM    TIMESTAMP                   10     6 No   
POLICY                SYSIBM    BLOB                   2097152     0 Yes  

  5 record(s) selected.


db2 "select med,decision,name from systools.policy"

MED                      DECISION                    NAME                     ------------------------ --------------------------- ---------------------------
DB2CommonMED             NOP                         CommonPolicy               DB2DatabaseRecoveryMED   DBBackupDecision            DBBackupPolicy

DB2TableMaintenanceMED   StatsProfileDecision        StatsProfilePolicy DB2TableMaintenanceMED   TableReorgDecision          TableReorgPolicy   DB2TableMaintenanceMED   TableRunstatsDecision       TableRunstatsPolicy 
 

  5 record(s) selected.

POLICY seems to hold the critical information, but it is a BLOB again. We can change it to XML by parsing the value:

db2 "select xmlparse(document policy) from systools.policy where name='CommonPolicy'"

1

---------------------------------------------------------------------------------
<PolicyDocument xmlns:db2="http://www.ibm.com/xmlns/prod/db2/autonomic/policy" xmlns="http://www.ibm.com/xmlns/prod/db2/autonomic/policylang" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.ibm.com/xmlns/prod/db2/autonomic/policy ../schema/DB2TableMaintenanceMED.xsd" medScope="DefaultMaintWindow"><PreconditionSection name="CommonMaintWindowPolicySection"><Precondition policyElementId="online_mw"><db2:MaintenanceWindowCondition timeZone="local" windowType="online"><DayOfWeekMask>0000011</DayOfWeekMask><MonthOfYearMask>111111111111</MonthOfYearMask><DayOfMonthMask>11111111111111111111111111111111111111111111111111111111111111</DayOfMonthMask><TimeOfDayMask>T220000/T050000</TimeOfDayMask></db2:MaintenanceWindowCondition></Precondition></PreconditionSection><PolicySection name="f"/></PolicyDocument>    


  1 record(s) selected.



Unfortunately, the output doesn't look like the official version of the policy document. We could extract information from it using XMLTABLE, but it would be based on an internal structure (which could change over time). Thus, we will bite the bullet and will work with AUTOMAINT_GET_POLICY. Read the second part here.

Wednesday, September 29, 2010

Automatic compression in DB2? Good question and some answers

Compression in DB2 9.7Image via Wikipedia
During the DB2 bootcamp I was teaching I was asked, why DB2 does not offer automatic compression. There are a lot of automatic and autonomous features inside DB2. Why could DB2 not automatically determine that compression makes sense and compress the data?

This is an excellent question. DB2 can indeed take control of several performance-related configuration settings once DB2 has been authorized (enabled) for that. Regarding compression it is interesting to see that DB2 compresses temporary tables automatically once a compression license has been applied and DB2 determines that it makes sense performance-wise. This same capability applied to regular data and indexes is what the question is about. Why not autonomous decisions for that?

The answer relates to several aspects: The scope of a decision and the business strategy. Let's start with the scope. For a temporary table DB2 has an idea of what to expect. It knows the query and has (rough) statistics about the data and expected intermediate result sets. Hence it estimate the impact, good or bad, more or less precisely and decide on whether it makes sense to apply compression. For regular data and indexes, the decision and the background knowledge are different. DB2 does not know which queries to expect, how the data might grow over time, what other concurrent activities might be planned, what resources would be available to compress and reorganize the data. The impact of the decision is by far of a much bigger scope and wide reaching.

The business side of the question whether to compress or not are also nothing DB2 knows about. Service level agreements, storage costs, costs for CPU cycles, response times, throughput goals, business priorities, and many more are something DB2 does not know about. Smart humans are needed to decide on what is best, not just for a single table or index, but for the overall design and how it fits into the business requirements and (cost) strategy.


And  that is where skilled DBAs come into the picture. Excellent question, some answers, and overall good news for DBAs...


Monday, September 27, 2010

Just in time - autonomics in action

Last week I was teaching a DB2 Bootcamp. The class room had big window fronts to the South-West and to the North-West and shortly after lunch the sun started to shine inside. First, it was only very bright, then even I in the front could notice how the temperature inside the room started to increase. After a few minutes the outside sun shades started to come down, finally stopping in a position that still left the nice daylight inside, but blocking the sun.

About half an hour later, it was time for me to start the presentation about "Practical Autonomics in DB2". They let you keep focused on your actual high-value job by taking care of mostly routine tasks. Autonomic Building Maintenance or Building Automation is exactly the same. I didn't need to worry about regulating the temperature or light. Sensors and some "smart algorithms" took care of it, so that I could concentrate on a more valuable job, delivering a presentation and transferring skills.

Do autonomics always give the best, the optimal results? No, but they get close to it without any human intervention. For the class room, it would have required keeping an eye on the temperature and pressing buttons to control the sun shades. Most of all, it would have required to take focus off the teaching and worrying about things a computer, in most of the cases, could do better. The same for autonomics in DB2.

Tuesday, August 3, 2010

I wish they had...

Highways in Germany as in August 2009Image via Wikipedia
The past few days we drove from the South of Germany to Westphalia and back. In most cases this is more or less some relaxed driving on a long stretch of Autobahn (our navigation system: "follow the road for 480 km"). This time however, we had to deal with strong downpours and related accidents as well as many construction zones. That's when I had to think about DB2 - naturally....

Some minutes on the Autobahn, a heavy downpour from almost black clouds started. Still, some drivers continued without headlights and with Bleifuss. The result was a complete stop of traffic because of an accident. One driver in a sports car had lost control of his Ultimate Driving Machine and slammed it into the divider and, with some rotations, back onto the highway which then was blocked for traffic. After one lane was mostly cleared in heavy rain, we continued for not even 5 minutes until we reached another similar accident scene. Later, we were passed again by some cars without headlight and in (too) high speed.

How does that relate to DB2 other than that DB2 can be a sports car and fun to use? I was thinking about the automatic and autonomic features. They allow you to drive your (database) engine on autopilot. Sure, some cars already switch on headlights automatically, adjust the speed of wipers, and have enough electronic to control the brakes and keep the car stabilized. But they don't seem to have a health monitor or utility throttling. Simple rules to be implemented could be "if too much rain and no sight, slow down" or "lift right foot from accelerator when no sight". Even a display to show alerts about, e.g., no grip because of 10 cm of water on the road is missing. So my hope is that with more heavy rain, people will finally learn and adjust. Which brings me to learning optimizers...

BTW: Just based on the information from the navigation system and the map of the Autobahn system, which route did we take?

Friday, May 7, 2010

Automate DB2 database maintenance

On developerWorks is a new article "Automate DB2 9.7 database maintenance in an embedded database environment". In an environment where DB2 is an embedded database system it is important to hide the existence of the DBS by making sure it just works. The article discusses how to set up the various automatic and autonomic features that DB2 provides. It's not only interesting for those dealing with embedded environments, but for any "lazy DB2" in general. There is so much other work to do that you don't want to deal with maintenance that can be automated... (and it is Friday and the weekend is up next...).

Monday, April 27, 2009

Passive House, Electric Cars, Noise, and DB2 diagnostics


It's a little bit over a year now that we live in our passive house. Thanks to 48cm thick walls and 4 pane windows it is mostly quiet inside the house, even with the flight path to/from FDH being very close.
What we are not missing from the US is the sound of Harleys, sometimes large groups of them. A lot of noise, you can hear and feel them approaching.

Putting that into perspective, I read about US lawmakers considering adding a requirement for non-visual alerts for motor vehicles (Pedestrian Safety Enhancement Act of 2009), i.e., to make so far quiet electric and hybrid cars louder. Worldwide everybody else seems to work towards making things quieter (reducing noise emissions). So this looks strange.

What I love about DB2 is that the many autonomic features help forgetting that a database is running. The diagnostic file has information about how DB2 was/is doing. Using diaglevel I can set the "noise level" I prefer, the default (3) is to log all errors and warnings. If you set it to the highest level (most noise), it gives you all kind of informational messages. Did you know that you can analyze the diagnostic messages using the tool db2diag? It was introduced in DB2 a couple years ago. If you haven't done so, try it out and look at the "noise"...

Monday, April 20, 2009

Home Automation, SOA, DB2, and an awning

Over the weekend I looked into a control for a patio awning. Right now we are planning a patio cover (like this one (English / Deutsch) with an automated awning. The idea is that the awning opens automatically when the sun gets too intense and it closes automagically when either the sun has retreated for a longer time, or the wind gets too strong, or it starts raining. You don't want the house to heat up too much when you are away, but most importantly you don't want the awning to be damaged because of the weather.

There are standalone versions of the control, some with wireless remotes. On the more interesting side for the technology-oriented person are controls which can be integrated or are part of a home automation systems (like the European Installation Bus (EIB/KNX) protocol). The idea is that you can control/automate your heating system, your shades, your lamps, your coffee maker, etc. The different devices offer data on the installation bus, and central processors can read and process the data, react and send out instructions. If it is getting dark outside, let down the shades. If you lock your front door, switch off all lamps. If it starts raining, send the kids outside for a bath...

How does that compare to SOA, Web Services, or DB2? All devices offer their data and services on the installation bus, requests are sent on that bus. It's simple to build new applications or to integrate new devices/services. The same idea is behind SOA and Web Services.
DB2 offers great automation features, autonomics. If the sortheap is too small, increase it by moving resources around - if more storage space is required for a tablespace, let the system add it. You don't want to deal with such issues while you are away (for the night or on the weekend). Let the software do it, so you can focus on other more important stuff.

With the same reasoning I would like some processor/software to take care of our patio shade, of the awning. I know it from my database system, I want it for my home.