Incorporate Git and IBM Cloud information into BASH command prompt

Many of you know that I love to use the command line. Because my day to day work includes interfacing IBM Cloud and GitHub, I have changed the BASH configuration to include related information into the command prompt. Here is what I did and how it looks like.

Building a Solution? The Cloud Architecture Center has Blueprints

Cloud Architecture Center

Remember the days when a simple text client and a small database server were the core enterprise solution? These days data flows from various endpoints to data lakes or data reservoirs, data streams are analyzed in real time to trade stacks, prevent fraud, to react to sensor data. How are other companies building their solutions or what are best practices? What products or services can be used? Great that the new Cloud Architecture Center offers blueprints.


Right now the IBM developerWorks Cloud Architecture Center features an architecture gallery where you can filter the available blueprints by overall area like data & analytics, Internet of Things (IoT), Mobile or Web Application. Another filter criterias are by industry or capability, i.e., you could look for sample solution for the insurance industry or a use-case featuring a Hybrid Cloud scenario.

Partial view: Architecture for Cloud Solution

For the selected architecture and solution you are presented with the overall blueprint (as partially shown in the screenshot) and are offered information about the flow, the included components are deployed services and products, and get an overview of the functional and non-functional requirements. Depending on the solution there are links to sample applications, code repositories on GitHub, and more reading material. See the Personality Insights as a good example.

The Architecture Center offers great material for enterprise architects and solution designers and the linked samples and demos are also a good start for developers.

(Update 2016-02-21): There is a new and good overview article with focus on Big Data in the cloud and possible architecture.

Watch this! Move your DB2 monitoring to the in-memory interface (WLM monitoring)

Since its first days as a database management system, DB2 has been been changed. It has been extended by new features to serve customer requirements and has been adapted to the state of the art in hardware and software technology. One major new feature has been the introduction of the DB2 Workload Management in version 9.5 and related more comprehensive monitoring with finer granularity (in-memory metrics monitoring) in version 9.7. As with many product changes, it takes a while for customers to really put them to use and reap the benefits, especially when the existing functionality still works.

Thus I was happy when I saw a new article on IBM developerWorks describing how to move off the (old) snapshot monitoring interfaces in DB2 and to the in-memory metrics monitoring. What is included in the article is an overview of the advantages of the newer interface. This should get you motivated to read the rest of the article (and then to migrate if not done yet). It contains a side-by-side comparison of old and new interfaces and has many sample SQL queries. The queries demonstrate how to obtain DB2 runtime metrics using the old and new interface for some popular monitoring tasks. You can find the documentation of the SQL interface to the in-memory metrics in the DB2 Knowledge Center in this overview. Most of the pages in the manual have further SQL samples to get you started.

So take a look, it will also help you with one of the upcoming DB2 quizzes on this blog.

Some fun with Bluemix, Cloud Foundry, Python, JSON, and the Weather

Bluemix: Bring Your Own Community

Today, I wanted to try out running a Python application in IBM Bluemix. And I wanted to check out weather reports for some possible weekend destinations. Why not combine it?

Because IBM Bluemix is based on Cloud Foundry, I am using the Cloud Foundry-related tools (mainly "cf") to interact with Bluemix. A first challenge is that a Python runtime is not built into Bluemix/Cloud Foundry and I have to tell it to prepare the runtime for me. It is called "Bring Your Own Community". When you click that icon, some helpful description comes up, pointing you to how to get started.

Bluemix: Get started with own buildpacks

The trick is use some scripting to package framework and runtime support and it is called buildpack. For Python there are several buildpacks available and I am going to use this one. It is specified whenever pushing the application from my local machine to Bluemix:

cf push weatherFN -b https://github.com/ephoning/heroku-buildpack-python.git

As a starter, I downloaded a sample Python-based Web application that is deployable to Bluemix/Cloud Foundry. It comes with 5 files, one of them a small "hello.py", the actual application. The other files are:

• runtime.txt - it specifies the Python version
• requirements.txt to list the dependencies on other Python modules
• Procfile - tells the runtime what and how to start
• README.md with some basic introductions

With the starter snippet ready, I looked for a source for weather data. Data for airports worldwide is available, e.g., at the US National Weather Service. I could fetch the current weather for Friedrichshafen airport like this encoded as METAR:

2014/06/27 11:20
EDNY 271120Z 22007KT 180V240 CAVOK 24/06 Q1017

However, it would require decoding (for untrained people like myself). So I looked further and I found OpenWeatherMap. They offer the weather data in various formats, including JSON. The data for Friedrichshafen can easily be fetched like this and look like in this sample:

{"coord":{"lon":9.48,"lat":47.65},"sys":{"message":0.0037,"country":"DE","sunrise":1403839575,"sunset":1403897051}, "weather":[{"id":800,"main":"Clear","description":"Sky is Clear","icon":"01d"}], "base":"cmc stations","main":{"temp":297.46,"humidity":41,"pressure":1010.562,"temp_min":297.15,"temp_max":297.95}, "wind":{"speed":1,"gust":4,"deg":61}, "clouds":{"all":0},"dt":1403869524,"id":2924585,"name":"Friedrichshafen","cod":200}

Python has a built-in JSON module and it is easy to load and dump the data:
    wdata = json.load(urllib.urlopen(url))
    print json.dumps(wdata, indent=2)

A single field can be accessed using array notation (description of current weather condition):
  wdata["weather"][0]["description"]

After some local testing I pushed the code to Bluemix as shown earlier. My Web app is called "weatherFN" as in "weather in Friedrichshafen". When the index page is requested the app redirects to the weather for Friedrichshafen:
http://weatherfn.mybluemix.net/weather/Friedrichshafen

Weather in Friedrichshafen

The same can be done for London or Recife (Brazil), the soccer match USA vs. Germany was held there yesterday:

Weather in London

Weather in Recife

Weather looks nice, I learned more about IBM Bluemix and how to use Python today. Now it is almost time for the weekend.

Last but not least, here is the entire Python code that I used:

 import os  
 from flask import Flask,redirect  
 import urllib  
 import json  
 BASE_URL = "http://api.openweathermap.org/data/2.5/weather?q="  
 app = Flask(__name__)  
 @app.route('/')  
 def index():  
   return redirect('/weather/Friedrichshafen')  
 @app.route('/weather/<city>')  
 def weather(city):  
   url = "%s/%s" % (BASE_URL, city)  
   wdata = json.load(urllib.urlopen(url))  
   print json.dumps(wdata, indent=2)  
   page='<title>current weather for '+wdata["name"]+'</title>'  
   page +='<h1>Current weather for '+wdata["name"]+' ('+wdata["sys"]["country"]+')</h1>'  
   page += '<br/>Min Temp. '+str(wdata["main"]["temp_min"]-273.15)+'<br/>'  
   page += '<br/>Max Temp. '+str(wdata["main"]["temp_max"]-273.15)+'<br/>'  
   page += '<br/>Current Temp. '+str(wdata["main"]["temp"]-273.15)+'<br/>'  
   page += '<br/>Weather: '+wdata["weather"][0]["description"]+'<br/>'  
   return page  
 port = os.getenv('VCAP_APP_PORT', '5000')  
 if __name__ == "__main__":  
      app.run(host='0.0.0.0', port=int(port))  

There are follow-up articles to this one describing how I enabled a custom domain for my Bluemix application and how I added a Cloudant / couchDB to my Python application on Bluemix.

Video: Introduction to IBM Bluemix User Interface

Currently, I am taking a look at IBM's new Platform-as-as-Service (PaasS) offering code-named "Bluemix". Right now it is in open beta and I signed up on its official website http://bluemix.net. Bluemix is based on the open source Cloud Foundry. But how do you get started and get a good introduction to what is offered?

I read the article "What is IBM Bluemix?" on IBM developerWorks. It gives some background information and details that Bluemix offers Development Frameworks (develop in, e.g., Java, Ruby, or Node.js), Application Services (DB2, MongoDB, MySQL, and others), and Clouds, i.e., you can deploy your applications (apps) to public, private, or other clouds.

Next, I watched the following video which gives a nice overview about the different elements of the Bluemix user interface, including the dashboard.




Equipped with that background information and knowing about the UI, the next stop is the Bluemix website offering articles and sample code. One of the examples is on how to build a simple business intelligence (BI) service using Ruby and based on DB2 with BLU Acceleration, code samples included.

If you haven't tried it yet, you can still sign up for the free beta here: http://bluemix.net. Enjoy!

XML or JSON: When to choose what for your database

There is a new article on IBM developerWorks titled "XML or JSON: Guidelines for what to choose for DB2 for z/OS". It has been written by two very experienced technologists with DB2 background. Though the comparison and the many code examples for how to use either XML or JSON are for DB2 for z/OS, most of it also applies to DB2 LUW. So I recommend reading the article regardless of which "camp" you are in.

At the end of the XML vs. JSON comparison is a resource section with a list of good papers and documentation. In my blog's page on DB2 & pureXML Resources you will also find additional material on that topic.

With that to read enjoy the weekend!

Security and DB2 LUW

Did you recently start paying more attention to credit card bills? Thinking twice before speaking on the phone or sending a text message? Cutting short on communication with your partner...? Awareness for topics such as privacy and data security has increased dramatically over the past few months. In some industries such as banking, the supervisory authorities - in Germany it is BaFin - have tightened regulations over several years, requiring changes to how databases are set up and administrated, how data can be stored and accessed. As I recently declared 2014 as the year of database security, I thought collecting some related DB2 resources would be a good way to promote it. Here we go...

When you work with DB2 for Linux, UNIX, and Windows, and are researching a topic, then the DB2 Information Center is a good start. It has an entire section on security (look at the navigation section on the left). It explains the DB2 Security Model, various security-related concepts, and has links and background information on some IBM InfoSphere Guardium tools. Many security and auditing tools as well as the Data Encryption (formerly Encryption Expert) product are labeled Guardium.

Other places to visit are the DB2 Best Practices, IBM Redbooks, and IBM developerWorks (list of DB2 security articles). There is a IBM Data Server Security best practices paper and also a redbook "DB2 Security and Compliance Solutions for Linux, UNIX, and Windows". You can also learn about security functionality when attending one of the offered Information Management bootcamps or taking a DB2 class through Learning Services.

Last but least, before I start my weekend, I would like to point to the blog articles I have written on DB2 security topics.

Have a nice weekend and watch your transactions...

DB2 10.5: Overview of product editions, licensing and features

On the IBM developerWorks website is a new article discussing the different editions of DB2 10.5. It is titled "DB2 editions: Which distributed edition of DB2 10.5 is right for you?". It gives an overview of what editions are available, some of the top notch features, and what licensing options are available. Another new article, "Compare the distributed DB2 10.5 database servers" has a similar intent, but focusses on features by edition and is organized in a table format. The third installment, "Licensing distributed DB2 10.5 servers in a HA environment", does exactly that. :)

Best Practices: Troubleshooting DB2 servers

How about PD, FODC, TRC, core, EcuRep, DIAG, caem, lock, mem, log, OPM, CPU, EDU, app, support, and many more terms in a single paper? The newly published Best Practices: Troubleshooting DB2 servers looks into what should be done to handle issues with DB2. It discusses what data to collect, how DB2 is actually configured to dump the right data, how to exchange that information with IBM, and how to look into typical scenarios on your own.

Because of what is covered, the depth and the embedded links to even dig deeper into troubleshooting topics, it is a good read for beginners and experienced DB2 users.

BTW: You can find all DB2 Best Practices here.

BLU: Begin Learning blU - A good overview and introduction

A so-called rapid adoption guide for DB2 with BLU Acceleration has just been published. It gives an overview of all relevant topics, provides the necessary information at a glance and then links to more resources for those interested in the details. Are you ready to Begin Learning blU (BLU)?

That's all for today. Quick, simple and to the point like the rapid adoption guide... :)

Hot out of the lab: DB2 Best Practices: Tuning and monitoring database system performance (Completely Revised!)

Fresh and hot out of the DB2 labs: The existing DB2 Best Practices paper on database system tuning and monitoring has been completed revised and updated. It covers DB2 10.0, DB2 10.5, and PureData for Operational Analytics. On 80 pages it starts with hardware configuration for good performance (CPU, disk, memory, ...), configuration of the operating system and more, then delves into initial DB2 configuration for different environments. The next big section is on monitoring the system performance and which tools and APIs to use. Once you have identified possible problems, they should be addressed. The paper discusses how to deal with disk, CPU, memory and "lazy system" bottlenecks.

This is a very good update to the DB2 Best Practices series on IBM developerWorks. I covered some of the papers in past blog entries.

DB2 in virtualized environments

I had written about DB2 and virtualization in the past. You can use DB2 in many virtual environments and there are already kind of "dated" redbooks explaining details about DB2 with VMWare, PowerVM, and others. Today I want to point you to two interesting resources:

• You can even virtualize a DB2 pureScale cluster on Linux: The "yes, it is supported" can be found in the Information Center when looking for KVM. All the technical details of setting such a cluster up are discussed in this developerWorks article: Virtualize the IBM DB2 pureScale Feature on Linux using Kernel-based Virtual Machine
• An overview of all supported DB2 virtualization environments, e.g., which hypervisor, which guest operating systems, etc., is provided on this developerWorks page.

(Updated) NoSQL and JSON with DB2

You may have heard that DB2 10.5, which recently became available, includes a technology preview "DB2 NoSQL JSON". JSON is the JavaScript Object Notation. The important stuff for JSON in DB2 can be found in the sqllib/json directory. Once I added the "lib" directory to the Java CLASSPATH, I could invoke a new command line processor "db2nosql" (found in "bin"):

hloeser@rotach:~/sqllib/json/bin$ sh db2nosql.sh
JSON Command Shell Setup and Launcher.
This batch script assumes your JRE is 1.5 and higher. 1.6 will mask your password.
Type db2nosql.sh -help to see options
Enter DB:jsontest


IBM DB2 NoSQL API 1.1.0.0 build 1.0.169
Licensed Materials - Property of IBM
(c) Copyright IBM Corp. 2013 All Rights Reserved.

Debug mode is off.
nosql>
nosql>Type your JSON query and end it with ;
nosql>Type help() or help for usage information
nosql>
nosql>Setup Tables and Functions seems to have not been created or have been created incorrectly. Please type enable(true) and enter to setup them up. You must have the correct admin privileges.
If you do not, type enable(false) and enter to see the SQL that will be used.

After entering the database name to use I got connected. Next was to enable the database for JSON processing as suggested above.

nosql>enable(true)
Executing SQL...
Database Artifacts created successfully

If you use "enable(false)" the tool will just print the DDL to execute manually. This is the creation of a system table SYSTOOLS.SYSJSON_INDEX and several functions. Next I set a JSON namespace. By default the user schema would be taken as namespace. Because I plan to look into storage details later on, I chose "TEST" (which will use the SQL schemaname "TEST" under covers).

 nosql>use test
Switched to schema TEST


Then I was good to go and insert two documents. Note that it is not necessary to create any table, collection, index or anything. Trying to find one of the documents I stored via predicate also worked.

nosql>db.obj.insert({name: "Henrik", country: "DE"})
OK
nosql>db.obj.insert({name: "Jim", country: "US"})
OK
nosql>db.obj.find({country: "US"})
nosql>Row 1:
nosql> {
nosql> "_id":{"$oid":"51c9acf301c690e828779af2"},
nosql> "name":"Jim",
nosql> "country":"US"
nosql> }
nosql>1 row returned in 134 milliseconds.

Some more commands are described in a recent developerWorks article on JSON in DB2. I also plan to write more about my adventures here in this blog.

Added 27.06.2013:
A couple more developerWorks articles are now available:

• Series on InfoSphere Guardium data security and protection for MongoDB
• Series on DB2 JSON NoSQL Capabilities

Finally, I have a video introducing the technology.

developerWorks article on setting up DB2 for encrypted communication with SSL

DB2 allows to encrypt data both in storage and in transit. Securing stored data makes sure that even if someone can access the disk, the actual data can not be read. Encrypting data in transit prevents sniffing, i.e., someone trying to listen to the communication channels and spy on the data (Footnote: this is not entirely true). There exist different options in DB2 for securing data on disk and during communication. SSL, the Secure Socket Layer, is a commonly used technology to encrypt communication, e.g., for Web access with the HTTPS protocol.

A new article has been published on developerWorks titled "Secure Sockets Layer (SSL) support in DB2 for Linux, UNIX, and Windows" that provides step by step instructions on how to setup SSL for use with DB2. If you haven't looked into that topic before, this is a good starter.

Video: Live migration of OpenBravo from Oracle to DB2

Interested in DB2's SQL compatibility and migration from Oracle to DB2? The following video is about a year old now and shows Serge Rielau moving the open source ERP system OpenBravo from Oracle to DB2. Since the time the video was created, some things have changed:

• DB2 is now at version 10.1 and has more compatibility features.
• The tools MEET and IBM Data Movement Tool have been mostly replaced by the Database Conversion Workbench.

My money, SEPA payments, DB2 z/OS, and pureXML

A new article about how DB2 pureXML is used to process SEPA payments has been published on IBM developerWorks. SEPA is the short for Single European Payment Area, an unification and simplification of cross-border payment processes within the European Union/Euro region and associated countries.

In the article Jane Man shows some tricks in handling documents, extracting information, and updating XML documents. Some of the problems they ran into are also discussed.

VIP jets, clouds, and DB2 on cloud

Two years ago I wrote about the "parking problems" at the meeting of the World Economic Forum. The annual meeting of the World Economic Forum is taking place in Davos, Switzerland, and many of the participants have to fly in through the Zurich (ZRH) or Friedrichshafen (FDH) airports. (I have not been invited so far, but could go there by car. Any invitations...?)

This year's meeting is about to start and so it happens that out of the clouds business jets descend to Friedrichshafen and more helicopters can be seen. With that impression on my mind I came back from an after lunch walk and asked myself: Where is a good overview of cloud offerings for DB2? Well, it seems it is on this page at IBM developerWorks. Offerings like the IBM SmartCloud Enterprise and services like the Amazon Elastic Compute Cloud and RightScale are listed. There are also some cloud computing resources at the bottom of that page.

While I continue to wait for an invitation to the World Economic Forum in Davos, you can directly get started with DB2 on the cloud.

XML Devotee session with z/OS topics

A session of the so-called XML Devotee Community will be held this Wednesday, September 12th, with two topics: 1) Using COBOL with pureXML on DB2 z/OS and 2) XML processing on z/OS. Details of the session, including phone numbers, speaker info and more is available at the community website.

Want to learn about DB2 security? New best practices paper

A new DB2 Best Practices paper has been published on developerWorks: DB2 best practices: A practical guide to restrictive databases. Did you know that instead of CREATE DATABASE foo you could also say CREATE DATABASE foo RESTRICTIVE? But what are the differences? The restrictive database has only very few privileges granted to the group PUBLIC and thereby starts off with increased security.

In the new paper the authors compare how the non-restrictive and restrictive databases differ in terms of privileges with a focus on common use cases and administrative groups. It helps to understand how to make use of restrictive databases. While in regular databases a typical problem to secure the database, in a restrictive database it means granting the needed privileges so users can "work".

How to communicate within your family (kids, significant other, DB2)...

Healthy families talk with each other. But (usually) I speak differently with my wife than with my kids. There is no manual on how to address everyone individually or how to conduct get heard by all of them. Fortunately, there is a guide for the DB2 family of products. And it has been updated recently.

When I give talks about the DB2 family I always mention that we strive to coordinate new features, learn from each other (yes, even mistakes), and try to speak the same language. Time travel queries (temporal data), row and column access control, the XML data type, binary XML, etc. are some of the more visible recent coordinated endeavors. Now the so-called SQL Reference for Cross-Platform Development has been updated to reflect new DB2 features. That page is very useful for some other reasons:

• It offers links to older versions of the SQL Reference for Cross-Platform Development.
• It has links to a lot of documentation for DB2 z/OS, DB2 for Linux, UNIX, and Windows, and for DB2 i.
• All versions combined, it has 10 years of best practices of application development.

That's all for now, I have to go prepare the dinner table. Some family discussion is about to start...