Db2 for your data lakehouse

Db2 as engine and data source
for the data lakehouse

A while ago, I was working with IBM watsonx.data to prepare a presentation on data lakehouse solutions. When talking about the (query) engines for a data lakehouse, typically, it is mostly about Presto and Spark. Did you know that Db2 can be used both as data source AND as query engine in watsonx.data (see screenshot)? Let's take a look...

Lakehouse: Bronze, silver, and gold levels of data

Is this a Data Lakehouse?

While working with customers and IBMers on data processing projects (to keep it as broad as possible), I often hear the talk about bronze, silver, and gold standards. These standards sometimes refer to the systems the data is stored in in terms of reliability, availability, performance, bandwith, and more. The IBM mainframe in a geographically dispersed parallel sysplex configuration may be considered such a gold standard. Lately, bronze / silver /gold standards are more frequently heard in the context of Data Lakehouse architectures and data sources or data zones. So, what is bronze, silver, and gold when discussing data and data lakehouse?

Great chatbots in no time

Chatbots take over customer service

Last week, I was coach again at a chatbot hackathon. Chatbots have been around for a long time and I have blogged about tips & tricks for developing chatbots and resources many times. With Covid-19 transforming the world towards more digitalization and self-service, chatbot deployments grew significantly. In this post, I want to look back at the hackathon and share some wisdom for building great chatbots.

IBM Watson Studio: Download pandas DataFrame as CSV or Excel file

Gist code snippet

Right now, I am working with IBM Watson Studio on some analytics side project. It is nice to work with Jupyter Notebooks, Python and pandas and data again. Even Db2 is involved.

From working with pandas DataFrames locally, I knew how to turn the data into CSV or Excel files. But working with a hosted environment, accessing the file system is not possible and some other solution is needed.

asd765 cloud 87ohhlj db2 askh security xbas chatbot

If you came here and wondered about the blog title, then read on. I plan to write about a couple of mixed, seemingly random topics. Why not express that in today's blog title...? :) It is almost end of the year and here is some news I wanted to share with you before the holidays.

Your chatbot with Watson Discovery News

Some months back I introduced you to a barebone news chatbot. Today, with the updated tutorial to build a database-driven chatbot in place, I want to show you how to easily combine Watson Assistant with Watson Discovery. Watson Assistant already provides steps to deploy an integrated search skill which is based on Watson Discovery. My approach is similar to the database integration: Deploy a cloud function and invoke it from the dialog.

Updated tutorial: Database-driven chatbot

If you want to build a chatbot that gets its content from a database, there is a good news. The existing tutorial “Build a database-driven Slackbot” was just updated to adapt to latest features of IBM Watson Assistant. First, define a skill that reaches out to a database service like Db2. Thereafter, use the built-in integrations to easily tie in the assistant with Slack, Facebook Messenger, embed the chatbot into your own application or use the WordPress plugin.

Architecture of database-driven chatbot

Digital ethics, trusted AI and IBM

Last week I gave a talk followed by a discussion at a university. The presentation was about the current state of Artificial Intelligence (AI) and AI research topics. A good chunk of the discussion was dedicated to fairness, trust and digital ethics. In the following, I am sharing some of the related links.

IBM Research has a site dedicated to  AI. On that, a section provides insight into topics on what they call Trusted AI. On the main IBM site is also a portal Trusted AI for Business, providing an introduction and overview for the non-research crowd. If you are interested and want to try out and learn about few problems hands-on, I recommend these links:

• AI Fairness 360 Open Source Toolkit: http://aif360.mybluemix.net/
• Detect the bias - a game and survey: http://biasreduction.mybluemix.net/
• Old, but still great: MIT Moral Machine:  http://moralmachine.mit.edu/

IBM experts are part of many public panels, workgroups and commissions. In Germany, there is the Enquete-Kommission "Künstliche Intelligenz - Gesellschaftliche Verantwortung und wirtschafliche, soziale und ökologische Potenziale". On the European level, it is the EU High-Level Expert Group on Artificial Intelligence.

Finally, as a showcase of current AI capabilities, I recommend this video of IBM Project Debater and the live debate at Think 2019. A short video explains how Project Debater works:


If you have feedback, suggestions, or questions about this post, please reach out to me on Twitter (@data_henrik) or LinkedIn.

IBM Cloud: The 5 minute barebone news chatbot

News chatbot with Watson Assistant

Do you have 5 minutes and want to build a really simple news chatbot? I tried this yesterday with IBM Cloud and the Watson Assistant and Discovery services. The result is on GitHub in my repository watson-chatbot-discovery-news. It utilizes my Watson Conversation Tool as user interface and for testing. Here are the details.

IBM Watson Assistant: Chatbot tool now supports testing client actions

Test your chatbot

Are you (already) using the Watson Conversation Tool I wrote? There is good news because I added support for testing client side actions. As you might know, IBM Watson Assistant features both client and server side actions to enhance responses with data from external services. In a blog post from this Summer, I wrote that I added support for testing server side actions. They are implemented using IBM Cloud Functions. Client actions are, well, executed on the application side. Read on to learn how to test them.

More tricks for building chatbots with IBM Watson Assistant

Have you heard? New tips and tricks!

If you are building chatbots with IBM Watson Assistant (Conversation), then you might have already stumbled over my collection of tips and tricks or my command line tool for testing chatbots. Earlier today I updated the tips and tricks and added section on collection projection. If you know SQL and relational algebra, then you know the concept of projection. The same can be applied to JSON data and used within Watson Assistant dialogs.

Now on GitHub: Understand and build chatbots the easy way

Recently, I posted about a then upcoming Meetup and my talk about chatbots. Here is a quick follow-up. To compile stuff for that presentation and some other upcoming talks, I created a GitHub repository "chatbot-talk2018". I has lots of links to get started and to deepen understanding around chatbot technology. Moreover, it contains a presentation in Markdown for GitPitch for you to use and extend. And finally, I wrote this brief introduction to some chatbot terms or concepts:

• Intents are what the user aims for, the desired action or result of the interaction. An intent can be to retrieve a weather report.
• Entities are (real or virtual) subjects or objects. For the example of the weather report, entities can be the city or country, e.g., Friedrichshafen in Germany, or date and time information such as "today afternoon".
• A dialog, dialog flow or dialog tree is used to structure the interaction. Typically, an interaction lasts longer than the user providing input and the chatbot returning a single answer. A dialog can be highly complex with several levels, subbranches, (directed) links between dialog nodes and more.
  For a weather chatbot, a dialog could be constructed that, after a greeting, asks the user about the location and time for a weather report, then asks if additional information, such as a weather outlook for the next few days, is needed.
• Slots are supported by several chatbot systems. Slots are used to specify the data items that need to be specified in order to produce the result of an intent. To return a weather report, e.g., at least the location and maybe the date or time is needed.
• Context is state information that is carried from step to step for a specific user interaction. The context typically stores the information that is already gathered as input (see "slot"), result-related data or metadata, or general chat information, e.g., the user name.

If you have feedback, suggestions, or questions about this post, please reach out to me on Twitter (@data_henrik) or LinkedIn.

Extended: Manage and interact with Watson Assistant from the command line

Remember my blog posts about how to manage Watson Assistant from the command line and how to test context for a conversation? Well, that tool did not work well for server actions which I used in this tutorial on building database-driven Slackbot. The good news is that I found time to extend my command line Watson Conversation Tool to support credentials for IBM Cloud Functions.

With the recent update to the tool there are two new features:

1. Use the option "-outputonly" with the "-dialog" option to only print the output text, not the entire JSON response object. I introduced it to be able to demo dialog flows from the command line. Not everybody needs all the metadata for every dialog turn. Here is how it looks like when in action:



Chatbot dialog on the command line

2. In order to test dialog server actions, I need to provide the credentials for IBM Cloud Functions (ICF) in a private context variable. I recently blogged about how to enable the Watson botkit middleware for those server actions. For my tool, just provide the ICF key token as part of the configuration file. A sample is part of the GitHub repository.

For feature requests, defects or suggestions, open an issue or a pull request against the repository. If you have feedback, suggestions, or questions about this post, please reach out to me on Twitter (@data_henrik) or LinkedIn.

Enable Botkit Middleware for Watson Assistant for serverless actions

Slack chatbot with Watson Assistant

Last week I blogged about using Botkit Middleware to create a Watson-powered database interface. I pointed to a tutorial demonstrating a Slack chatbot for a SQL database (Db2). I described that it is relatively simple to replace the Conversation connector with Botkit Middleware. Today, I want to point you to some code I wrote to enable the server-side (serverless) actions in Watson Assistant.

Easy Database Setup the Serverless Way

Serverless Slackbot with Db2

A tutorial I wrote, featuring a database-backed Slack chatbot, is now live. It uses Db2 as database system to store event data. The client accessing the database is written in Node.js and is implement with IBM Cloud Functions in a serverless way. During the development of that tutorial I faced the question on how to perform the database setup. Should I guide users through the user interface to create a table and insert data? Should they install a Db2 client and execute a script locally? I solved the problem in a serverless fashion. Here are the details.

Chatbots: Some tricks with slots in IBM Watson Conversation

As you might remember, I have been using the IBM Watson Conversation service and DB2. My goal was to write a database-driven Slackbot, a Slack app that serves as chat interface to data stored in Db2. I will write more about that entire Slackbot soon, but today I wanted to share some chatbot tricks I learned. How to gather input data, perform checks and clean up the processing environment.

Slots

With my chatbot interface to Db2 I want to both query the database and insert new records. Thus, I need to collect input data of various kind. The Conversation service has a neat feature named input slots that simplifies that process. Within a dialog node (a logical step within the chat flow) I can specify a list of items the Conversation service should check for. I can tell in which variable to save that input and what question to ask if that data was not provided yet. Optional slots, i.e., optional data, can be enabled.

Chatbots: Testing Contexts

Watson Conversation Tool in action

Some weeks ago I blogged about a tool I wrote for the IBM Watson Conversation Service (WCS). It allows you to manage your conversation workspaces from the command line and to test dialogs. Yesterday, I added a new feature to it that helps me (and you) to examine and modify the dialog context. Here is how you can test contexts with my watson conversation tool.

Best practices for lively chatbots

TJBot as lively chatbot

More and more chatbots are being developed and there are good reasons for it. Not all chatbot projects succeed. Often, missing user acceptance is stated. The dialog system might not have hit the nerve, might not have fitted into the target environment. Would you talk with a friend who does not remember your name is repeating the same five phrases over and over again? I would not. So what can be done to make chatbots more lively, more human-like? Here are some best practices and ideas on how to implement them.

Introduction

I started my series on chatbots with lessons and tips from a chatbot hackathon. In that blog I focused on general aspects of building dialogs and designing a conversation system. The language needs to fit the audience. It is something we will look at again today. In a recent blog post I shared tips and tricks for building chatbots. It is possible to carry context throughout a conversation and embed conditions and advanced expressions into the dialog flow and single reponses. We will use that to implement some of the best practices found below. Building lively chatbots could also mean to give the bot a face. The open source project TJBot (pictured) is an example for that. The TJBot can listen, speak and see, give additional feedback and interact through its arm and its light. We won’t cover those aspects, e.g., hardware design or user interfaces, in this blog entry.

More Tips and Tricks for Building Chatbots

Chatbot Architecture

You build your first chatbot and it is working ok. Did you know that you can make chatbots even more interactive? That you can access conversation metadata and application variables inside the dialog nodes? You can even use predicates to tailor output to the usage scenario. As a follow up from our “Lessons and Tips from a Chatbot Hackathon“, let’s dig deeper into important features of the IBM Watson Conversation service on the IBM Cloud with Bluemix.

EgoBot: Fun with a Slightly Mutating ChatBot

Fun with the Bluemix EgoBot

Over the past day and evening I had some fun with a slightly mutating chatbot. The API for the IBM Watson Conversation service offers REST calls to query and change the workspace, the parts that make up a chat. So why not try writing a chatbot that is egocentric? A chatbot that answers questions about itself, that is happy as long as everything is related to itself? Well, let me tell you about this fun project I call EgoBot.

The EgoBot is at an early stage right now. It supports queries about some of its metadata and adding new intents. And it has both an English and a German version (does language change its character...?). You can see a sample session below.

Chatting with the Bluemix EgoBot

To find out more about this chatbot head over to the EgoBot GitHub repository. The bot is written in Python and has everything to get you started with either an English or German conversation. Let me know about your Friday fun.