Where Does CPU Go When It’s Missing?

Fans of science fiction—and science fact, for that matter—are certainly familiar with the concept of parallel universes or worlds that exist next to ours but are separate from it.

Most often, the people, places, and events in these parallel worlds are similar to our own world in most respects but have many fundamental differences.

While parallel universes are a staple in science fiction and advanced physics, you don’t need to probe into either to see them in action. In fact, look no farther than to your trusty IBM i to witness a case of parallel universes.

On your IBM i, two universes coexist:

1. The world of the IBM i operating system
2. The world of the Licensed Internal Code (LIC)

While these “universes” share the same physical foundations (POWER, etc.), the world of the operating system, which we can communicate with, and the world of Licensed Internal Code are quite different.

Work Management with Jobs and Threads

Nowhere is this easier to see than in the area of work management. Work management in the IBM i universe shares some concepts with LIC, but completely differs on others.

Work management in the IBM i operating system involves jobs, subsystems, memory pools, job queues, routing entries, and so on. If you are an administrator or operator, you deal with this world the whole time.

In this universe, jobs use CPU. If you want to find out where all that good CPU is being used, you turn to the tool of your liking for displaying jobs.

Each job consists of one or multiple threads. Threads are pieces of computer work that can execute independently from each other, but are also united under a common job. This enables the threads inside a job to share resources more efficiently than they could if they were individual jobs. Think of them as subdivisions within a job.

Jobs and threads: that is the familiar world of the operating system.

Work Management with Tasks

Now let’s take a look at the parallel universe of Licensed Internal Code (LIC). We can’t access this world unless we are equipped with IBM service software or in-depth knowledge of the System Service Tools.

In the strange and unusual world of LIC, the main work management concept is not the job but the task. In terms of work management, it really doesn’t get any more atomic than this, unless you get down to the actual processor cores. There are no subtasks.

Tasks come in four flavors:

1. Initial tasks
2. Resident tasks
3. Non-resident tasks
4. Lightweight process tasks

Initial tasks are the tasks-counterpart to the SCPF job in IBM i. During an IPL, one initial task is started, which then starts up the other tasks. It has the same job as the init process on Linux and AIX or the session manager process (smss.exe) on Windows.

Resident and non-resident tasks are used for LIC-specific functions. These tasks don’t have a direct counterpart in the IBM i world. They are used to manage things like database operations, communications, and paging. (Resident tasks are memory-resident; they remain in main storage as long as they are active.)

Lightweight process tasks are the LIC’s counterpart to IBM i’s threads. Each lightweight process task corresponds to an IBM i thread and vice versa. That they are called “lightweight” seems to be a bit of a misnomer since they represent all the work that applications perform on the system.

This double distinction into operating system and LIC work management, and of LIC tasks into four different kinds, will of course lead you to the one, inescapable conclusion:

The LIC is the place where your missing CPU goes!

CPU Usage and System Tasks

Have you ever noticed that the CPU used by individual jobs does not add up to 100 percent? It’s not a matter of rounding—that only contributes a few percent difference maximum.

No, the CPU that your jobs use—or really the LIC tasks that embody the threads that make up those jobs—is only one part of what uses up CPU on your system.

The other CPU consumers are the resident and non-resident LIC tasks (the initial task does not really account for CPU usage after the IPL). These tasks, together called system tasks, represent under-the-hood work, the mechanics of the LIC universe.

In some cases, the system tasks consume a larger portion of CPU than the jobs themselves. A classic example of this is the restore of a large number of objects. While the restore command is run inside a job, the OS offloads the brunt of the actual restore work (think unzipping) to specialized LIC assist tasks.

As a result, when a large RSTOBJ or RSTLIB is performed, you see the total system CPU go up—sometimes to over 50 percent—but you won’t see any noticeable CPU increase for any job.

On the contrary, your application jobs may well be using less CPU because they are temporarily being displaced by the assist tasks.

If you ever notice what seems like an unusually high amount of CPU in comparison to the CPU used by your jobs, then the best place to look is system tasks.

In our IBM i monitoring solution, Robot Monitor, we recently added a display for exactly that, CPU usage of system tasks. It shows you the top CPU-using system tasks.

Image

Some System Tasks and Their Functions

Knowing which system tasks are using your CPU is desirable because it opens up a window into a world (the LIC) that is normally closed to you but which can very concretely impact the way that applications work in the IBM i universe.

What sort of tasks would you see? Unfortunately, documentation from IBM on this topic is spotty at best. One source is an old IBM Redbook containing a partial list of these tasks in an appendix. Some of the most important system tasks are contained in this list, but many are still missing. Still, it’s better than no documentation at all.

Thanks to our good contacts to Big Blue, I was able to find out the function behind some additional system tasks. (Thanks, Gottfried!) These are listed below.