prstat –z disaster in crontab (realtime issue) possible hit to you as well, if you are too keen on automations.

Many thanks to Shiva & Swathi for sharing their real time experience and appreciating their wish to share this to in my blog.

Further, the following is very nice work from Swathi with screenshots and well written to understand the situation and how to resolve the same!!!

Read on!!!

Sharing one situation of my learning experience.

Please be careful while using below commands in cronjobs.

prstat -Z
top

Please use top -d 2 to capture output of top command. This will run top command twice with an interval of 5 seconds(default).

For the first run top won't give any valid output...only 0's will be displayed.
For Example:

And when using prstat -Z in cronjob , You  might have used in below format.

Disasters that can happen:

• First run of the cronjob will trigger prstat -Z and will run continuously ...so our script won't exit.
• As per our cron schedule, cronjob will be triggered again ....so prstat -Z will run again with new process id and as a consequence our script won't exit again.
• Here we will be having 2 instances of same job running. In case we schedule the cronjob every 5 minutes, within one hour 12 instances of our script and 12 instances of prstat -Z will be running....resulting in an increase of CPU & memory usage.
  

If this continues ......

Our cron jobs won't run..... (Surprizing !!!!)  Why?

5. There is one location where we can check what happened to cron....

$ cd /var/cron

6. Why does a cronjob get rescheduled???

But the script runs successfully when run manually...what could the reason be?? Insufficient RAM ???

The reason is,

By default a cron can run only 100 jobs at a time from its queue. (new learning to me :()

Let check, how many jobs a cron is running currently ?

Here 277 is process id of cron.

 
See, Cron is already holding 100 jobs which haven't exited. This is the reason for rescheduling/halting of all cronjobs.

All this happened because of prstat –Z

 
A whole day !!!!!!!! watching, the jobs to back normal

After killing all these prstat processes.....everything came back to normal...

All cronjobs are running on time....  Hooooooray!!!!!

Morale:- To those who fond of automations, these are disastrous sometimes where you have no control.

-Thanks

Geek DBA.(On behalf of Swathi & Shiva)