me.. in the making

Talk at FUDCon.in

2011-11-02T07:37:00.000-07:00

After working with JBoss and Drools for more than an year at Red Hat, I am giving my first JBoss talk!

I am giving an introduction to JBoss Drools (An open source Business Rules Integration Platform).

As the expected audience is primarily students, I intend to give them a short background of expert systems, give them an idea about inference engines and their applications in the industry using drools as an example.

As the schedule permits only 40 mins for the talk and 10 mins for Q&A, I'll show them how to make a simple application using drools if time permits.

Being a happy drools user for quite sometime, I really hope I am able to convince people to give it a try :)

Oh and now is the time for the uber-cute FUDCon's Happy Guy!

mavu.in

2011-03-15T12:59:00.000-07:00

Hi All,

My blog has a new home. I've had the domain for quite sometime and I finally decided I'm going to move and be active. See y'all there!

Demo and packaging done

2009-08-17T06:59:00.000-07:00

After some re-assessment of our goals for GSoC, we decided to bring out the standalone version of beacon and continue with the integration later.

I got sponsored for sysadmin-test by Fedora-infra and have set the demo up here. To access it, one needs to create an account on http://publictest3.fedoraproject.org/accounts/ and then use that test account.

Also, I finally managed to get over my packaging blues with some kind help from Rakesh Pandit and have submitted a review request here.

On the whole, this GSoC has been fun (I know the project is not finished yet, but the GSoC phase is coming to a close). This year, it was not as much about GSoC as it was about implementing a product which would be guaranteed in use by the community. I interacted with the community more than last time.

Last but not the least, a heartfelt thanks to a very cool mentor, an amazing upstream (esp. Nandeep), and a very supportive community.

Standalone DocBook editor is ready!

2009-07-13T10:35:00.000-07:00

Finally finished the DTD for all the tags :) *phew*

Also added a document tree which helps easy navigation, addition and deletion of elements etc. Please check this feature out here. It is really cool.

Checklist: Deliverable 1 done.

For the other two deliverables,test results and documentation, I have an idea. I will write the documentation in DocBook format using Beacon itself.

Note: Since the demo is still in testing do not trust your documents with it yet. We'll make it stable soon. Better yet, it would be great if community could report bugs/suggestions. :)

XSLs done!

2009-07-11T21:24:00.000-07:00

Finally sent my XSLs to Nandeep for review yesterday. He must have committed it by now.
Also found a DocBook CSS which publican uses and replaced the existing one. Using the same CSS might help in integrating later on.

Will finish the DTD today.

Now off to fill the mid term evaluation form...

What's Next

2009-07-07T10:19:00.000-07:00

Mid-terms evaluations are knocking on door and its been quite a wild ride coding. Web development is a different ball game altogether and I am beginning to understand why they call JavaScript the world's best worst language :D

After I have spruced the Beacon code and finally finished the DocBook DTD with the initial subset, I am going to start the next big task of getting Beacon integrated with Zikula and Publican for a seamless documentation experience.

It looks like Zikula has some text editors and a plug-in for them. I have to find the best way to get Beacon fit in that work flow. Fortunately Beacon already has a feature that allows one to embed it on any page. So it shouldn't be that difficult.

Am still looking for a best way to get Beacon integrated with Publican. It would be nice if the community can suggest a way to do it.

DocBook Editor 2

2009-07-02T11:57:00.000-07:00

Just completed a tiny DTD for DocBook corresponding to the small set of tags for which I had written the XSLs. I am so glad it works! (http://dev.gentooexperimental.org/~n9986/beacon/editor/ works best with firefox)

Now on it is just a matter of adding more elements to get a richer tag-set for DocBook till the mid-term evals. The basic infra is all set up (will try to keep improving usability though).

Also added a brand new and better CSS.

On the other side I think I should try and get more community involvement into developing the editor as the number of DocBook elements is *huge*.

I'll start by explaining what beacon is and how it works.

Beacon is a What you see is what you mean editor which relies heavily on XSL for XML to HTML conversions and vice-versa.

Most of the work involved in making a plugin is writing the two XSLs. One is required for converting the XML to HTML so that it can be displayed in the browser. And the other for converting the HTML back to XML. All these operations are done on the server side using PHP (or Python).

One may wonder, why rewrite XSLs if they already exists, like for Docbook for example. The reason is that Beacon needs some 'hooks' in the generated HTML so it can pick up the nodes using Javascript. This is required to do multitude of tasks like rendering inline editors, maintaining structural sanity, addition/deletion of new nodes, etc. Since HTML and Docbook XML tags are very different, there needs to be some way to map the XML tags over to HTML. These hooks are done using title attribute on tags. So for example, the tag will be rendered in HTML as:

< p title = "docbookPara"> Somethings are best with a title < /p >

Title tags tend to be most unobtrusive here and works well for DOM manipulation which is made even easier thanks to jQuery.

Once the HTML has been rendered, some Javascript magic makes it editable via inline editors of various types. We are still working on the node adding/deleting feature. Once that is done we will have a more or less a complete editor.

The validation of the generated HTML is kept in check via a Javascript based DTD.

So the user cycle of Beacon is:

new document -> XML -> HTML -> XML

All ajax-y communication is handled by JSON (except for file upload which uses an iframe of course).

Since its a web based it relies completely on Ajax for any post first page load. The UI is pretty much like a desktop Application with Tabs, sidebar etc.

To sum it up, a plugin now requires:

Couple of XSLs
Javascript based DTD
Some trivial PHP code
CSS + some template XMLs

I will write a tutorial on how to make a plugin for docbook next. Will be nice if folks chip in to get in as many tags supported as possible.

DocBook Editor 1

2009-07-02T01:45:00.000-07:00

It's been very quiet in here. This month has been a marathon of coding and a lot of other events on the home front.

First off, a few non-GSoC related developments:

I finished college. (Phew!)
I got placed in Red Hat as an Associate Software Engineer. (gleee!)
I won the Sun India Code For Freedom contest for my B. Tech Project. (ZOMG!! yes that was COMPLETELY unexpected!)
I went out for a small trip this weekend after almost 2 years.

Now to the DocBook Editor...

I started off by learning XSLT (tough!) and making an initial RPM for Beacon and submitted it for review. I created a Feature page and worked on deciding on an initial subset with the help of the Fedora-docs list.

Here are a few links to complement the above:

https://fedoraproject.org/wiki/DocBook_Editor_Documentation
https://fedoraproject.org/wiki/Features/DocBookEditor
https://fedoraproject.org/wiki/DocBook_Editor

Then, I constructed an initial XSL by referring to the example DocBook file in the Fedora Documentation repository and based on the initial tag set that was selected after a community review.

Initially, I fixed a few bugs in Beacon to learn the source code and worked on making it more convenient to add plug-ins by changing the implementation so that beacon needs only a DTD from the plug-in makers (which is us) rather than having us code the whole Javascript. A successful implementation of this would save a lot of time in feature addition and creation of plugins.

Apart from the DTD, I was learning the tools more from implementation perspective like JavaScript and jQuery library used in Beacon.

The last week has been the most exciting. Working with Nandeep Mali, the lead Beacon dev and my point of contact for the Beacon upstream, we got the DTD feature implemented to some extent and its working like a charm. The WYSIWYM is also working and just few more additions need to be done to make this a solid XML editor.

The DTD is actually a giant Javascript object which contains details about every node and its structure (like child, parent, siblings). One may think that why not use an existing WYSIWYG editor like TinyMCE. This is not very feasible because WYSIWYG editors are notorious for
the output they produce. This is what Beacon is trying to avoid by adding a different type of editor. The user will be places with a fill-in-the-blanks type of editor so the generated XML will retain its sanity and conform to Docbook DTD. Makes it easy for the Docs team as well.

Now for the treat: http://dev.gentooexperimental.org/~n9986/beacon/editor/

The above URL is a demo of what we have been working with (I hope the Gentoo URL is not a problem. Beacon was a Gentoo SoC project and I cannot get PHP with JSON and XSLs on my fedora account).

The work till now mainly consisted of writing the XSLs for DocBook and making changes to the beacon core. One final enhancement in the usability of beacon would be a drag and drop feature that has not been committed so far as it is very slow at the moment. Once that is ready it will be very easy to add nodes at any point and still not mess up the corresponding XML.

I have been writing the DTD for DocBook since yesterday and will be able to commit it by very soon. So we can expect another blog post about it very soon.

Has been a fun journey so far. More to come in the next post.

GSoCer again!

2009-05-02T04:26:00.000-07:00

It feels awesome as I type this :) I am a GSoC student for Fedora this year too! I'll be working on Implementing a DocBook plug-in for Beacon. I am being mentored by Yaakov Nemoy and Karsten Wade is my technical guide! I feel honored to work with such amazing people :)

Hope the documentation team finds the editor useful.

What's been up...

2009-02-23T06:19:00.000-08:00

Phew! back to blogger after a long time, I must say after trying so hard to get over my hesitation, I am not quite successful in getting myself into blogging.

What has been up recently is that I finally got started with my Red Hat internship. My target is to introduce java profiling in SystemTap. I have been reading the SystemTap code to find out where exactly the tokens are parsed and where is the code translated into the corresponding C code.

To an extent I have been successful thanks to the comments in the main.cxx stating which phase begins when, but I guess the use of kprobes has somewhat hidden how the token representing the probe point is mapped to the location where it is actually found (or maybe it is not and I could not find it).

Also I have found a few C based java application profilers like JMP (TIJMP for the JVMTI supporting versions of java). I am looking at their code to know how to interface with the JVM using such tools.

Looking forward to be able to accomplish the task.

On the Sarai Fellowship front I am still in the process of recording the speech samples for my speech recognition project for the OLPC. Recording speech samples and typing out the text has turned out to be a lot more time consuming and laborious than I initially anticipated (Might also be because I need to arrange my schedule according to the times when the children are available which is becoming difficult because of my classes). I am also very hopeful of getting an XO soon thanks to Sankarshan Sir, Mr. Amit and Mr. Sayamindu.

I am praying it all comes out fine.

Next, I was invited to the National Institute of Technology, Durgapur to talk to the newerbies about how to contribute to FOSS in their annual FOSS festival Mukti.in.

It was an amazing experience. I talked about the philosophy of FOSS, why should they contribute and whats in it for them. Then I moved on to what it takes to be a contributor and how to contribute. The How part was divided into two categories, for those who want to contribute code, and for those who want to contribute by other means like helping out with the artwork, documentation, translations, publicity etc. I also talked about Fedora and OLPC in slight details as I could supplement the information there by my own experience. The presentation slides can be found here.

It was nice to try and answer questions from the audience when they gave me their skill set and asked me where could they fit in. It was also heartening to see people interested in my GSoC project and asking me about system side programming and SystemTap.

Over all, I was satisfied at the end of the talk and felt it was a job well done though I was tensed about screwing up the schedule of the event by extending the talk by about an hour.

Boot (limn/chart)?

2008-10-11T04:18:00.000-07:00

Had drafted a small write-up on bootlimn sometime ago. About what is bootlimn, how does it work, how is it different from bootchart and how to interpret the output of bootlimn.

Just pasting the write-up here for anyone who is interested..

Bootlimn

===========================================

One line description:
It is an analysis and visualization tool for the linux boot process.

===========================================

Working:
Bootlimn uses systemtap[0], a kernel-probing language to extract the
data in an event-based structure where the systemtap scripts probe for
certain functions to be called or a small time-period to elapse before
triggering a corresponding probe handler.
The probe handler contains details as to how to log the information.

This information is stored in XML format for standardization and for
facilitating its use by interested third parties. This information is
parsed using a SAX interface and is used for rendering an SVG image,
whose format is heavily inspired by the svg output of Bootchart[1].

===========================================

Interpretation of results (most important for any user):

An example file is sent along with this text. The XML folder contains
the output as given by the systemtap scripts.

The final bootlimn output consists of an SVG image and five text files.

a) SVG image ( This image has a template similar to Bootchart):

Header: The time shows how long (in seconds) the boot process took.

CPU stats: The first rectangle shows the CPU stats. The pink plot (in
the background) showing the CPU utilization and the blue plot (in the
foreground) showing the CPU throughput.

Disk stats: The second rectangle shows the CPU stats. The pink plot
(in the background) showing the Disk utilization and the green plot
(in the foreground) showing the disk throughput.

The syscalls that have been probed are listed along with their color
coding.

Process Tree: The process tree differs from the classical process tree
in Bootchart in the following ways:

i) The processes are listed in the chronological order of their initial
call and not necessarily as parent child blocks as in bootchart. As the
boot is sequential, a child is never rendered before a parent but the
child and the parent may be separated by a few intermittent processes.
The parent-child relationship is shown by dotted lines connecting the
parent and the child.

ii) All the actions are rendered. But to bring the image to a practical
size, process blocks instead of processes have been used. The processes
with the same name have been merged into a process block (the individual
rectangles in the tree), and all the calls made to the processes in the
process block are rendered sequentially. Hence, one might observe
sys_exit being called more than once on the same process block but the
block might have a sys_clone called before the first exit. The criteria
for trimming the tree can be easily altered to suit various purposes
but changing the condition in the uniqueprocess generator (in the
SVGRenderer.java).

iii) The color code represents the last sys_call that acted upon the
process and not the state directly. This was done because Bootlimn,
unlike Bootchart, does not poll the /proc directory and probes the system
calls instead.

iv) Flexible level of detail: By default, to render the entire image
with manageable dimensions and still be accurate, the timestamps collected
were in milli-seconds. And while rendering each pixel represents 0.1s.
As, no information is discarded while rendering, changing the level of
detail is very easy.
The level of detail in the case of bootlimn is defined by:
The timestamps unit (systemtap offers options to gather timestamps in ns,
ms etc. i.e. by changing the gettimeofday_ms in the systemtap scripts
in stp folder, we can change the level of detail).
The image size and the scale factor in the renderer.java
The scalability of SVG images can be used to keep the image size manageable.

There is no provision to concentrate on a part of boot process and give a
separate detailed view of that part as of now.

b) The text files.
The systemtap scripts are written so as to gather as much information as
possible.As displaying all the details on a graph is not possible, there are
five complementary files that give all the details collected by the systemtap
scripts.

i)The Ioblock.txt gives all the block IO details:
It prints the text output of the ioblock tapset.
It has:
type - whether it was a request for IO or a signal to end
time - timestamp
devname - block device name
ino - i-node number of the mapped file
error - its value is zero on success
sector - beginning sector for the entire bio
flags -
BIO_UPTODATE 0 ok after I/O completion
BIO_RW_BLOCK 1 RW_AHEAD set, and read/write would block
BIO_EOF 2 out-out-bounds error
BIO_SEG_VALID 3 nr_hw_seg valid
BIO_CLONED 4 doesn't own data
BIO_BOUNCED 5 bio is a bounce bio
BIO_USER_MAPPED 6 contains user pages
BIO_EOPNOTSUPP 7 not supported
rw - binary trace for read/write request
vcnt - bio vector count which represents number of array element (page,
offset, length) which make up this I/O request
idx - offset into the bio vector array
phys_segments - number of segments in this bio after physical address
coalescing is performed.
hw_segments - number of segments after physical and DMA remapping
hardware coalescing is performed
size - total size in bytes
bdev - target block device
bdev_contains - points to the device object which contains the
partition (when bio structure represents a partition)
p_start_sect - points to the start sector of the partition
structure of the device

ii) The Perpro.txt gives the per process CPU usage details
It has:
time - timestamp
pid - process id
execname - name of the process
probefunc - the probing function
utime - the user time of the process
stime - the system time of the process

iii) The Process.txt contains the Process details ( the process tree
is derived out of the same XML as this file.
It has:
time - timestamp
pid - process id
ppid - parent process id
execname - process name
probefunc - probing function
pexecname - parent process name
misc - null as of now. any additional information can be added

iv) The Readwrite.txt contains the details of individual system reads
and writes.
It has:
rcount - read count so far
wcount - write count so far
time - timestamp
pid - process id
execname - process name
pexecname - parent process name
type - read or write
file - the file to which data was written or data was read from

v) The Stats.txt contains the CPU and disk statistics. This file is a
direct mapping to the first two rectangles in the image showing CPU and
disk stats.
It has:
time - timestamp
CPUutil - CPU utilization
CPUtput - CPU throughput
diskutil - Disk utilization
disktput - Disk throughput

==============================================
Source:
The svn version of bootlimn can be checked out from [2] and a tarball can
be found at [3].

==============================================

References

[0] http://sourceware.org/systemtap/
[1] http://www.bootchart.org/
[2] http://code.google.com/p/bootlimn/source/checkout
[3] http://code.google.com/p/google-summer-of-code-2008-fedora/downloads/list

============================================================================

Bootlimn on F9

2008-10-11T02:31:00.000-07:00

On a hurrah note, finally able to run bootlimn on F9. And am too glad that it required no modification of bootlimn code. I just had to get my systemtap settings right.

The output of the bootlimn from F8 and F9 can be downloaded from here.

Project selection

2008-09-16T00:22:00.000-07:00

Wanting to continue with systemtap, we came up with this idea of instrumenting XEN or KVM for project 2. Quoting my mentor " writing useful tapsets that one can use to write meaningful scripts to instrument or gather information from the
running guest."

Mr. Masami Hiramatsu Kindly pointed out the VESPER project which is a framework to gather the state of guest kernel.
It looks as if they would like to support systemtap.

The thread on the systemtap mailing list regarding the same can be found here.

I would be really grateful to have some feedback regarding this.

Adieu GSoC and DgpLUG classes; and Hola Red Hat!

2008-09-15T23:29:00.000-07:00

Funny, the juggling ended!

I remember trying my best not to miss the DgpLUG classes while struggling to meet my mid term evaluation targets. While I have been blogging about my GSoC project and its updates; I haven't mentioned DgpLUG here till now. Its the Linux Users Group of Durgapur, who came up with this nice initiative for training a few newbies in open source technologies. Thank you folks! It was awesome ^_^

On the GSoC front, its nice to see Bootlimn have so many downloads. I admit that the progress has slowed down a bit given my classes and the recent hunt for a nice and useful project for the Red Hat internship ( yeah! I was as amazed when I was offered and I'm still trying to not sound all stupefied when I am talking about or mailing with regard to it. ) But Bootlimn is far from dead. I'll start working on it again once I settle down with this new routine.

Please do let me know what working well and whats not with bootlimn.

oopsey

2008-09-03T04:39:00.000-07:00

I am terribly sorry for the multiple bloopers . The code did not get committed last time.
Please checkout the code now.. Revision 18 is the latest.
Sorry again --with an embarrassed look--

checkout : here

Very high on my TODO... port it to Python.. we desperately need that one.. successfully running java the first time seems a miracle now when I get so many head-banging feedbacks saying java not working.

Another development. We were halfway through while trying to run it on Ubuntu today. Though we had to manually modify grub/menu.lst
and had to struggle with java a bit. (JRE ... *bah*)

see this too : http://sourceware.org/systemtap/wiki/SystemtapOnUbuntu

A request: If anyone has any problems or even if you are able to successfully run it, please let me know.

I'm alive

2008-08-18T13:13:00.000-07:00

Yes I am :D ... just in case my inactivity here raised any doubts...and I won't surprised if it did because, despite several reminders from my mentor that a blog update has been pending, I have been putting it off for the time that I have something substantial ( or .. was it my laziness? ). Now that the pencils-down date has arrived, I see no further excuse for postponing it.

Since the last time..

I got my disk and CPU info without much use of guru mode code (used the queue_stats tapset.. thought had to create my own copy of it where I could change the default time unit to milliseconds instead of microseconds.

The tapset says..
# qstats.stp: Queue statistics gathering tapset
# -------------------------------------------------------------------------
# The default timing function: microseconds. This function could
# go into a separate file (say, qstats_qs_time.stp), so that a user
# script can override it with another definition.
function qs_time () { return gettimeofday_ms () }
# -------------------------------------------------------------------------
Till that is not done.. I might have to stick to my own copy of queue stats.

My renderer module is up. Even though it supports only svg for now, I'll extend it to support other formats very soon.

I gather per process CPU statistics which show how much system and user time they take (got this idea from bootprobe).

I trace sys_open and gather statistics like which process reads/write to what file etc.

I also trace the blockIO (in this case I just provide a way to bring out the blockIO information as gathered by the tapset in XML format).

The idea behind tracing points 3, 4 and 5 is to have as much information as possible at least in text format so that even if it cannot be rendered (will terribly clutter the graph if rendered), we can get as much detail as possible.
As all the above information is timestamped, correlation is very easy.

The bootlimn (as it has been named tentatively) installs and uninstalls very cleanly.

A jar file is packaged along with the source code. It can be run simply by executing ./bootlimn.sh .

A build.xml (to be used with ant) is also available.

Failure of a part of bootlimn does not crash the entire application. It still tries to give as much output as possible.For example,if one of the XML files cannot be parsed, the others are not affected and neither is the renderer module unless it is *very* critical for the creation of the graph.Even if the XML generated is screwed, the bootlimn still renders till the first occurence of improper entry.

The user can specify where to stop by changing the -c option in stpcaller.

Known bugs (Taken care of)):

The XML created, sometimes, has negative timestamps. (see update 3)

The IOblock parser gives errors at times.(This again is because of the screwed XML) ( see update 1).

What needs more work:

All the unique processes are rendered. The user as of now has no control over the degree of detail.

The state transitions can be bettered.

The CPU wait stats can also be added ( code already present in the stps, just requires slight modification in the XSD and corresponding changes to parser and renderer.. will do it soon)

Other formats of images to be supported.( next task at hand after debugging)

Header information needs to be added (this will be done soon too)

And anymore that will be suggested when the code is reviewed ( code can be checked out from here ).

Update 1: A temporary workaround is to define the sector, bdev etc fields ( which get some funny values on very rare occasion) as a String type so that just a single instance of screwed up XML does not hinder the parsing of the entire file. Not the best solution but just a temporary workaround as there is no further calculation based on these fields and the only function that uses them is a 'tostring' which converts them to a string anyway.

Update 2: Egads!!! revision 14 is sort of broken.. I am rectifying it.. please don't checkout the code now.

Update 3: The negative timestamps error has been solved ***phew***. The code can now be checked out. Logging has been changed to disk as opposed to in memory (see comments for further details).

Mid term evals!! eeks!

2008-07-04T05:25:00.000-07:00

Quoting from my previous post "Now I just need to extend this structure to include the CPU and Disk info."

Well.. it turned out to be anything but just.

For 1, as I am not to use /proc files, that rules out passing those values to my systemtap script using command line arguments.
So, we decided to look into the kernel code and see how those entries are filled in the first place so that we could tap the information from there itself.

From there, I figured out that the files that particularly interest me are
1. /proc/proc_misc.c
2. /proc/array.c
3. /block/genhd.c

Then I mailed the systemtap mailing list and they pointed me to this. It is a very interesting tapset but the following bits left me a little unsure about using it-

Note that blktrace needs to be running in order for these scripts to
have any effect - __blk_add_trace() and therefore the tapset probe
isn't actually called unless tracing is active. Of course, that means
that you need to enable CONFIG_BLK_DEV_IO_TRACE in the kernel.

Things works fine this way, if you're careful - for some reason, if you
don't define overrides for *all* the callback functions, *none* of them
get called. The same thing is apparently true wrt optimization - if any
one of the callback functions gets optimized out, they all do. So in
your script, you need to define handler functions for every event type
whether you use it or not, and furthermore the bodies of the unused
handlers need to contain code that won't be optimized out

Installing blktrace sounded like adding additional baggage to us. Also, I haven't tried out blktrace so I do not know how to use this and whether using it will solve my purpose. I could of course play around with it for sometime but mid term evaluations start on 7th so I probed sys_read and sys_write to count the total reads and writes, and then used guru mode to find out the number of I/O tasks pending at the moment.

Yeah , I know guru mode needs to be used only when there is no other alternative . I couldn't see any. Any suggestion would be very welcome.

Thankfully, it works and I have all the diskstats that I require.So far so good.

But CPU information is turning out to be trickier. The variables that I require are not accessible.

A stap -p2 -e 'probe kernel.function("do_task_stat") {$foo}' -u ( A trick I learnt from fche )
gives the alternatives as : task buffer whole vsize eip esp wchan priority nice tty_pgrp tty_nr sigign sigcatch state res ppid pgid sid num_threads mm start_time cmin_flt cmaj_flt min_flt maj_flt cutime cstime utime stime cgtime gtime rsslim tcomm flags ns

But the variables utime, stime are defined within the function and a
stap -p2 -e 'probe kernel.function("do_task_stat") {$utime}' -u
gives semantic error: not accessible at this address: identifier '$utime' at < input > :1:40

I guess the only variables accessible are the parameters that are passed to the function and the return value as the probe is places at the start of the function definition. and the variables I need have their values worked out inside the function and they get seq.printed from there itself .. so these values are not even returned.

The kernel.statement construct is not what I can use as absolute addresses may be different on different computers.

As of now .. I am trying a very very ugly solution..

Edit 1: License stuff :D
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; version 2 of the License.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

%{
#include< linux/kernel_stat.h >
#include< linux/sched.h >
%}

function get_str_info:long(val:long)
%{
struct task_struct *temp = (struct task_struct*)(long)THIS->val;
long x =(long)((temp->state & (0| 1 | 2 | 4| 8))| temp->exit_state);
THIS->__retvalue = x;
%}

function get_u_info_1:long(val:long)
%{
struct task_struct *temp = (struct task_struct*)(long)THIS->val;
cputime_t ut;
long usr;
struct signal_struct *si = temp->signal;
struct task_struct *t = temp;
do {
ut = cputime_add(ut,t->utime);
t = next_thread(t);
} while (t != temp);
ut = cputime_add(ut, si->utime);
usr = (long) ut;
THIS->__retvalue = usr;
%}

function get_u_info_2:long(val:long)
%{
struct task_struct *temp = (struct task_struct*)(long)THIS->val;
cputime_t ut;
long usr;
ut = temp->utime;
usr = (long)ut;
THIS->__retvalue = usr;
%}

function get_s_info_1:long(val:long)
%{
struct task_struct *temp = (struct task_struct*)(long)THIS->val;
long sys;
struct signal_struct *si = temp->signal;
struct task_struct *t = temp;
do {
st = cputime_add(st,t->stime);
t = next_thread(t);
} while (t != temp);
st = cputime_add(st, si->stime);
sys = (long) st;
THIS->__retvalue = sys;
%}

function get_s_info_2:long(val:long)
%{
struct task_struct *temp = (struct task_struct*)(long)THIS->val;
cputime_t st;
long sys;
st = temp->stime;
sys = (long)st;
THIS->__retvalue = sys;
%}

probe kernel.function("do_task_stat")
{
if ($whole)
{
u = get_u_info_1($task)
s = get_s_info_1($task)
state = get_state_info($task)
}
else
{
u = get_u_info_2($task)
s = get_s_info_2($task)
state= get_state_info($task)
}
}

Of course the above code needs to be debugged for successful compilation : function get_state_info:long works but the rest need some more work ( probably due to the calls to other inline functions: next_thread()and cputime_add() within the function ). The reason I am putting up the unfinished code is because I need to know whether its worth spending time debugging it.Is it the right way to proceed?

What I am basically trying to do is.. take the values of task and whole (they are passed as parameters to the function do_task_stat and hence are available. From there I calculate the variables I needed the same way as is being done inside the original function. *very ugly* but am at a loss of ideas (for now at least).

Anyway, I'll just work on this today and tomorrow. If I am unable to find a better way, I guess I'll move on to the cleaning up tasks so that my Bootchart is perfectly ready with the process and disk info at least.

Then I can come back to this after mid-term evaluations.

Current status of the project

2008-06-21T12:27:00.000-07:00

Here is what I have been up to till now..

I have my Bootchart prototype finally running.

What all do I have till now?

1. An install script.
2. A shell script that runs instead of init, runs my systemtap script and then calls init.
3. My systemtap script, of course that does the probing for me and gathers the process info and outputs it in XML format.
4. A script that calls my parser and saves the text output ( the renderer is to be built after mid-term evaluation)
5. A java parser that parses the XML output using the SAX interfaces.

The parsed text-mode output looks like:

No of Processes '1249'.
Process Details - Time1214080512744, Pid451, PPid:450, Execname:sh,Probefunc:sys_execve,Pexecname:stapio.
Process Details - Time1214080516148, Pid1, PPid:0, Execname:init,Probefunc:sys_execve,Pexecname:swapper.
Process Details - Time1214080516167, Pid454, PPid:453, Execname:init,Probefunc:sys_execve,Pexecname:init.
Process Details - Time1214080516203, Pid455, PPid:454, Execname:rc.sysinit,Probefunc:sys_execve,Pexecname:rc.sysinit.
......
Note: Processes are not unique.

Now I just need to extend this structure to include the CPU and Disk info.

Another thing I'd like to mention, mounting a temporary file system is necessary even if I log in memory and dump the output in a file at the end of probing. A 'probe end {//do the logging in file }' results in a read-only file-system error
even though you not writing to the file till the system is up.

quick update

2008-06-05T12:33:00.000-07:00

okay.. so I coded the XML parser to retrieve the Process information using SAX interface.. (suits my purpose better than DOM)..

I need to modify the renderer to use this now.. and then integrate it to bootchartd.. then 1/3 of my project will be done.. or so I hope.

Writing a mail to my mentor now... will put up details later.

scribbling away

2008-05-31T14:16:00.000-07:00

After half a week of trying to figure out and being worried that I'll seriously fall behind, things are much brighter now :)

I have finally figured out what all information is required.
Gathering process information is the simple part. We have pre-defined functions for it:we can probe the sys_fork() and the sys_execve() and sys_exit() (though I still wonder why vfork() and clone() cannot be probed) and use the pre-defined functions there like pid() etc.It was the disk throughput, CPU utilisation part which was a little confusing..I do have some ideas how to implement, not sure if they work though.

1.was thinking of using the processor queue , disk queue for it and use the qsq_ functions. They'll probably point out the bottlenecks very clearly,say if the number of processes waiting is large.This, from the tasks waiting point of view.

2.Or, to look at it from the CPU and disk point of view, we could probe the kernel inline functions like idle_balance etc.

One more thing, one hint that my mentor gave, logging in memory is better option than logging to a file at bootup.

But anyway, my deadline is 11th June so I can very well experiment with these ideas.

I am really happy to finally get to the coding part :) Wrote a script that returns the process information (pid, ppid, execname, pexecname ) whenever a new process is created.Trivial but still... its code!! Once I am able to implement the CPU and Disk info part, I can modify the bootchartd )

p.s Just realised the parent child relationship can be shown easily in the bootchart.. but that comes later in the rendering phase

Follow up on the mail

2008-05-22T07:36:00.000-07:00

Thanks Mr. Herald Hoyer for the very prompt reply:)
hmm.. so these ideas go into my TODO too

- possibility to bootchart what happens in initrd
- easily configure the "stop" marker
- automatic image generation with date and time in the name
- determine how fine grained the image is (show only processes with at least x milliseconds, do not show kernel threads)

I'll try to implement them in the best possible manner

progress so far...

2008-05-21T19:50:00.000-07:00

The last time I blogged about my project was like a month ago.. so its time for some serious documentation..

Done with the SystemTap tutorials, and halfway through the language reference (further progress will be made as and when required by the project.. as of now... this suffices..).

Done with the git (this and this). The git repository for the Bootchart is here and ill be submitting my patches to my mentor.

Finished reading through/analyzing the current Bootchart code. The results of the analysis phase are as shown above (uploading on blogger for the first time... dunno whether it will be readable...)

Well... I had it bad trying to represent Bootchart in a single root no crosslinks manner (the only structure that freemind supports..as far as I know ) .. and ah yeah... me used it for the first time yesterday :| But its cool...

Hmm.. another digression... I was wondering if there was a way to upload files without actually owning a host or domain...
(had put up this doubt on an IRC... but I guess I made such a pathetic choice of words that it was interpreted as how to upload pics in your blog....not a really bad option but would have loved to have let everyone see the cool folding and unfolding in the .mm files :) and also .. that way I would have been able to use it anywhere.. ) but I was told nopes.. no way.. you cannot do that unless its just text or code.. (in which case I could have gone for pastebin.. thanks General..)

Back to documentation...These are general, non-verified ideas that ill try out and see if they work...

while I was reading that code... I realized that the script directory (that contains bootchart, bootchartd and bootchartd.conf) will be the one that I need to replace using systemtap ... (the conversion from polling to probing happens here).The parsing and the rendering will be conceptually the same.. (maybe I can reuse some portions).. just that ill need to change the format to a standard XML format for which ill have to design the XML format 1st.

Oh and yeah.. my mentor sent me a link to the 30SecondStartup project./me is just mailing them to see if they would like to have some additional metrics showing up in the Bootchart.. will be awesome if this project is of some use to them :)
Looking forward to everything that GSoC would be :)

and /me needs sleep :( signing off ....**yawn**

Oh and I forgot.. GSoC -India informal meet-up will be held on the 8th of June in Delhi :)
Will be terrific..meeting all other GSoC students :)

A thing of beauty ..

2008-04-30T06:32:00.000-07:00

They made me realize I am alive after 3 hours of DSP :o

Aaaah now someone help me stop staring at them and get back to work :|

yippie dippy doo :)

2008-04-23T03:42:00.000-07:00

I got accepted as a student in GSoC :)
Feels awesome :)

The vivas and the practicals are going fine too ( for a change, phew!)

Hmm.. now that I'm finished with my celebrations.. its time for work now .

Oh I forgot to mention.. I have my Bootchart up ( that was supposed to be the 'update 2' but anyway) and I have been playing around with SystemTap; it is working fine too.

Next, as per the time line I need to learn git, JAXP and start exploring BootChart and figure out what additional metrics can be added. I am supposed to be done with it by the 12th of May.. so run.. run..run.

Installing git now.. and then ill look for some good tutorials on it.

update 1..post scriptum

2008-04-15T11:10:00.000-07:00

Oh! i just came back to add that now i finally know why command line is much more powerful than GUI.

Ah and yes .. installing a package using yum is waaaay better than googling out, downloading and installing ( an obvious fact for most people out here but for me anything learnt for the first time by direct experience is worth blogging ;) )

Update 1

2008-04-15T10:04:00.000-07:00

My first real attempt to blog about my project.
Well, after fixing my computer finally, I have graduated to stuff that is directly related to the project.

Milestone 1. Installed SystemTap successfully with Eugene's help. Had to install kernel-debuginfo , kernel-debuginfo-common and kernel-devel
(I was unable to find version 2.6.24.4-64, could only find 2.6.23.15-137 for kernel-debuginfo-common initially which was creating problems ) .

Anyway, its done and now I also see 2.6.24.4-64 on http://rpmfind.net/linux/rpm2html/search.php?query=kernel-debuginfo-common.

Downloaded Bootchart. Will fiddle with it later :) Update 2 will feature that most probably
Currently trying to run some small scripts following the tutorials :)

Aah thats it for now..

This past week

2008-04-14T03:07:00.000-07:00

Well... to summarize the happenings of this week...

1. I had my mid sems (part II )
And the reason they are so special is not because I worked hard, did well and now deserve a break... its just that they usually serve as a reminder that I know absolutely nothing and my end-sems are hardly 3 weeks away :( (and yeah count the internal pracs and the external pracs too .. they start this week GOSH!!!!!!!!!!!!!!1)

2. I finally settled my computer issues :D
The solution: both the operating systems are on the primary hard disk .. so neither of them is cribbing..
All the data is on the external hard disk..so it is safe considering the weird experiments I am trying out on my poor comp.

As it turns out : A Windows recovery first creates a recovery partition (the first 1 and is called D: :|)
and then re-installs itself on C:... (quite odd the naming system though)

The GRUB somehow could not figure that out and kept trying to boot from D: (as it was hd0,0) when the others option was selected.

All that was required was to change the menu.lst's "other" option to (hd0,1)... thats it and it took soooo long :( and some remote assistance from a kind guy.

Next , I realized that my network settings were screwed up.
That again took a lot more time than was actually required.. (I just wish I paid attention in CN classes )... don't ask me what did I do? Just kept fiddling with everything ... all the network settings till it was fixed and now I don't know exactly what fixed it..

Okay... now that its done.... my computer is running a yum update....which will be followed by SystemTap and Bootchart installations ...

Hmph...

/ me also makes a note to complete my files today :(

Baby steps :)

2008-03-30T08:37:00.000-07:00

Hmm .. well.. i installed fedora *Successfully* after some initial hiccups (grub loading .... error 15... phew!!! )
coincidentally... today was my first stint with hardware repair too ( you wont find it that big ... but to me yeah it was a big deal .)

Next is mission python and SystemTap :D

me feeling aaawweessoommeeeee :P

Introduction

2008-03-27T00:47:00.000-07:00

Hello everyone!

Hi, this is Satya. Well.... I'm finally blogging !!!!

Hmm to get to the point.. I plunged into the Open Source movement full on (details coming up in the later half....) and I am applying for the Google Summer of Code ( whoa!!!!).

I intend to work on a cool Fedora project which involves implementing Bootchart using SystemTap, gather the system information in XML format, write a parser for the same and then render it. We'll also look out for what additional information can be extracted to make the Bootchart more useful. We'll also make it more flexible by allowing the users to control the stop point of the probing ( I'll publish my application a little later )....

So.. this blog is intended to keep a record of my involvement in the Open Source generally (cool), track my project development (okaaaay) and allow my mentor (that is.. if I get the project) to keep an eye on me :-|

Hmm... before I end my first post..

Here are the names of a few people you'll probably come across a million times when you read my blog...

Mr. Eugene Teo: my 'probable' mentor for the project (and the person who is making me blog!!!!). He has been a very sweet and patient guide.

Nirbheek: an Open Source fanatic who passed the bug on to me (and who has been very patiently solving all my technical queries..)

Runa, Aneesha and Ankita: the very sweet people at Linuxchix who were very helpful in getting me started and are always willing to help....

Man! with so much support...I am sure ill be able to get this done!!!!!

now lets hope... fingers crossed......