Thursday, July 17, 2008

Btrfs and ZFS

This kind of irritates me

Umm, who that matters is complaining to Sun about not releasing ZFS as GPLv2? I can't think of anybody. Its probably just all the reiser4 like fanbois who don't know thing 1 about filesystems or kernel development in general. Sun is doing open source in their own way and thats cool, I think promoting an attitude of openness in any form is better than the way it used to be, even if the CDDL makes it so you can only use it on solaris.

Things on the btrfs front are moving ahead nicely. Ric is very supportive of me working on it so I've been able to go back to actively contributing to it again, which has been nice. Now that the orphan code and new dir format stuff is done I'm going to spend the next few weeks hammering on it to make sure its ready for his next release, and then its back to development. I like this way of doing things, makes sure everything is sound, which is important for a filesystem. I know I just test my particular area for the patch I'm working on to make sure it works when I send it off, so if there are any more complex interaction bugs I won't generally see them until after my patches are merged.

All in all glad to be back to actively contributing to btrfs, and now with a bunch more interest from the community the project as a whole will pick up some more speed and meet Chris's release timeline without problems.

Monday, May 5, 2008

Linbackup, house and more!

So I started working on my backup program that I've been thinking about doing. I got frustrated not having it up somewhere so I could work on it anywhere so I was trying to figure out where I was going to host it. On the one hand being a loyal Red Hat employee my first thought was Fedora Hosted, but then I don't really like interacting with the Fedora community all that much and I especially don't really like jumping through hoops and filling out 30 forms just to get a svn repo to dump my crap, so I decided to use Google's project thing. That was nice and simple, I highly recommend it if you are just looking for someplace to hold all of your stuff. You just pick your name, add your description and boom you are good to go. I had to wait like 10 minutes for the svn stuff to be created and such, but after that I was good to go. Progress is coming along, I got it compiling and got rid of a few minor memory access problems on friday so all the parsing stuff seems to be working well. Hopefully this week I'll be able to read in an entire config, then I can move along to the actual functionality stuff.

Sarah and I put an offer in on a house yesterday and it was accepted. We're super excited, its a nice big house with a 2 car garage and an acre lot, which the dogs will love. Only drawback is there is no fence, but thankfully the house was relatively inexpensive so we've got money left over to build a nice fence in the back.

Done with school finally for the summer. Time to hunker down and start really hacking away at ext4, make sure all the features that are there are actually working and such. Hopefully it will be ready to be used in F10. Working on Linbackup for a full day once a week should help it come along pretty quickly, hopefully I'll be able to get it on the F11 feature list.

Wednesday, April 9, 2008

Shell history

Apparently all the cool kids on fedora planet are running

history | awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head

which is cool to try and figure out what commands you run the most, unfortunately if you are like me and only really use X in order to have hundreds of terminals open, this isn't too effective, since the history never really gets saved in any sort of coherent manner. The only box that I only have one terminal open at a time is the box where I do my testing, so I only ssh into it, do what I need and reboot

[root@unused ~]# history | awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
232 cd
202 vi
117 quilt
96 ls
53 make
29 grep
27 mount
26 git
19 ./fiemap-test
15 ./postmark

and even that misses stuff that I know I do alot of. It is kind of cool tho.

Monday, March 10, 2008

lots of jbd work

So that bug I talked about in my last post kept me busy for the better part of a month, spent entirely too much time reading and going through the jbd code, and when I was done turned out the fix was very small, we were filing buffers onto the running transaction on the wrong list, yay for weird get write access/do write crap. I think that when I get some time I'm going to go through and clean up this code, the only case we want to do something special is in the journal_get_create_access() and journal_get_undo_access() sections, and other than that we really don't care about the buffer until we actually dirty it.

I also want to fix how we do space accounting for the journal. Right now there are alot of "magic" numbers. For instance, we reserve 1/4 of the journal + the outstanding credits for the current committing transaction, so if there is less than that free on the journal we won't start new transactions, when we could easily just calculate how many log control blocks would be needed for the committing transaction, and reserve that number of blocks + the outstanding credits. This would keep us from waiting when theres plenty of room to start another transaction, which will help performance. As soon as Jan's new jinode stuff goes in I think I'll start working on that.

In the meantime I have a sysfs bug I have to try and explain before I can fix :(.

Tuesday, January 22, 2008


Hmm been a while since I've updated. Working on a bunch of stuff, got some e2fsprogs work done so it now correctly spits out the minimum number of blocks needed to resize the fs if you specify a size thats too small. Also fixed that annoying "you can't mkfs an 8TB fs unless you force me to" thing so now that works properly. Still shouldn't do that unless you are on 2.6.19 or one of the RHEL kernels since the fixes for ext3 to actually work on that large of an FS aren't in the earlier kernels. Its really more of a 2^31 block limit thing, but since most people use 4k blocksizes that comes out to about 8TB.

Been working on a few random RHEL bugs, one that I can't reproduce but IBM can. They are super helpful, willing to test and stuff which really helps in the whole fixing process. Looks like somebody is calling journal_stop twice on the same handle, which shouldn't really be possible since there are things in place to keep that from happening, but we will see.

School has started up again, taking statistics and linear algebra, my last non CSC classes I have to take. When I pull A's in these classes they will finally let me switch my major, which will be nice. Then I get to go into overdrive and take a bunch of 400 level csc classes to try and graduate by next spring. I'm going to start studying for my GRE this summer and take it in the fall to get ready for grad school. I'm glad I'm finally getting somewhere with my education.

Friday, December 21, 2007

Work and such

Well its been kind of nuts the last two weeks. First its sad to see Szulik go. He was a fantastic man to work for, never shy always said hi and chatted you up in the elevator/parking deck/hallway/bathroom/anywhere. I only hope our new CEO will not be like every other upper management person from every other company.

Got up to check my grades this morning, found out that apparently I made a 0 on my final exam in my Introdution to C class and an F in the class.... Went and pulled up the teachers spreadsheet with all of the grades and found my grades and saw that I in fact made an A, so spent 30 minutes frantically calling/emailing everybody in order to try and get that fixed. Unfortunately the teacher has gone home for the break, however the Undergraduate advisor freaking rules, she's going to take care of everything and hopefully I should have my grade changed when everybody comes back from the break. Hopefully after this semester they'll let me actually switch to Comp Sci, just in time to take all of the electives since I've now taken _all_ of the csc core classes, and then take my GRE and apply for grad school. Thankfully grad school only looks at your CSC gpa, which tho my overall gpa is like 2.1, my CSC GPA is a 3.8, so just have to take my GRE and figure out what I want to do for my PhD.

Spent a few days working out what I would talk about for my part of the kernel panic analysis class. Did the taping today, that was alot of fun. Hopefully my examples didn't completely suck, hard thing about teaching this class is I can't use real life examples, I had to write crappy code that should never be used anyway that would panic my box so I could show how the box paniced :(. I also spent some time trying to figure out ext3's disk format. I'm writing in an option to e2fsprogs to give you the minimum number of blocks a given fs can be resized to. Unfortunately because of the way ext2 was designed its kind of crappy trying to figure that out. First you have to figure out how many inodes you have in use, and calculate the number of group descriptors you need in order to handle that number of inodes. Then you have to go through and figure out how many actual data blocks you need, and then see if you need to add more group descriptors in order to handle the data blocks. Then if your last group descriptor is there strictly to support the inodes that are present you have to resize the group descriptor to handle all of the overhead (the inode bitmap/block bitmaps and if there the backup superblock) + 50 blocks because if you go to resize the fs with a last group descriptor thats only the size of the overhead it will fail, you have to have the overhead+50 blocks, yay for arbitrary numbers. The explanation is a group descriptor is kind of useless if it doesnt have blocks to store data in, which is all fine and dandy, but if you are trying to resize a ext2 image to the smallest possible size for a live cd/usb/whatever its kind of crappy to have 50 blocks just hanging out at the end of the fs for no reason.

Anyway, taking a few days off for work, plan on going in and finishing the --minimal thing and have it submitted upstream by next Friday. Tomorrow/Sunday I'm going to do ACL's for btrfs and maybe start on online fsck. I'd like to tackle the tree locking stuff, but Mason would definitely do it better than I so I'll work on some of the low hanging fruit, maybe try and figure out a way to make read performance not suck so badly.

Friday, December 7, 2007

cleaning up e2fsprogs

Finally nailed down the sysfs panic I've been looking at this week. Kind of a weird problem, looks like we can get multiple directory entries for a directory hanging off of a parent, which wreaks all sorts of havoc. I still don't quite understand how we're getting different dentry's for both of the dirents, but hopefully my patch makes this panic go away.

Spent all day today converting e2fsprogs over to have 64 bit block support (so you can have more than 16TB filesystems). Finally got all the kinks worked out so it would build and format my 18TB volume and now I've got corrupt block descriptors, but some sort of progress is being made. I hope to get it all set and formatting early next week and then I'm going to go through and clean up e2fsprogs. The way it handles endianness is really weird, so I'm going to go through and make it so instead of directly setting the filesystem data directly do it through set/get methods that will handle the endianness appropriately. This will also help the hi/lo weirdness. Since ext4 has to maintain backwards compatibility with ext2/3 it can't really modify any of the ondisk structures, so in order to handle large number of blocks you can't just change the old 32 bit number to 64 bit, instead add another 32bit number at the end of the structure that ext2/3 won't read/care about, and then ext4 will read that number and combine the two numbers for the overall count. So with these new functions we'll have a unified and clean way of doing that.