[OSDI] Read Frame Usage in Kernel Code

Intro

When a user-space process ask for memory spaces, the kernel may give out some "pages" which can be located by virtual addresses. However, kernel only gives out real physical memory space when the process start to write data on that address. That is, the pages are able to convert to a real page "frame" after their first write.

The conversion is done by looking up the page table. The page table is different from process to process, and it is stored in every process memory descriptor.

Structures

In Linux kernel, the structure and their hierarchies can be listed below:

• task_struct: this describes a process
• (char *) comm: name of the process
• (struct mm_struct *) mm: memory descriptor of the process
• (struct vm_area_struct *)mmap: start pointer for virtual memory areas

And, the virtual memory area of a process is a linked list structure in kernel code, so we start from the first one which is stored in task->mm.

• vm_area_struct: a virtual memory area
• vm_start: start virtual address
• vm_end: end virtual address
• vm_next: next vm_area_struct in the linked list, NULL if it's the last one

Process

To scan through all the process, and find the one we want:

struct task_struct *task;
for_each_process(task) {

    if( task == NULL )  continue;
    if( task->mm == NULL )  continue;

    if( strcmp(task->comm, "reclim-me")==0 ){
        ...
    }
}

To scan through the virtual memory areas of one mmap:

for( vma=mm->mmap ; vma!=NULL ; vma=vma->vm_next) { ... }

And, in real world, there are several layers for the page table, which means we have to look up one after one, and check if the address exists:

pgd = pgd_offset(mm, address);
if (!pgd_present(*pgd)) continue;

pud = pud_offset(pgd, address);
if (!pud_present(*pud)) continue;

pmd = pmd_offset(pud, address);
if (!pmd_present(*pmd)) continue;

pte = pte_offset_map(pmd, address);

ptl = pte_lockptr(mm, pmd);
spin_lock(ptl);
if (pte_present(*pte)) {
sum ++;
}
pte_unmap_unlock(pte, ptl);

Implementation

Finally, for this OSDI lab, we have to print out the number of frames of a process. To know the current answer, we can:

• cat /proc/<pid>/statm | awk '{print $2}'
• get_mm_rss(task->mm)

To get states of virtual memory areas:

• cat /proc/<pid>/maps

However, we have to implement our own calculation in this lab, and here's the code:

https://github.com/heronyang/frame-usage

By detecting the number of frames of the program "reclim-me", we learned that kernel gives out the memory only when the process starts to write.

Other

It seems that the TA mistyped "reclaim" into "reclim" which is a meaningless word; however, I follow the original code in my github at this point.

Reference

• http://www.makelinux.net/ldd3/chp-15-sect-1

[OSDI] Major and Minor Number for Device Nodes

As we all know, we operate the devices as a file under linux. And the files are listed under /dev, such as /dev/sda, /dev/ram0, etc. However, in OSDI course, the instructor mentioned that the name, sda, ram0, etc, is only for human to read. And, for the machine, we are having a major and a minor number for each device. How they are converted is as below (it's one-to-one conversion):

• device.txt
• /sys/block/*/dev /sys/block/*/*/dev

Reference

• http://stackoverflow.com/questions/16863651/how-do-you-find-the-major-and-minor-numbers-for-devices-in-linux
• https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/devices.txt

Setup Debian VMs on VMWare

To have a better experience on Debian VMs on VMWare, it's always a good idea to install "VMWare-Tool". However, some settings have to be made, and this guy had written them down: http://benohead.com/vmware-the-path-usrbingcc-is-not-valid-path-to-the-gcc-binary/

[OSDI] Virtualization v.s. Emulation

Often, I question myself the difference between "Virtualization" and "Emulation". Hopefully, I found a good explanation on the Internet as below:

Difference by Definition

Virtual machines make use of CPU self-virtualization, to whatever extent it exists, to provide a virtualized interface to the real hardware. Emulators emulate hardware without relying on the CPU being able to run code directly and redirect some operations to a hypervisor controlling the virtual container. -- stackoverflow (geekosaur)

It turns out the main difference is whether (part of) the code is directly run on the host machine (virtualization) or on a software-emulating hardware (emulation).

Again, from this website, we learn:

• Virtualization: involves simulating parts of a computer's hardware
• Emulation: in emulation the virtual machine simulates the complete hardware in software

Example

Virtualization

• Wine

Emulation

• Bochs
• QEMU

Reference

• http://jpc.sourceforge.net/oldsite/Emulation.html
• http://stackoverflow.com/questions/6234711/what-are-the-specific-differences-between-an-emulator-and-a-virtual-machine
• http://stackoverflow.com/questions/1584617/simulator-or-emulator-what-is-the-difference

Fixed, Fluid, Adaptive, and Responsive Website Designs

It's quit often to see these names while developing websites. However, I had no chance to learn deep into these things until today. In my own understanding, these names are mainly focus on the "dynamic width" problem all developers may face while designing the layout.

Background

Website width varies from 800px on small mobile devices to 2560px on expensive Macbook Pro Retina. To have the best user experience, most of the time we should test the website on different devices (screen sizes), and make sure the everything is working well (at least it's under the designer's control).

First, it's always a good idea to start the design on the size that most people are using. We can check the "Screen Resolution Statistics" on w3schools.com here: http://www.w3schools.com/browsers/browsers_display.asp

Second, we should test the website on different widths. The simplest way is to adjust the width of your browser, like:

(using http://www.fiftythree.com/pencil as example)

Original width:

Decreased width:

Remember that it's okay to let the user scroll up and down (the height of the website is larger then the screen size), but it's a pretty BAD idea to let the user scroll left and right since most of the mouse doesn't have that kind of control button or trackball.

Design Solutions

I can still recall the days in elementary school, my teacher asked me to put "screen resolution of 1024x768 pixels" at the bottom of the website. I think that's because

• we are not having good CSS/Javascript at that time to solve the dynamic width problems, 
• users' devices are mostly having the same resolution (1024x768 pixels), 
• and it's easier.

Now I learned that this kind of website design is called "fixed-width", it's kind of old fashion but classic. And, here, I am listing out all the comman design solutions:

Fixed

"a set width and resizing the browser or viewing it on different devices won’t affect on the way the website looks." -- teamtreehouse

This is the kind of design I just mentioned, and it's widely used for the traditional websites. Here are some examples:

• http://yahoo.com.tw/
• http://www.nctu.edu.tw/ (center aligned) and http://dcpc.nctu.edu.tw/ (left-top corner aligned)
• http://www.redenvelope.com/
• http://www.cmu.edu/

- Pro: The layout is totally under controlled since the designer doesn't have to care about the screen resolution, but only cares about the selected and fixed width instead.

- Con: It may be hard to read on different size of screens, users have to scroll left and right if the website width is larger than the screen width.

Fluid

"built using percentages for widths. As a result, columns are relative to one another and the browser allowing it to scale up and down fluidly." -- teamtreehouse

The main point of fluid is that the columns are assigned specific "percentages" of the whole width, which means the columns are having fixed proportions no matter what the screen width is.

• http://css-tricks.com/examples/PerfectFluidWidthLayout/
• http://andreagandino.com/

- Pro: The website remains almost same designs on different sizes of screens, which is corresponding to the fixed proportions.

- Con: Designs will be destroyed if the screen width is too small. For example, 20% of a 800px mobile screen is 160px, which may not be large enough for displaying the original content. Therefore, it may be a good idea to set the minimum website width (apply "min-width" in CSS).

Adaptive

"introduce media queries to target specific device sizes, like smaller monitors, tablets, and mobile." -- teamtreehouse

Adaptive website designs offer serval layouts based on different types of devices, such as smaller monitors, tablets, and mobiles. Just for example (may not be true in real world practice), they may apply "original layout template" for width over 1024px, "tablets layout template" for width between 600px and 1024px, and "mobile layout template" for width less than 600px. Really good explanation I found is in this website.

* Some people think adaptive designs are as same as responsive designs (which will describe later). I think they are similar and with slight differences.

Original:

Smaller Screen:

Mobile:

• http://heron.nctucs.net/
• http://liquidapsive.com/
• http://www.joyintermedia.com/
• http://www.stanford.edu/

- Pro: This works pretty fine on most of screens, it's offering different layout templates based on it's type (traditional monitor, tablet, mobile, etc).
- Con: Instead of designing "one" layout, the designer should bring out "several" layouts to reach this purpose.

Responsive

"built on a fluid grid and use media queries to control the design and its content as it scales down or up with the browser or device." -- teamtreehouse

It's pretty the same as the adaptive design, but it applies the "fluid" design on different templates which are targeting to different types of devices. And, I believe this design offers the best user experience since the designer has to care about every layout for any size of the screen.

• https://devart.withgoogle.com/ 
• http://mondaymusic.es/

- Pro: I think this is the best design for the websites, it cares every layout possibilities for users on any devices.
- Con: It may cost a lots to reach this design, and the code could be complex. More tests should be held to make sure there's no bug.

Note

• Lots of websites are not supporting mobiles in the same page, but offer any other version instead. It detects the devices you are using, and redirect you to the mobile version one if you're on a mobile device. Example is here: http://facebook.com/ and http://m.facebook.com/
• To detect the screen width (viewport width), we should apply "CSS Media Queries".

Sum

I draw the graph as the summary of this post, and please tell me if there's any mistake in the graph. Also, I believe that these designs are just "principles", which means that we don't have to fix to any of them, but wisely use them in different situations.

Reference

• http://teamtreehouse.com/library/build-a-responsive-website/introduction-to-responsive-web-design/fixed-fluid-adaptive-and-responsive-2
• http://ux.stackexchange.com/questions/24406/what-is-the-exact-difference-between-fluid-and-responsive-design
• http://stackoverflow.com/questions/17439352/bootstrap-responsive-layout-vs-fluid-layout
• https://www.responsys.com/blogs/nsm/cross-channel-marketing/difference-responsive-adaptive-web-design/

* thanks Shumin for raising this topic

Keylogger for Mac OS X

For security or hacking purpose, people install keyloggers on their own laptops or on other people's laptops. It's quiet interesting to play around this kind of program.

logkext: https://github.com/SlEePlEs5/logKext (this contains version 2.4 at this point)
However, in its github, there's no compiled packages. The older version 2.3 at Google Code is having compiled packages (https://code.google.com/p/logkext/downloads/list).

This is  an open source keylogger for Mac, to start the application simply type "sudo logKextClient".

Removing www.mrlmedia.net Virus (Mac)

I found that my Chrome is acting weird these days. Strange Ads are popping in lots of websites, and I have no idea how to turn it off. Then, I started to figure out the solution.

Here are some steps may help in most cases, but not in this case:

• Turn off extensions/plugins, so that there's won't be unwanted js code while loading the websites. => However, the problem remained the same; and different browsers are having the same problem (tested on Chrome and Safari)
• Clean cookies and other personal settings  => not helping anything
• Remove everything under "~/Library/Application\ Support/Google/" and "/Library/Application\ Support/Google/", then reinstall Chrome => not helping anything, since the problem happens on different browsers

So, I start to trace the problem on Chrome Developer Tools (Networks):

1. The right and original request for Google Search.
2. Chrome is trying to get www.mrlmedia.net/get-js...; however this is the problem.
3. After getting get-js, Chrome start to run sf_main, and this is loading the Ads
4. direct.html is the IFREAM for the Ads

The root problem is that "somebody" request the get-js while I am browsing the website. But, I don't know who's the guy.

However, it's easy to block the request by adding www.mrlmedia.net into the blacklist:

• vim /etc/hosts
• add 127.0.0.1 www.mrlmedia.net

Then the get-js request will be blocked:

The Ads are removed now.

Better Solution

Thanks for Niccolò Ventura and Steven Foong handing solutions in the comments of this post, and I am writing them down here for conclusion:

>> sudo rm /Library/LaunchAgents/com.vsearch.agent.plist /Library/LaunchAgents/com.vsearch.daemon.plist /Library/LaunchAgents/com.vsearch.helper.plist Library/Frameworks/VSearch.framework # so the virus won't auto-start when the system is up
>> sudo rm -fr /Library/Application\ Support/VSearch/ # remove the virus