Status

Beagle GSoC progress tracking

This post shall be used for tracking weekly progress, coming week’s targets, weekly issues and other updates related the project.

All the dates belong to year 2014 unless specified otherwise. Week starts on Thursday and ends on Wednesday - in accordance with beagle.org weekly meetings. That's when we make updates.
Source code in available in the following 2 repositories:  Kernel driver, Android app 

Last updated: 14th August ’14 at 10:54 AM IST


Week 13 : 7 August – 13 August

Accomplished:

  • Bone: Tested w/o compression on video playback.
  • Android: Added full screen support and tested with video playback. Check video below.
  • Bone: Compression is now integrated into bone.
  • Android: Tested with and without compression on video playback
  • Docs: Partial work on Android documentation using java documentation APIs
  • Cleanup: Cleaned up kernel driver and java code
  • Docs: Partially documented kernel driver

Full Screen support with video playback test

Issues:

  • Android: There is some issue with decoding on Android. Since decoding requires pixel wise addressing, it is a time taking process. When this is placed in the IO thread, IO is being blocked for sometime giving low frame rates.
  • Android: Starting a thread for decoding on data reception is also slow probably because of the time taken to start a new thread. Will try with maintaining a decode queue and a dedicated thread, decoding the queue.

Plans for next week:

  • Driver: Complete driver documentation
  • Android: Complete java documentation
  • Android: Fixing the issues with decompression. The driver works fairly good even without compression however.

Week 12 : 31 July – 6 August

Accomplished:

  • Checkout the following video demos below
  • Bone: Driver now fully working on bone
  • Bone: Tested with starting an xsession on framebuffer created by driver
  • Bone: Tested with various applications in the xsession
  • Framebuffer: Removed all redundant operations in render functions – No more URB related operations in hline_render
  • Android: Now uses full resolution image to show the frames. Fixed all force closes caused when using a full resolution image.
  • Android: Decreased workload on the USB reader thread by shifting frame updating to an independent runner

Without vertical sync and not full resolution

Final version with vertical sync and full resolution

Issues:

  • Bone: Even after disconnecting the device and stopping the xsession, there are continuous calls to damage handle. This is preventing the unloading of the udlfb module from kernel. rmmode gives module busy error. However, the driver works on reattaching device at this stage.

Plans for next week:

  • Framebuffer: Will integrate compression. Standalone tests were already done however integrating into driver requires some corner case considerations.
  • Framebuffer: Code cleanup. There are many unused functions and operations which came with udlfb but were replaced, commented out or never reached
  • Android: Code cleanup. Reorganise code to decrease workload in USB reader thread with seperate threads for decompress and UI updates.

 

Week 11 : 24 July – 30 July

Accomplished:

  • Framebuffer: Implemented a compression algorithm using Run Length Encoding. This is really efficient, Time: O(N) without using any additional buffers
  • Framebuffer: Tested the RLE on a sample frame. The RLE encoding apoted gave reductions in size from 1.6 MB -> 15.7 KB
    Framebuffer: Documented the RLE scheme. Special cases aren’t documented yet.
  • Bone: Fixed URB allocation issue while testing on BB
  • Bone: Tested deferred io support on bone. Ignoring damage handle calls with resolution besides 1024×768
  • Bone: The setup frame in now working on bone. After the setup, endless calls of image_blip are preventing xinit call. More in Issues
  • Bone: Was able to reach a closure (without endless calls) in probe function and start xserver. More in Issues
  • Demo: A demo video of the driver with auto updation of frames on Android (driver running on PC)

Issues:

  • Bone: Even after ignoring data transmission for the damage handle calls (besides the first frame), the calls are made continously and because of this, call to start xserver on framebuffer isn’t working
  • Bone: During tests, I once reached the end of initial setup and started the xserver on fb. This rendered the 1st frame from fb to the device but later the continious calls to damage_handle started and xserver failed to send the 2nd frame to device and stopped

Plans for next week:

  • Fixing the issue with endless calls to damage_handle (actually image_blip). This should mostly complete the testing of driver on bone.
  • Integrating the compression implemented into the driver and minor modifications to compression for corner cases.
  • Completing the documentation on compression corner cases.
  • Implementation and integration of rle decoder in java.
  • Testing the compression integrated driver on Bone.


Week 10 : 17 July – 23 July

Accomplished:

  • Framebuffer: Raw frame data is now sent to Android. Successfully removed hline_trim and hline_compress functions without any issues. Removed udlfb urbs and replaced with a new usb transfer code. The new code can transfer upto 16384 Bytes at a time compared to the previous urb method which does 256 Bytes at a time. We are however, transferring only one page (4096 Bytes) at a time after setup. Will improve the size/transfer in future.
  • Android: Tested auto updating of setup frame and the xserver frames with a button click on Android.
  • Android: Implemented a Frame class to hold raw frame buffer data and UI updates are done with the data in this frame. New frame from PC will override the current frame data. This updated frame data can be pushed to the ImageView with a button click currently.
  • BBB: Installed the required packages on BBB – fbcon, X11, fbdev and added framebuffer support. Partially tested the driver. (in issues about a full testing)

Issues:

  • Framebuffer: The random page fault is now more serious with more frequent occurrences now. Strange thing is that there is no code changes from the version in which the it’s frequency was very rare.
  • Framebuffer: Increasing the work load after each frame reception on Android seems to be triggering the above issue.
  • Android + Framebuffer: Need to encode the page information along with each page transferred for proper frame updates in future and also a decoding scheme on Android. Currently the new page data is filled in a Bytebuffer of framelength and on overflow, it is being filled from beginning. This won’t work when the page faults are in the middle of the frames.
  • BBB: The data sent per transfer when testing the driver on BBB is just 16 Bytes! Possibly the reason for xinit failure.

Plans for next week:

  • Fixing the paging issue and then we shall have a continuously updating frames on Android.
  • Tweaking the Bytebuffer size being used for a Frame. A higher or lower value is triggering an insufficient bytes to copy pixels error on Android.
  • Once the above two are done, will be encoding damaged pages information along with the page data.
    Testing in BBB and resolving the write buffer as just 16 Bytes issue.
  • Once these issues are resolved, we shall be having a driver that can run a xserver and update frames on Android automatically. The next work would be on compression.

This is a screenshot of the auto updated frame from the raw frame data sent from driver. I tried recording a video only to find out that the video was corrupted :-( Will make one for next week report.

Frame

This frame is auto-updated from kernel driver


Week 9 : 10 July – 16 July

Accomplished:

  • Framebuffer: Understanding the way udlfb – kernel framebuffer, works.
  • Framebuffer: Removed some of the displayLink specific commands sent along with the frame data by the udlfb driver and tested working.
  • Framebuffer: (Work-in-progress), Removing the issues when commenting out the compress and trim functions in the driver.
  • Framebuffer: Figured out the reason for BUG: Soft lockup as shown in the kernel log here,  Tested some patches but the effects are to be figured. (issue)
  • Android: Since getting raw data from driver to device is having some issues, I obtained framebuffer data on PC, transferred it to device and completed the raw frame processing on Android. Images below.
RGB888 - Bitmap

RGB888 – Bitmap

RGB5 - Bitmap

RGB888 – Bitmap

Issues:

  • Framebuffer: BUG: Soft lockup – The reason is because the program gets stuck in an infinite while loop on removing the compress function.
  • Framebuffer: When the hline_trim function is removed kernel paging error occurs. Tried getting the prefetch calls out but didn’t help
  • Framebuffer: The BPP (Bits per pixel) value is to be found out. The raw data collected on linux is 24bpp (RGB888) but however, looking at the udlfb code and call backs, it looks like the driver is using 16bpp (RGB565). Not a major issue

Plans for next week:

  • Framebuffer: Resolve the framebuffer issues.
  • Framebuffer: Send raw data onto Android and see how it is rendered with the existing code for RGB565 – bitmap decoding.
  • BeagleBone: Test the driver on a BBB. So far, all testing and development is being done on Linux PC. (Should pass the test mostly)


Week 8 : 24 July – 30 July

Accomplished:

  • Framebuffer: Attempted to start xserver with DisplayLink userspace driver and udlfb (issues)
  • Framebuffer: Success with starting an xserver on the framebuffer added by udlfb. Details are updated on git README in udlfb repo. fbgrab or fbdump read the expected screen in the framebuffer. Shown in the fbgrab image below.
  • Android: Changes to the way data received is displayed on Android. The changes are how the TextView data is built without resulting in a log sequence of Garbage collections.
  • Framebuffer: Attempted to use core keyboard as input device for framebuffer (issues)
Image from fbgrab on Framebuffer created by driver

Image from fbgrab on Framebuffer created by driver

Issues:

  • Framebuffer: Displaylink userspace driver with udlfb is giving segmentation faults. Fixed the EDID segmentation faults though. This has become hard to debug as the segmentation fault crashes my PC and the log couldn’t be saved. Besides, even with the log files, tracing the root cause for the fault is hard to detect as the logs only show Hexadecimal address in the trace and not the function call or the module that caused it. Alternative found. Can be ignored for now.
  • Framebuffer: Other methods either resulted in a crash or when the x server starts, it doesn’t start on the right framebuffer probably. Because there is data receiption at the device end, no new kernel logs on starting xserver on fb. Also, the framebuffer fbgrab/fbdump doesn’t show the expected frame (only a green screen). Alternative found. Can be ignored.
  • Framebuffer: Framebuffer data encoding has to be figured out. The raw dumps on the Android device show negative numbers also. Also format of the frame sent by the framebuffer has to found out. Mostly PPM.
  • Framebuffer: Framebuffer uses compression before sending the data to the Android device. The algorithm has to be understood and the respective counter part to decompress has be done on the Android side.
  • Inputdevice: When the core keyboard is added as inputdevice the xserver start command gave errors about the inputdevice entry. On commenting this entry, the xserver starts.

Plans for next week:

  • Framebuffer: Figure out the encoding of the data. Since negative values are also received in Java code.
  • Framebuffer: Understanding the compression used in framebuffer (things would be simple if it is RLE ) and writing the counterpart in Java for decompression. Alternatively compression may be bypassed for initial stages.
  • Android: Conversion of the framebuffer frame data into bitmap that can be showed in a ImageView on Android.
  • Android: Decompression part in Java if it is not bypassed right now.
  • Inputdevice: Will try to fix the issue with using core keyboard as an input device for framebuffer added by udlfb.


Week 7 : 26 June – 2 July

Accomplished:

  • ADK driver: ADB +ADK working: ADB over Wifi is now working for logs while in ADK mode
  • ADK driver: Testing with test data from sent from userspace – test data is generated in kernel space earlier
  • Android app: Fixed fc on opening w/o accessory attached.
  • Android app: Added accessory connection status, read values and write options in the app UI
  • Framebuffer: Added android device as DisplayLink device for Framebuffer. Probing works now, only on the interface of interest – No duplicate probing issue.
  • Framebuffer: Updated endpoint address for BULK writes to device. Now working.
  • Framebuffer: Logged raw initialization sequence data sent by the framebuffer to DisplayLink devices on the Android device. Will be analyzing this for future work. Can be seen in the image below.
Raw data dump into a scroll view

Raw data dump into a scroll view

Issues:

  • FB data interpretation: The meaning of the initialization sequence data from framebuffer is not clear. Also the encoding of the data might be needed. Not sure if it is UTF8 encoded right now.
  • FB Triggers: It is unclear as of yet if there is a need for another module or userspace code which should be issuing the triggers for sending updated frames from udlfb driver. Right now the driver isn’t doing any work after the probe function.
  • Android app: A couple of FCs when the device is detached from ADK mode. Should not be a big issue. Can be fixed easily.

Plans for next week:

  • Interpretation of the data sent from the FB driver in the initialization sequence only if that happens to affect the working of a basic version.
  • Finding out how the continuous frame updation part works in the driver. May be a need for external module?
  • If the above two goes well, try to interpret the frame data and see if we can get an image on the Android device. udlfb uses run length encoding on the frame data sent to a DisplayLink device.


MID TERM WEEK

Week 6 : 19 June – 25 June

Accomplished:

  • Fixed identifiers encoding issue: Accessory app now shows up on device automatically when it enters accessory mode.
  • Completed Kernel – Java communication: Tested with about 3 bytes. Will be extending after the cleanup.
  • Partially tested Java – Kernel communication: Data received but encoding has to be tested and there are some failures which are to be tested as well.
  • Android app from scratch: Developed the android app from scratch, available on git. Currently it can only be used for communication testing through adb logs. No UI messages but that’s an easy step anyway. It’s working.
    • Userspace using libusb (in issues)
    • Kernel driver only on the ADB interface (in issues)

Issues:

  • Java – Kernel communication: This has to be tested well. There were a couple of write failures on Android end which resulted in indefinite wait on the kernel driver front. The driver read from device code also has to revised to correct for such indefinite wait issues.
    • libusb: NULL handle with libusb_open_device_with_vid_pid
    • kernel: Read function hangs up with error -110 (our earlier friend with control messages :/). Will read about adb and see if there should be an initial sequence that has to be sent before attempting read logs from BULK IN endpoint.

Plans for next week:

  • ADB + ADK : Check if this is possible. Some documentation on ADB. Just a couple of more attempts. This would be a great debugging tool! There is a slightly, not as good, alternative if this fails.
  • Device – Kernel tests: Test the java – kernel communication. Figure out the reasons and apply fix for random failures. A few pointers: The write handle on Android being nulled.
  • Kernel – Android for large data: Extend this for larger amounts of data given to the driver from outside, currently from userspace.
  • Start work on DisplayLink driver – udlfb


Week 5 : 12 June – 18 June

Accomplished:

  • Fixed the usb_control_msg returns error code -110 issue
  • The device is now set to ADK mode from the Kernel space.
  • Probing and reprobing – On connection probing happens and issues commands to put the device in ADK mode. Device disconnects by itself and reattaches in ADK mode. Probing happens again but a different set of tasks are done this time.
  • Registering device on /dev/* for further communication with the device.
  • Some reading on the kernel framebuffer and DisplayLink driver udlfb.
  • Extracted a framebuffer from /dev/fb0 – more about this in issues
  • Testing accessory setup in ADB+ADK mode. (So far all accessory setup was in ADK mode only). More in issues.
  • Initialized the Android repo. Testing is still done on a different project (not yet pushed to github). Lack of ADB while in ADK mode is a big pain as of now.

Issues:

  • Not able to interpret the format of frame extracted from /dev/fb0. Will be using DisplayLink instead.
  • Although the device is now attached in ADB+ADK mode, eclipse logcat shows device disconnected. No success on ADB while device in ADK mode yet.
  • When the accessory setup is done in kernel space, even though the identifiers are right, the corresponding accessory app doesn’t show by itself (unlike in userspace ADK setup).

Plans for next week:

  • Completing kernelspace-java communication. Should be able to send data over bulk, receive it in the app and show the data on UI.
  • Try to have ADB log running while in ADK mode.


Week 4 : 5 June – 11 June

I was in San Francisco till 8th June and arrived only on 11th June early morning. No progress as I was on a trip at SF or travelling most of this week.


Week 3: 29 May – 4 June

I was in San Francisco attending a conference from 1st – 5th of June.


Week 2 : 22 May – 28 June

Accomplished:

  • ADK mode from user space code now working.
  • Detection and saving of endpoints in the interface.
  • Partial work with issuing control messages.

Issues:

  • usb_control_msg returns error code -110

Plans for next week:

  • Fixing the above issue and complete adk mode setup.
  • Setting up the Android app.


Week 1 : 19 May – 21 May

Accomplished:

  • A kernel module which can register a usb-driver for Android device.
  • Probing when device is connected or disconnected

Issues:

  • Had issues with printk’s after usb_register but fixed just a while ago.

Plans for next week:

  • Request Android device to change to ADK mode from the usb driver
  • USB endpoints to facilitate data transmission.


Week -1 : 12 May – 18 May

Accomplished:

  • Introduction video done
  • Status blog started
  • Wiki initialized
  • Updated complete proposal on blog
  • Some reading on kernel modules and USB driver

Issues:

  • None

Plans for next week:

  • A kernel module/driver which can register for an Android device
  • Probing on various device connection status
  • A basic structure that can be worked with


Week -2 : 5 May – 11 May

Accomplished:

  • Getting started
  • Community bonding
  • Reading up on USB driver

Issues:

  • Minor – Reference resource for coding practices

Plans for next week:

  • Introduction video
  • Start tracking blog post
  • Wiki initialize
  • Kernel coding practice

 

One thought on “Beagle GSoC progress tracking

Leave a Reply