GSoC ’14 proposal for Beagle.org

Android based remote display

Organization: BeagleBoard.org

Abstract: “One cable to rule them all”. I intend to build a system where all the basic peripherals – keyboard, mouse and display, can be connected to BeagleBone by attaching it with a USB cable to Android phone. This will be implemented as a kernel module and an ready to use out-of-box app for Android. This module shall have a greater reach and also serve as seed for extending to other systems – Windows, iOS, Ubuntu phones etc.

I had a well formatted proposal in my submission. Copy pasting messed up the formatting and I'm too lazy to correct it now.
About you
  • What is your name?
    Praveen Kumar Pendyala (Praveen)
  • What is your email address?
  • What is your eLinux wiki username?
  • What is your IRC nickname?
  • What is the name of your School and in what country?
    Indian Institute of Technology Bombay, India.
  • What is your primary language? (We have mentors who speak multiple languages and can match you with one of them if you’d prefer.)
  • Where are you located, and what hours do you tend to work? (We also try to match mentors by general time zone if possible.)Have you participated in an open-source project before? If so, please send us URLs to your profile pages for those projects, or some other demonstration of the work that you have done in open-source. If not, why do you want to work on an open-source project this summer?
    Hyderabad, India. I tend to work at night (through night). I’m flexible about the timings though. I have been working on an open-sourced project for two years now. Source available at https://github.com/praveendath92/MDroid All projects I have worked on so far are open-source.
About your project
  • What is the name of your project?
    Android-based remote display
  • Describe your project in 10-20 sentences. What are you making? For whom are you making it, and why do they need it? What technologies (programming languages, etc.) will you be using?

    The aim is to build a system where an Android device can act as a display, keyboard and mouse to BB, all at the same time – “One cable to rule them all”. This shall be implemented in the kernel space, more like a kernel USB driver, rather than an app running in userspace on a Linux OS.

    A portable remote display, controller and input device for a BB will all be available with single USB connection to something that is so ubiquitous, Android. This could be a great aid for the developers – absolute beginners and experts, in developing and debugging of applications running on BB. The task can be divided into sub-tasks as follows,

    • Communication: ADK using AOA protocolAll communication will be over USB where BB will be acting as an accessory that talks to Android using Android Open Accessory Protocol (AOA) [1]. More detail are covered in the timeline
    • Kernel ADK Host: Will be writing own driversAlthough there are a lot of open-source implementations [2], [3] to implement the ADK host on BBB, all of them are at user-level using libusb and doesn’t go with our targets of kernel level implementation so, I will be writing a module which can put the device in ADK mode and communicate. More on this in the last section, ‘related tasks for proposal’
    • Display: udlfb[4], framebuffer[5]udlfb is the original, fully functional, kernel framebuffer driver used by DisplayLink – the company that makes USB display modules. Essentially, udlfb has functions to fetch frames from framebuffer and able to transmit this to the DisplayLink devices. So, modifying udlfb to get frames for ADK host module, instead of sending to DisplayLink devices or writing a new module, referring to udlfb, for fetching frames from framebuffer are two options for this. There are a few other kernel drivers developed on top of udlfb – displaylink-mod, displaylinkfb and xf86-driver-displaylink, which can be seen as potential substitutes.
    • Android: Android SDK, ADK sample appThere are already sample apps which demonstrate on how to get data from ADK host [6]. I will be implementing the required video display, touch and keyboard events catcher on top of these.
    • Data compression: If required, libdlo/udlfbDisplayLink drivers also have data compression techniques. After doing some bandwidth tests on ADK with AOA protocol we will come to decision. Since a compression would also mean decompression on the Android app side and this shall be increasing the expected work on Java side.
  • What is the timeline for development of your project? The Summer of Code work period is about 11 weeks long; tell us what you will be working on each week.
Till week 0: (present – 18th May)
Aim: ADK connection in kernel space, Coding practices and debugging tool (ADK + ADB)
Details: I was successful at writing a kernel USB driver to detect my device and tested an user-space code to put the device in ADK mode [3]. I intend to translate [3] into kernel space. I gained good knowledge about USB drivers in Linux kernel and kernel programing while working on my proposal but this is limited to writing small test modules and I intend to spend sometime to get comfortable with the coding practices in this space to produce clean code in the weeks to come. Also, I plan to test the ADK and ADB working together as this shall be my debugging tool.
Week 1 – 2:  (19th May – 1st June)
Aim: Kernel USB driver for BBB as ADK host
Details: By now I will be having a module which can detect my device connection and put it in Accessory mode. In this period I shall be extending this driver so that it can communicate with the Android device in one of the 4 modes available (actually 6 if we consider backward compatibility too). Android device will be requested to use productId – 0x2D05 – accessory + audio + adb mode. I will be developing functions that can send/receive data through each of the available interfaces – accessory, audio. The usage plan is:
      • accessory (2 bulk interfaces) – keyboard, mouse and other control signals
      • audio interface – video transmission
      • adb interface – for debugging
Week 3 – 4: (2nd June – 15th June)
Aim: AOA protocol, Receiving ends at Android and testing.
Details: The host has functions to send data and now I will be implementing the receivers which are capable of handling data sent from the host through audio interface (hopefully isosync). Also, I will be writing routines to send data over accessory interface to host. Once these data receivers and transmitters in the java part are done I will be testing against the host modules for a proper communication.
Week 5 – 7: (16th June – 6th July)
Aim: Current screen frame extraction.
Details: The aim of this period will be extracting a frame of current display. Not by taking screen shots since that has a considerable delay and from what I found out it gives a frame rate of around 10 fps only. We are aiming for a better frame rate so, I shall be using displaylink’s libdlo library [4] or framebuffer [5]. Modifying libdlo to get frames for ADK host module or writing a new module, referring to libdlo, for extracting frames directly from framebuffer are the two available options. The goal is to be able to extract frames of the current display at a considerable good frame rate. ~20 fps should be fine.
Week 8: (7th July – 13th July)
Aim: Framebuffer transmission and processing on Android
Details: Extracted framebuffer will be transmitted over audio endpoint. Will also work on processing this framebuffer at the Android end and displaying it on screen.
Week 9: (July 14th – July 20th)
Aim: Stream of framebuffers. Stream processing on Android
Details: Will target extraction of framebuffers as a stream, transmit to Android device and further processing this and displaying it on screen.
Week 10: (21st July – 27th July)
Aim: Keyboard input and code cleanup
Details: Will be implementing keyboard input interface. Data input through Android device will be transmitted over the accessory endpoint and the received values will be processed using a key processing module. Any required code cleanup in the modules developed so far will be done too.
Week 11: (28th July – 3rd August)
Aim: Mouse and touch events support.
Details: Will be implementing methods on Android app to check the coordinates of touch or drag events and accordingly sending information to BBB. This involves the following steps, in that order:
      • Local coordinates finding: Identifying the coordinates of touch even in the local coordinate system – Android context.
      • Mapping: Implementing a mapper which can do the translation for a corresponding mouse click on BBB. We already know the resolution of the BBB display and the resolution of tablet/phone. By running a couple of tests we can find out the mapping.
      • Transfer: Once we get the final coordinates of the click on the screen we will be sending this to Android over the accessory part of adk.
      • Execute: Based on the data received, mouse click events will be executed on BB.
Week 12: (4th August – 10th August)
Aim: Integration checks, Bug fixes, improvements and buffer period.
Details: Will be checking that the integration of all the modules and routines developed so far is working together as expected. Any bugs will be fixed and possible improvements will be made. I think this could also be like a buffer if any of the previous sub-tasks were left out.
Week 13: (11th August – 18th August)
Aim: Documentation, TODO list, code commenting checks and submission work.
  • Convince us, in 5-15 sentences, that you will be able to successfully complete your project in the timeline you have described. This is usually where people describe their past experiences, credentials, prior projects, schoolwork, and that sort of thing, but be creative. Link to prior work or other resources as relevant. Provide references such as professors who know your work if you like. Please feel encouraged to visit our IRC channels, #beagle and #beagle-gsoc on irc.freenode.net, and ask for help.

I’m persistent, hard working, and more importantly, very keen on completing this task. I have a very clear idea about all the aspects of the project – technologies that I’m going to use, possible alternatives, sample snippets and source codes of relevant tasks. I have  great perseverance and I am very productive. I have never seen myself failing in any task I took up, and I’m entirely sure that this project will not make an exception to my trend.Last November, I had to work on a paper deadline for a top-tier conference within 15-20 days and I completed the task producing about 7000 lines of code [7]. Oh, and we were nominated for Best paper award. All my co-authors were in US and I was the only one working remotely from India maintaining a good collaboration with them all the while.

My perseverance and dedication are hinted at by my academic performance. I stood 1st in India in IIT JEE 2010, an exam taken by more than 470,000 students – this, I believe, is, in addition to being a manifestation of my aptitude for engineering, a function purely of my commitment to performing well.
I worked on a lot of projects – both individual and in teams – with and without mentors and irrespective of the initial conditions I met a commendable success. I pushed all codes to github [8]. I have been working on a long-term project which started as a hobby for over 2 years now, and am continuing to support it [9]. I maintain a good sense of attachment with my projects and always strive to complete them soon – yes! I work for long hours until I reach a point where I can be very certain about it’s completion in the given timeframe. I’m always open to suggestions, and in fact, I’d love to incorporate any suggestions for the better.
[7] Feel free to contact Joonho Kong (joonho.kong@rice.edu) – our lead author. Codes are up at:
You and the community
  • If your project is successfully completed, what will its impact be on the BeagleBoard.org community? Consider who will use it and how it will save them effort. Give 3 answers, each 1-3 paragraphs in length. The first one should be yours. The other two should be answers received from feedback of members of the BeagleBoard.org community, at least one of whom should be a BeagleBoard.org GSoC mentor. Provide email contact information for non-GSoC mentors.
  1. “One cable to rule them all” - This could be an apt description of this project. A display, a mouse and a keyboard ! Just by connecting just connecting a single usb cable, between the BBB and the Android device. This shall be coming out as a ready to use out-of-box package so, be it an absolute beginner or an expert user, they can all swiftly adapt this solution for their needs.
    This will surely be a good aid while testing and development of applications on BBB. I could see this as a great tool for field testing – just pull out your device, test your application, debug, fix and done. No need of carrying bulky devices around.
    Besides the out-of-box uses, the code produced can be used as a base for extending it to other systems – windows and linux (Ubuntu phones are coming up!). I plan to maintain a good modularity so, by simply replacing the ADK communication module with the relevant ones, this can easily be extended to other platforms.
  2. Vlamdimir (av500) – “1) many Beagle users with to attach an LCD screen, if not for the simple reason to see “something”, and old Android phone or tablet can be a cheap solution to achieve this goal.
    2) using ADK to talk between the BBB and Android devices offers many more possibilities for future work like also sharing the phone’s internet connection over the same link etc or making a sound dock using the ADK2.0 audio API.
    3) having a “BeagleBone Remote Display” app in the Android market also can act as a nice demo and marketing tool for Beagleboard.org, especially if it works out of the box with stock BBB software.”
  3. Vlad Victor Ungureanu (vvu) – “This project captured my attention because nowadays everybody has a tablet and it is super easy to interface your BBB with a tablet based display then using an LCD which needs more hacking with device trees. Also after the BBB side software is done the Java app that will run on the Android device can be converted in an Windows/Linux app that does the same thingie. Mainly it can be transformed in a VNC app. Again there are many problems with HDMI TVs that do not support the modes of HDMI that the BBB has.”
  • What will you do if you get stuck on your project and your mentor isn’t around?
I always divide my tasks into subtasks and this project is no exception to that. Like I mentioned in Q2 of ‘about your project’, my project is divided into independent (to a certain extent) subtasks. Should such a situation arise, where I got stuck and my mentor isn’t around, I shall make changes to my timeline and proceed accordingly with other subtasks. Also, since most of the approaches I intend to take for my sub tasks are based on popular tools, I could use community forums and see if I can get some help. And not to rule out the possibility that I may figure out the issue when I return after a while.
  • Please submit to the beagleboard-gsoc mailing list a statically-linked ARM Linux “hello world” style executable that prints out your name and the date. Please keep it under 1MB. Provide a link here to that executable as archived on the mailing list and provide any instructions required for invoking it. You are welcome to test it on an ARM QEMU environment. Please feel free to visit our IRC channels, #beagle and #beagle-gsoc on irc.freenode.net, and ask for help.
Posted on the mailing. Here it is for reference. Tested on ARM QEMU and instructions to run are available in README.md
  • Is there anything else we should have asked you?Yes. Below are a few questions that I think might be relevant.

How many hours per week can you spend on the project ?
I shall be completing my bachelors by the end of April and I shall be completely free till 2nd week of September. I don’t have any alternate plans for my vacations. So, in short, I can spend as many hours as it may need for successful completion. Around 50 hours per week and I can spend up to as long as it requires.
Did you do any tasks/tests related to this proposal ?
I started by testing BeagleBoard as ADK host by burning a custom firmware on BeagleBoard [10]. I have also implemented a kernel usb driver which detects my device when plugged [11] and I tested a user-space implementation for changing device to ADK mode [3].

Leave a Reply