Each project will have a short, medium, and ultimate goal. The ultimate goal is not expected to be finished in one summer, but we hope that the student will continue as a contributor and see it finished.

All projects will end with a demo video that will be posted on YouTube and a tutorial providing a step-by-step description of how the demo can be replicated by others.

Project Proposals

PCD Video Format

Starting from the discussions at, we would like to implement a file format for storing sequences of clouds. Things to have in mind:

  1. For encoding point cloud data, we would like to distinguish between a container format to manage frames of different format.
  2. In every frame, a codec is specified that is used for en- and decoding.
  3. Codecs that we can provide for point clouds are: ASCII, "binary dump" and "compressed (octree)"
  4. Every codec defines its own point cloud parameters.
  5. A possible container format could be "tar". Filenames contain timestamp information.
  6. Optional meta information should be stored in separate frames (EXIF, Camera/Coordinate system transformation)
  7. a VLC-like video player for 3D video files with complete UI and performance optimisations.

Extension of pcl::people module for detecting people in unconventional poses

In pcl::people module, a method for detecting upright people in RGB-D point clouds is implemented. We want to extend this module in order to make possible the detection of people in positions other than upright. The current algorithm could be improved by implementing some of the techniques presented in

Benjamin Choi, Cetin Mericli, Joydeep Biswas, and Manuela Veloso. Fast Human Detection for Indoor Mobile Robots Using Depth Images. International Conference on Robotics and Automation (ICRA), 2013
Hao Zhang, Christopher Reardon, and Lynne E. Parker. Real-Time Multiple Human Perception with Color-Depth Cameras on a Mobile Robot. IEEE Transactions On Cybernetics B (TMSC-B), vol. 43, no. 5, pp. 1429-1441, October 2013

for selecting point cloud clusters candidate to belong to a person which are then classified with more sophisticated people detection techniques.

Human action recognition from skeleton information

Implement an algorithm able to recognize human actions from pre-segmented RGB-D sequences by classifying the skeleton information provided by PCL's skeleton tracking algorithm. An example of paper that could be implemented is:

Ferda Ofli, Rizwan Chaudhry, Gregorij Kurillo, Reneé Vidal, Ruzena Bajcsy - "Sequence of the Most Informative Joints (SMIJ): A new representation for human skeletal action recognition", 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

If time permits, this technique could be extended to be able to classify actions online, segmenting them out from a real time stream.

Multiview Registration

Surface registration is a fundamental step in the reconstruction of three-dimensional objects. However, while PCL already includes several fast and reliable methods to align two surfaces, the tools available to align multiple surfaces are relatively limited. We aim to extend the functionalities of pcl::registration by adding efficient algorithms for automatic multiview registration from unordered views. In particular, we will consider the dual quaternion algorithm for multiple view surface reconstruction from point cloud data as described in the following paper:

[1] A. Torsello, E. Rodolà, and A. Albarelli. "Multiview registration via graph diffusion of dual quaternions". Proc. IEEE Intl. Conf. on Computer Vision and Pattern Recognition (2011)

The approach allows for a completely generic topology over which the pairwise motions are diffused. It is both orders of magnitude faster than the state of the art, and more robust to extreme positional noise and outliers. The dramatic speedup of the approach allows it to be alternated with pairwise alignment resulting in a smoother energy profile, reducing the risk of getting stuck at local minima, and enabling the integration of the algorithm within a global alignment framework, namely:

[2] D. F. Huber and M. Hebert. "Fully automatic registration of multiple 3D data sets". Image and Vision Computing 21 (2003) 637-650

The student will thus port the original code of [1] to PCL and implement a basic version of the global alignment algorithm in [2]. Also, he/she will evaluate a full multiview registration pipeline built by assembling the various tools available in PCL.

Adapting 3D tools to maintain RGB information

As most non-expert users rely on the executable apps and tools, it is important that these automatically deal with the RGB color channels. This is the case most of the time, but several surface reconstruction methods do not preserve the color information. Some of them could be adapted easily (like MLS and GT3), while others would require some implementation (like Poisson reconstruction based merging of colors from multiple RGB-D images).

OpenCL Module for PCL

As GPU programming becomes more and more ubiquitous in scientific computations, we would like to port a number of commonly used PCL algorithms from various modules (e.g., filtering, registration, segmentation). This will accelerate the processing allowing for real-time performance and opening new application possibilities.

RGB-D object segmentation in clutter

Integrating cluttered scene segmentation methods into pcl::segmentation, based on: A. Richtsfeld, T. Mörwald, J. Prankl, M. Zillich and M. Vincze: Learning of Perceptual Grouping for Object Segmentation on RGB-D Data; Journal of Visual Communication and Image Representation (JVCI), Special Issue on "Visual Understanding and Applications with RGB-D Cameras", July 2013 link and Z.-C. Marton, F. Balint-Benczedi, O. M. Mozos, N. Blodow, A. Kanezaki, L. C. Goron, D. Pangercic, and M. Beetz: Part-based geometric categorization and object reconstruction in cluttered table-top scenes; Journal of Intelligent and Robotic Systems, January 2014. The projects use PCL partially already, and source code can be obtained. It would be important to separate the various steps, and make them interoperable, as it was done in the pcl::registration framework.

Sample Consensus (SaC) fitting of superquadric shapes

Currently the pcl::sac module deals with simple shape primitives like planes, cylinders, spheres, etc. but superquadric shapes offer great flexibility in approximating a wide range of watertight surfaces. However, fitting them is difficult, especially in the presence of outliers, which is why a SaC approach would hold several advantages (robustness to outliers, improved speed and a probabilistic criterion to balance quality and accuracy). The first steps of the work could be based on the following publication:

K. Duncan, S. Sarkar, R. Alqasemi, and R. Dubey; Multi-scale superquadric fitting for efficient shape and pose recovery of unknown objects; in IEEE International Conference on Robotics and Automation (ICRA), 2013.

Object Discovery in Kinfu data

This project aims at adding new functionalities to PCL for the task of environment exploration based on data processed by Kinect Fusion algorithms. Kinect Fusion algorithms have recently introduced a new approach for environment and object reconstruction, which is extremely efficient, can be carried out with low-cost sensors and can potentially yield to highly accurate reconstructions. PCL already includes modules that are able to merge together, in real-time, multiple RGB-D frames acquired from continuous acquisition within the same 3D coordinate system (e.g. KinFu). On top of that, we propose to implement a recent approach that exploits specifically this new data representation for the goal of automatic object discovery: A. Karpathy, S. Miller, L. Fei-Fei, “Object Discovery Via Shape Analysis”, ICRA 2013 Additionally, the project will also investigate possible improvements to the original technique in terms of additional - or more efficient - shape measures, as well as in terms of better reasoning about empty/unseen space.

RGB-D Flow

Optical flow has been an important technique in computer vision, both for detecting moving objects in static scenes, and for separating ego-motion from dynamic objects in moving cameras as in A. Talukder, L. Matthies, "Real-time detection of moving objects from moving vehicles using dense stereo and optical flow", IROS 2004 Recent work by Herbst et. al has proposed RGB-D flow. E. Herbstand, X. Ren, F. Dieter, "RGB-D flow: Dense 3-D motion estimation using color and depth", ICRA 2013 Adding an implementation of this approach or similar to PCL would be helpful for several applications. The registration techniques in PCL typically assume static scenes, so detecting movement not caused by ego-motion and omitting it could enable registration even in dynamic environments. Detecting independently moving objects can also be informative for detecting people moving in the environment. Background subtraction is often used for this in static scenes, but is not suitable for use when the sensor is also in motion — RGB-D flow could be used to address this.

SSD: Smooth Signed Distance Surface Reconstruction

Implement, adapt to the data structures present in PCL and document the SSD algorithm for watertight surface reconstruction from point cloud data as described in the following paper:

Fatih Calakli and Gabriel Taubin, "SSD: Smooth Signed Distance Surface Reconstruction", Computer Graphics Forum, Vol. 30, No. 7, 2011

Distributed Computing for PCL

We are looking for an expert to adapt some of our algorithms to run on a cluster. We are thinking of using technologies such as Hadoop , or any other open-source implementations of distributed computing libraries. The end result should be a cluster of machines running point cloud processing services such as large dataset registration, feature extraction, object recognition etc.

This is a rather involved project, and expertise in the field of distributed computing is necessary. We recommend to discuss with the GSoC mentors on the mailing list for a more in-depth view of our ideas.

Custom Projects

As PCL is a research-oriented community, we are more than happy to welcome students to come up with their proposals (even contribute with implementations of their own MSc/PhD work!). Most probably there will be a mentor with high interest in your field of research.

Students may suggest their own well thought out and documented ideas, which should also have project timeline for completion. These will only be accepted if one of the mentors approves to support it. The project should use a substantial, proven technology and be of general interest. Research-based projects are accepted only if they are proven to be reasonable for the duration of Google Summer of Code 2013 program.

Project Mentors

Do not miss your chance to work with one of the awesome mentors that we have lined up for this program! They all work or study at top institutions across the world and are top contributor for the PCL library.

(in alphabetical order): Please use to get in touch with the mentors, rather than using their private e-mail addresses!

For admin issues please contact Alex Ichim or Federico Tombari. Thank you!