libmegapixels/docs/overview.rst

Overview
========

Libmegapixels is a library for accessing V4L2 cameras in Linux. It's the camera
set-up code originally written for Megapixels but split off and improved as a
library to make it easier to debug and integrate.

The way you'd normally open a webcam in Linux and capture frames is roughly:

- Open a :code:`/dev/video*` file descriptor
- Run the :code:`VIDIOC_QUERYCAP` ioctl to check the device has the :code:`V4L2_CAP_VIDEO_CAPTURE` flag and check which methods of capturing frames are valid.
- Run the :code:`VIDIOC_S_FMT` and :code:`VIDIOC_G_FMT` ioctls a few times to narrow down a format to use.
- Set-up mmapped video capture on the device and run :code:`VIDIOC_STREAMON` to start the capture
- Get frames from the kernel

This works for UVC webcams you usually encounter on Linux devices since these are
relatively simple devices.

If you try to use the camera on a modern phone the whole workflow is completely different.
Instead of dealing with just a video device you also have to deal with
a :code:`/dev/media*` device for setting up hardware pipelines and then deal with
various :code:`/dev/v4l-subdev*` devices to configure all the nodes in that media pipeline.

One (or more) of the nodes in that media pipeline will be the regular old :code:`/dev/video*`
device again that will provide your application with the frames, but due to the way
the pipelines work a lot of the normal auto-configuration things applications do no longer work.

In the new pipelines the video device only deals with getting the video frames into
userspace. Due to this the video device does not actually know what formats and
resolutions are valid so using the old ioctls to query this information is useless.

One of the v4l-subdev devices will be representing the sensor in the device, this does know what
modes are available but to know that mode will work you need to make sure all the nodes in the
pipeline can run at that mode and all these nodes need to be manually configured. But in
reality it's even more complicated becaues this only describes what resolutions and modes
the drivers for these components support and it does not account for limitations in
the actual hardware like not all the MIPI lanes being connected between the sensor
and the SoC limiting the possible bandwidth, or even more basic bandwidth limitations
due to the length of PCB traces.

Using libmegapixels
-------------------

The solution for this mess in Megapixels which is now transferred to a library
is using config files that provide the pipeline information. This hardcodes a
series of known working modes and also provides the metadata for producing
high quality pictuers from the data coming from the pipeline.

The general use of libmegapixels is this:

.. code-block:: c

   #include <libmegapixels.h>


   // Create the empty device config object
   libmegapixels_devconfig *config = {0};
   libmegapixels_init(&config);

   // Find the config file path for this device
   char configpath[PATH_MAX];
   int ret = libmegapixels_find_config(configpath);

   // Load the config
   libmegapixels_load_file(config, configpath);

   // Additionally load UVC cameras with autodetection
   libmegapixels_load_uvc(config);

   printf("Found %d cameras\n", config->count);

   // Open the first camera
   libmegapixels_camera *camera = config->cameras[camera_id];
   libmegapixels_open(camera);

   printf("Camera has %d modes\n", camera->num_modes);

   // Select the first mode
   libmegapixels_mode *mode = camera->modes[mode_idx];
   struct v4l2_format format = {0};
   libmegapixels_select_mode(camera, mode, &format);

   // Now you can do regular V4L2 things on the camera with the
   // supplied FDs. video_fd for the /dev/video device.
   ioctl(camera->video_fd, VIDIOC_QUERYCAP, &cap)

Libmegapixels replaces most of the original V4L2 setup code and provides a
filled in :code:`struct v4l2_format` to get all the exact mode information.

To pick a camera and mode you'd iterate over the found cameras and then over
the mode structs to find something that matches the needs of the application.

The camera struct provides the FDs for the various devices you might need:

- The :code:`video_fd` field has the FD for :code:`/dev/video*` for actually
  getting the frames.
- The :code:`sensor_fd` field is for the :code:`/dev/v4l-subdev*` device that
  represents the sensor. This is for setting camera V4L2 controls while capturing.
- The :code:`media_fd` field is for the :code:`/dev/media*` device that has the
  pipeline for the current camera. This is mostly for the libmegapixels internals.

The config files
----------------

The libmegapixels library ships with configuration files for some devices. The
device configuration can be stored in multiple locations so it can be overridden
by packagers and end users.

/usr/share/megapixels/config/%model.conf
   This is the location where the config files from packages are stored. This is
   also where the build script from libmegapixels will place the internal config
   files here.

/etc/megapixels/config/%model.conf
   This is the place for extra and/or override config files

$cwd/config/%model.conf
   The config is loaded from the current working directory to make testing and
   debugging the code easier and to run it from the root of the git repository.

The %model argument in the path is referring to the device-tree compatible names
of the device. This can be found in :code:`/proc/device-tree/compatible`. This
stores the name of the hardware seperated by null-bytes in decreasing specificity
order. Libmegapixels will check all of them in order for every location above.