This is the last post summing up our experiences from putting together the 3 Inch Canvas exhibition. This time we sum up what we learned about using videos on mobile devices.
If one looks at the second 3 Inch Canvas exhibition from the perspective of what technical aspects required the most effort, two things stand out. One is quite obvious: the work required to develop the exhibition applications. The other aspect is perhaps less obvious. It is the effort required to manage the video formats for the participating works.
The video experience created by services like YouTube or Vimeo or just the media players on PC’s and Mac’s give an illusion of effortlessness. However, the fact is that a lot of work have gone into these services and applications to hide a lot of the complexities of video processing.
Mobile phones unfortunately do not yet come with video players that are as versatile as those on desktops. And even if they did, there are still a few things that would be worth considering.
Video formats
The first thing you will face when trying to display videos on a mobile device are the limitations in terms of supported video formats. For information on video formats, see this excellent post before reading on.
The reason to bring up video formats is that not all format are supported on all mobile devices. From practical experience, Nokia devices seemed most the sensitive to the video format used. If a video played on a Nokia device, it would most likely work on Android and Windows Phone devices as well. We did not explicitly test on an iPhone.
You can use the video information for the X6 device, found here, as a guideline for creating videos that work on most mobile devices. In order to view videos at full X6 resolution (meaning 640×360, called nHD in the spec) you need to code the video using MPEG4 Simple profile. In video editing software this format is referred to by many terms, eg. just MPEG4 or MPEG4 Visual or MPEG4 Part 2.
The H.264 encoding that seems to be often automatically proposed by MPEG4 encoding software – referred to with terms such as H.264 or MPEG4 Part 10 or MPEG4 AVC – is not the same and is not supported by some Nokia devices such as the X6.
Video quality
The other aspect of video coding is the quality of the video. The video quality is decided mainly by two parameters: the video image size in pixels and the coding speed.
The video image (or display) size is a simple and straightforward concept. The more pixels used in coding the video, the better the video quality will potentially be. This is very similar to the effect of more pixels on still images.
For videos with an aspect ratio of 16:9, some of the most common pixel sizes are multiples of the fullHD video which has a pixel size of 1920 x 1080. The 16:9 formats are commonly referred to by the number of vertical pixels, hence the fullHD format can be referred to as 1080. Other common sizes are eg.
- 360 x 640 (referred to as 360)
- 480 x 854 (referred to as 480)
- 720 x 1280 (referred to as 720)
The other factor is the coding speed. This parameter indicates how many bit/second of data is used to code the video. Advanced coders tend to use an encoding referred to as variable bit rate. What this means is that the information (in this case video) is coded with a fixed quality and the average bit rate over a suitable time period is given. See this for details.
If we compare to jpeg images, the coding speed is a bit like the “file size” of jpeg images. The bigger the file (for a given pixel size), the better the image quality.
Coding for mobiles – selecting the appropriate video image size and coding speed
The video image size and the coding speed are connected. The coding speed is a measure of how much information is “consumed” to code the video. More information results in better quality until some level when increasing the coding speed no longer improves the quality.
The bigger the video image size, the more information is required to code the video at a given quality level. Thus a fullHD video (with a 1920×1080 display size) will need a higher bit rate than a small 640×360 video.
Why is this interesting? It is interesting from the perspective of how big a file or how much network bandwidth is needed to show a video. If we use a fullHD vide0, the bit rate might be 10Mbit/s. This means that a one minute video will need about 75Mbits of disk space. The smaller video can be coded at 1/10th of the bit rate, meaning about 1Mbit/s. In this case the resulting file will be just 7.5 Mbytes in size.
Where this matters the most is when a video is streamed over a network connection. Streaming means that the video is stored on one system but is being shown on an other system. The transfer of the video between the systems happens “in the background” at the same time as the video is being shown.
The fullHD video encoded at 10Mbit/s will need a 10Mbit/s network connection to be streamed. If the network bandwidth is lower, then the network will not be able to “top up” the device on which the video will be shown fast enough and the video will pause. This results in a very unpleasant viewing experience.
If we were to try to stream the video from a server to a mobile device, 10Mbit/s means that this can only be done over a wireless LAN (WLAN) connecting. The bandwidth of the cellular network is not (yet) capable of 10Mbit/s.
If we wanted to show the video on the mobile device we notice that fullHD (1980×1080) is actually unnecessarily big, mobile devices do not have display with that kind of resolution. We can make the video “smaller” by converting it to a more suitable size, let’s say 360 x 640.
Now the coding speed comes into play. If we just reduce the bitsize to 360 x 640 but do not lower the coding speed, we will end up with a video that still needs a 10Mbit/s bandwidth.
We need to also reduce the coding speed to something more appropriate, in this case 1Mbit/second. A video that is encoded at 1Mbit/second just might be able to make it over a cellular network.
We can also make the video even smaller. In this case the video will not fill the entire screen of an advanced mobile phone. We can show the video as a small “postcard” on the mobile or we can enlarge the video on the device to fill be full screen. In both cases, the end result will be a less than perfect video viewing experience.
Finding the optimal encoding speed and image size is a tricky issue where trying out different combinations is the key to success.
The 3 Inch Canvas videos
For the 3 Inch Canvas exhibition, the videos were encoded into 3 different formats.
- MPEG4 at the nonstandard size of 800 x 450. This format was used for the videos that were shown on the Windows Phone devices. The video image size was selected because the Windows Phone screen is 800 pixels wide. 800 x 450 is therefore the maximal video size for a video with and aspect ratio of 16:9 that can be shown on the device without being scaled either up or down. Experimenting with the coding speed showed that a speed on about 1.5Mbit/second yielded an optimal video quality, increasing the coding speed did not improve visibly the video quality.
- MPEG4 at the nHD size 640 x 360. This video format was used on the non-interactive works that used Nokia X6 devices. It was also the format in which the videos were submitted to Arena Interactive for the Artwork of the day service. A coding speed of about 0.9 – 1Mbit/second was found to be appropriate.
- HD or fullHD videos at 1280 x 720 or 1980 x 1080. This format was used for streaming videos from the Windows Phone devices to the big screens (see this post for more information). The video coding speed of about 5 Mbit/sec for HD and 10 Mbit/s for fullHD was found to yield good results.
The exhibition participants were requested to encode the videos themselves. The main thinking here was that the artists in this way would be able to control the quality of their videos.
There were no issues with the HD ro fullHD videos submitted by the artists. This comes as no surprise, as this is the format in which artists would create their videos to be shown on PCs.
For the smaller formats, there we quite a few issues, which resulted in the need to ask artist to recode and resend their submissions. Some of the common issues were:
- MPEG4 (or MPEG Simple profile) videos were very often encoded and sent as H.264 videos. This is in part our fault, as we did not realize up from how many terms were used for the two formats in various software packages and especially their presets.
- Inappropriate coding speed. Often the coding speed for the smaller videos was set to be very high. There were no cases of too low coding speed.
Designing for the small screen
Not directly connected to video encoding, but very much related to the overall video quality is the need to take the small physical screen size into account.
It was quite obvious that some of the videos had not been tested on small screens. This could mainly be seen in texts shown in the video such as credits at the end or texts that were part of the work itself.
When a video is being edited on a big screen, even small text is very easily readable. However, when the video is being shown on a small screen, the small text can become totally unreadable. Even if a mobile phone is not used during editing, it is still possible to test the video by shrinking the video preview window to the same size as a mobile phone screen. If the text is readable in the shrunk window, it will also be readable on the mobile screen.
An other thing to pay attention to is reading speed. If the text is quite small, users tend to take a bit longer to read it. Text that is scrolling or long text that is supposed to be read, needs to be displayed long enough so that users also on mobile devices have time to read it.
Conclusions
The main conclusion about using videos in mobile phone exhibitions is that good preparation on behalf of the organizer is required. The organizer should know up front what video formats are needed and provide detailed instructions to participants on how the participating videos should be encoded.
The organizer can of course do all the coding work (this is what we did in the first 3 Inch Canvas exhibition) but that is a lot of work and also makes the organizer ultimately responsible for the video quality.
The 3 Inch Canvas 