360 ° panoramic video support in Wowza Streaming Engine

2016-10-13 21:49:00

For a long time, the creators of video content have been trying various options that allow you to immerse yourself most deeply in watching videos. Increase resolution, add volume (3D), and now also panoramic shots. Due to the fact that the frame size can not be limited to the standard screen size of a TV or other playback device, it becomes possible to shoot with image capture up to 360x360 degrees. In this case, the viewer is in the center of the sphere onto which the picture is projected. He can turn his head to see that part of it that may be of interest at the moment. This allows you not only to look at the event through the eyes of the operator, but to watch exactly what you want. For example, a camera installed in a concert hall actually allows you to achieve the effect of presence and look at what is happening, as if you are in the hall at the place where the shooting equipment is installed.

This allows you to create virtual reality (VR) and place a real viewer in it. Of course, this is a very promising topic, which is increasingly in demand both for live broadcasts of sporting events, concerts, reports, mass events, and for creating videos, for example, for electronic guides.

To date, cameras with a single matrix for panoramic shooting have not been invented, so several shooting devices are used, the image from which is subsequently glued (stitching) into a single picture. As with classical shooting, the quality depends on the equipment. These are the types of lenses, the number and the correct location of the cameras, the quality of gluing, and so on.

The market for such solutions cannot be called saturated, but there are already cameras for amateur photography, as well as professional solutions that are complex prefabricated structures. Among the budget segment there are cameras LG, Samsung. They don’t know how to transfer live streams; they can only record, which, however, can be published on a service in the future.

There are more expensive cameras like ORAH 4i. The camera is more advanced in terms of parameters, and can also be used to publish a live stream to a media server using the RTMP protocol.

In addition, 360 Designs also produces rigs and ready-made assemblies based on BlackmagicCinema 4K and Studio 4K cameras. There are different types of structures containing from 3 to 42 chambers. And if we take into account that each of them is equipped with a 4K matrix, then the resulting signal will just have incredible value and degree of detail. However, as well as the price, as well as the need for high qualifications and experience for assembly, maintenance and operation.

There is one more difficulty - the signal from the cameras must be glued and transcoded into a format suitable for streaming. For this purpose, you can use special software. For example, VahanaVR. This is a Windows application that allows you to capture many sources, glue them and lay them on the surface of a sphere. Further, the material is written to a file, and also transmitted as a compressed RTMP stream to a media server.

The RTMP protocol allows you to stream to the Wowza Streaming Engine server. For any such service, this is still the same stream (the same protocol, the same type of audio and video compression, the same principles of image formation), only with a higher resolution and bitrate. That is, the concept, in comparison with the classic video format, has not changed.

Only one question remains open - how to watch such a video?

Answer - we need a special player that will allow us to see in some way more than the frames of the standard window and the viewing angle of a person allow.

The implementation options are as follows:

  • Virtual reality glasses. A person turns his head how he will be in the place where the broadcast is from, and can see everything the way he wants.
  • Players of mobile platforms. The user moves the device, and the video changes depending on these movements. In principle, the same thing as glasses, only now it's a phone or tablet.
  • Demonstration of glued video with fit into the window of a standard player. The result is an uncomfortable picture for viewing, which in reality does not allow to achieve the effect of presence.

In fact, they represent the general idea, but with the difference that the player for playback can be a software or hardware-software solution. Not every player will support working with such a stream of video data, it is necessary to provide means for changing the scene overview vertically and horizontally :. the ability to control the look with the mouse and keyboard, and, if available, through the VR helmet. On mobile platforms, simply moving the device in space is enough to change the view.

The following components must be provided:

  1. Source of video and audio signals;
  2. Tools for gluing video signals into a panoramic image in real time;
  3. An encoder capable of compressing a signal without distorting the picture;
  4. Media server for distributing the stream to subscriber devices;
  5. A special player that supports panoramic images.

Throughout the chain, the quality of the final product is provided, first of all, by the camera and methods of gluing several pictures into a single image. If you take a camera like Orah 4i, then everything is simple. They connected to the Internet and transmitted via RTMP stream to the media server. But if you want and financial opportunities you can achieve a higher quality shooting. For this, you can use almost any filming device. But there are a lot of pitfalls that do not allow to solve the problem “on the forehead”, simply by installing cameras with wide-angle lenses around the circumference. It should be remembered that they must simultaneously transmit frames from the matrix to the stitching device, it is necessary to adjust the color temperature, ensure uniformity of installation vertically and horizontally, and so on. The working design requires serious engineering. Do not forget about the sound.

After the shooting device is ready, synchronous frames arrive at the gluing device, which is a powerful workstation equipped with image and sound capture boards.

Before you finally choose what to build such a station on, you need to understand where the stream it generates will go. Wowza Media officially announced support for 360 streaming, which allows you to choose the Wowza Streaming Engine as a media server, you just need to find out where it is best to give an IP stream. The official recommendation is Vahana VR from VideoStitch. The product allows you to assemble a station with the required number of video inputs. It allows you to glue the picture by placing the camera signals on the surface of the sphere, write it to a file, and also transfer it as an RTMP Live stream to a media server. The workstation is equipped with the required number of DeckLink / AJA / Magewell boards, an Intel Corei7-5930K processor or more powerful, an NVIDIA Geforce 980 Ti GPU. At the output we get RTMP, SDI / HDMI signals, as well as the output to the Oculus Rift or HTC Vive virtual reality helmets.

The next step is to choose a compatible player, and here again, it is best to choose from officially compatible. VideoStitch offers the following solutions:

  • Krpano: support for Flash and HTML5 / WebGL, RTMP and HLS;
  • VRPlayer: an open source workstation player (OpenSource) with support for a wide range of formats and geometric projections. There is also a version for Android (but there are problems with 4K playback);
  • Sphereplay: Supports up to 16 different projection layouts in 2D and 3D. Supports playback of spherical video HD, 4K, 8K. Available for iOS and Android.

You can also use the Panframe SDK developer kit to create your own solution for a specific task. The package allows you to:

  • Create custom solutions for iOS, tvOS, Android, HTML5, Flash, OSX and Windows;
  • Provide 2D or 3D navigation with support for Oculus Rift and Microsoft Kinect;
  • Supports work with Live and VOD content;
  • Supports RTMP, HLS, DASH;
  • Works with H.264 and H.265;
  • Supports resolutions up to 8K for both desktop and mobile devices.

Do not forget about virtual reality helmets (for example, Oculus Rfit or HTC Vive), which allow you to track the position and inclination of the head, the speed of its movement and rotation. There are also simpler solutions from Samsung (Gear VR), which are glasses in which the smartphone is attached.

Apparently, the choice is great. You can take a ready-made one or create your own solution, providing full customization of all the functionality.

Due to the fact that absolutely all components have both a wide range of settings and the ability to develop their own solutions and interfaces, you can build a more complex structure where the end user not only receives the player window in which he controls the overview, but also add there various elements of infographics, metadata, etc. For example, a football match, which can be equipped with a score, a change in audio accompaniment depending on the direction of the gaze, information about the players and their position on the field, to ensure high quality (up to 8K) images, which will allow you to fully plunge into the game, owning even more complete a picture of what is happening than the fans in the stands.

If you do not take into account the methods of obtaining additional data, then the solution also does not seem complicated. It is only required to transmit video, audio and metadata synchronously.

Summing up, we can confidently say that today there are almost all components for immersing a user in virtual reality. And this is not just watching 360 0 x360 0 videos, but creating a full-fledged virtual environment, including audio, overlaying additional data (the so-called augmented reality), moving control, zooming the image and so on. Moreover, the quality of the resulting image is practically unlimited, except for the physical capabilities of the equipment. If all this is not necessary, then just buy a camera with support for streaming 360 0 (for example Orah 4i), install a media server for distributing the stream to subscriber devices - Wowza Streaming Engine and publish streams on the site. You can use a ready-made player (SpherePlay, Krpano or VRPlayer), or develop your own product according to the task based on the Panframe SDK.