Efficient Realtime Dashboard with Pusher WebHooks

Today we want to talk about a code update we have made on Panda, taking advantage of the Pusher web hooks released a few days ago.

This is a fantastic feature for us as it has considerably reduced the number of api calls made to Pusher, saving us bandwidth and money.

One of the great features of Panda is it’s real time dashboard. Panda can display the new encoding jobs, their progress, and your stats instantly with no effort. Thanks to Pusher, implementing this was really easy.

At such a scale the right design for your software is to separate your app into smaller apps, small apis and workers. In our case, the encoding dashboard is completely separated from the api but they share the same Pusher app. When an event occurs on Panda (like a job progress), the api triggers a Pusher Event and the web app reflect that change immediately.

Before the release, we were facing one problem: we were sending a lot of unnecessary api calls to Pusher.

Panda’s api has no interaction with the user. It only communicates through private and public apis. This means it doesn’t know whether someone is listening to the Pusher channel. We are then pushing thousands of messages even if nobody is looking at the dashboard. The api triggers messages, the web app listens. This is not a two way messaging.

Now with Pusher Web Hooks, our api is able to know when at least one person is looking at the dashboard. As each customer uses a different private channel, that solves our problem.

Our Pusher dashboard is self-explanatory.

Now we would like to show you how we implemented it. First we need to setup an endpoint for the web hooks. Let’s make a small Sinatra app and use the Pusher::WebHook object, which comes with the new gem.

class PusherWebHooks < Sinatra::Base

    post '/pusher' do
        wh = Pusher::WebHook.new(request)
        if wh.valid?
            wh.events.each do |event|
            case event["name"]
                when "channel_occupied"
                    channel_occupied(event["channel"])
                when "channel_vacated"
                    channel_vacated(event["channel"])
                end
            end
        else
            status 401
        end

        return
    end
end

We can setup our notification url inside the Pusher dashboard and we are ready to receive web hooks.

Because splitting your code is the right thing to do, Panda’s encoders are not communicating directly from the api neither. So we somehow need to send this information to them. To do that, we will use Redis Sets as we already use Redis to distribute messages across our farm of encoders.

def channel_occupied(channel_name)
  redis.sadd("pusher:channels", channel_name)
end

def channel_vacated(channel_name)
  redis.srem("pusher:channels", channel_name)
end

Now our Panda encoders can use this information to check whether the channel we are sending the message to is occupied.

 def push_websocket(channel_name, event_name, data)
    if redis.sismember("pusher:channels", channel_name)
      Pusher[channel_name].trigger!(event_name, data)
      true
    else
      false
    end
 end

Redis commands on sets are O(1) operations which makes the design very efficient.

How simple is that? More information is available inside the Pusher docs. Now you have no excuse not to implement Pusher WebHooks. :)

Vivien Schilis

Introducing stacks for advanced profiles

Friday we realeased a feature that we called Stacks. A stack is an encoding environment including a set of encoding tools and commands. This feature will provide us a way to upgrade ffmpeg without breaking your encoding commands.

The current stack, called 'corepack-1' is still the one used by default. The new stack we are introducing is called 'corepack-2' and brings speed plus lots of new formats and codecs.

If you are using some advanced profiles, you should consider:

  • using a Preset. They are easier to maintain and we provided lots of encoding options directly using either the web dashboard or the api.
    You can find the full documentation on our website
  • upgrading to the new stack. Your current encoding command might be incompatible with the new stack but it should be fairly easy to update it.
    You can find the full documentation on our website

Here is an example on how to migrate your profile to the new stack using Ruby.

$ rails c
profile = Panda::Profile.find '12345'

profile.command 
> "ffmpeg06 -i $input_file$ -vcodec libx264 -acodec libfaac -vb $video_bitrate$k -ab $audio_bitrate$k -y $output_file$"

profile.command = "ffmpeg -i $input_file$ -threads 0 -c:v libx264 -c:a libfaac $audio_sample_rate$ $video_quality$ $audio_bitrate$ $fps$ -y $output_file$"
profile.audio_sample_rate = 44100
profile.stack = 'corepack-2'

profile.save!
> true

Instructions when upgrading to the new stack:

  • $audio_bitrate$ and $video_bitrate$ return the ffmpeg encoding option instead of the value of the bitrate.
  • Make sure you set ‘-threads 0’ in your commands. We run custom ffmpeg binaries on our ec2 instances so no need to put -threads 16, our system is setting the number of threads automatically.
  • FFmpeg version is currently 0.10
  • ffmpeg06 doesn’t exist anymore.
  • We hightly recommand to add the $audio_sample_rate$ and $fps$ variables in your command. ffmpeg can become quite slow or fail on some videos if those variables are not set (no need if you are setting them manually: -ar 44100 -r 25)
  • If you have any issues upgrading, remember that a logfile (encoding.path + ‘.log’) is beeing uploaded to your bucket when an encoding job fails.

In about a month 'corepack-2' will become the default stack for the newly created profiles. Make sure you specify the stack name if you consider running on the old stack.

POST /profiles.json, {..., 'stack': 'corepack-1'}

Stack on presets cannot be set as they run on their own private stack constantly updated.

Vivien Schilis

Interview with Panda co-founder

Last week we had the pleasure of being interviewed by the folks at VidCompare. Head over here to read the full interview where I speak about what got us to where we are and what the future holds for Panda!

Status update

It’s time we break the silence !

Since the last post, your host has arrived on the team. I am excited to bring my experience in Video, Linux, and Web development that I gained during the last years at STVS, another amazing company. But for now, let me tell you of the latest updates on the PandaStream platform.

Web interface speedup

After our ruby 1.9 migration, the average page response got divided by two, down to around 350ms. Missing, is the /cloud page that still takes around 1 second to load so that is something we are going to work on next.

Web+Heroku ♥

For the Heroku customers ; you can now access to a specialized version of our interface by logging into Heroku and choosing the Panda Stream Add-on. This can be handy if you want to follow visually the progress of your encoding and get real-time notifications. You can also access it by typing the following command in your console. Enjoy !

$ heroku addons:open pandastream --app your_app_name

Watermarking support for the API

Watermark support has been added last week and it works on all video format. I proposed that we follow the CSS absolute element convention for the image positioning and it works like that :

p = Panda::Profile.find(your_id)
p.watermark_url = "http://www.pandastream.com/images/panda_logo.png”
p.watermark_bottom = 5  # in pixels
p.watermark_right = 5   # in pixels
p.save

http://www.pandastream.com/docs/encoding_profiles#watermarking

In the process, we also added some more fields for the custom presets that might be useful for you :

$max_audio_bitrate$ : keeps the original bitrate until it goes over audio_bitrate
$max_video_bitrate$ : keeps the original bitrate until it goes over video_bitrate
$fps$  : value of the original framerate
$filters$: scale your video considering the `aspect_mode` attribute of your profile and apply watermarking options (only avaiable for ffmpeg06)

http://www.pandastream.com/docs/encoding_profiles#variables

WebM speed-up

After we updated our FFmpeg build, we saw almost a double speedup for the WebM encodings. This brings WebM up to speed with the h.264 transcoding. Next, we will publish some encoding speed results for your eyes so that you can compare our platform to our competitors.

Uploader CDN

v1.3 of the uploaded is now avaible on our CDN for your very convenience. You can link directly to the below url and we take care of the rest. In any case there is a JavaScript bug, we deploy the new version and you have nothing to do. the /1.3 path also guarantees that we won’t push backward-incompatible changes to this version.

How to use ? Just add the following script to your page and you are ready to go !

<script 
  src="http://cdn.pandastream.com/u/1.3/jquery.panda-uploader.min.js"
  type="text/javascript"></script>

For more details, check-out the documentation

A big props goes to the awesome JBundle which we use to package and distribute the uploader to the CDN.

Check it out : https://github.com/ismasan/jbundle

To keep you informed on the new platform updates, follow this blog or our twitter account

New features: custom output paths & iOS update

Today we have deployed some very interesting improvements on our platform. Let’s review them all.

Custom output filename paths (store your files in subdirectories!)

Many of you have requested a way to store video inside folders instead of having everything in the root of your bucket. We think we found a nice solution to solve this problem.

When you upload a video there is now a new optional attribute called path_format. It enables you to specify destination and name of a video and it’s encodings.

The path_format variable is a string representing the complete video path without the extension name. It can be constructed using some provided keywords.

The better way to explain this is with a few examples (the following examples use the Panda Rubygem version 1.1.0)

Panda::Video.create(:file => ..., :path_format => "my-example/:id")

This will save the output files inside the folder “my-example/” and the ID will be used as part of the filename. Here is the example of the generated files after having encoded the video.

http://yourbucket.s3.amazonaws.com/my-example/_video_id.flv
http://yourbucket.s3.amazonaws.com/my-example/_video_id_1.jpg
http://yourbucket.s3.amazonaws.com/my-example/_encoding_id.mp4
http://yourbucket.s3.amazonaws.com/my-example/_encoding_id_1.jpg
…
http://yourbucket.s3.amazonaws.com/my-example/_encoding_id_7.jpg

Panda::Video.create(:file => ..., :path_format => "my-example/:id/my_file_name")

This will save the output files inside the folder “my-example/id/” and the text my_file_name will be used as part of the filename.

http://yourbucket.s3.amazonaws.com/my-example/_video_id/my_file_name.flv
http://yourbucket.s3.amazonaws.com/my-example/_video_id/my_file_name_1.jpg
http://yourbucket.s3.amazonaws.com/my-example/_encoding_id/my_file_name.mp4
http://yourbucket.s3.amazonaws.com/my-example/_encoding_id/my_file_name_1.jpg
…
http://yourbucket.s3.amazonaws.com/my-example/_encoding_id/my_file_name_7.jpg

The API provides some other keywords.

:id         => The id of the video or the encoding.
:video_id   => The id of the video.
:original   => The original filename of the uploaded video
:date       => The date the video was uploaded. ('2009-10-09')
:profile    => The profile name used by the encoding ('original' is used when the file is the original video)
:type       => The video type, 'original' or 'encodings'
:resolution => The resolution of the video or the encoding. ('480x340')

How should I construct the path_format?

You should take care with the way you construct the path_format. It allows you to save your videos in a very simple a flexible way but it’s up to you ensure uniqueness of the filenames inside your bucket.

If you create a video with :path_format => "my-panda-video" all videos will be called my-panda-video with a different extension (my-panda-video.flv, panda-panda-video.mp4). But this should not be done, because if your original file has the same extension as your encoding profile, the encoding output will override the original video.

Some tips

The presence of :id always ensures uniqueness.

Using :video_id does not ensure uniqueness.

e.g. “folder/:video/:type/my-video” works if you don’t have two profiles generating two different encoding having the same extension name. Otherwise one is overwritten.

Using :video_id and :profile together does ensure the uniqueness if you add profile name to all your profiles.

e.g. “folder/:video_id/:profile/my-video”

All other keywords are not unique.

How do I download my encodings now?

The API returns a new variable when asking for the properties of a video or encoding called “path”

GET /encodings/12323.json
{
  "id":"1234567890"
  "extname": ".mp4"
  "path": "1234567890"
}

You can now construct the complete path this way (this example is in Ruby):

encoding = get('/encodings/12323.json')
encoding_url = encoding['path'] + encoding['extname']
screenshot_url = encoding['path'] + "_4.jpg"

If your are Using the Panda Rubygem version 1.1.0 or above, you will be able to get your encodings as before.

encoding = Panda::Encoding.find '12323'
encoding.url
encoding.screenshots[3]

New iOS (iPhone and iPad) preset

The iOS preset has been re-worked and improved to make it easier to use. There are two main changes.

  1. We will no longer upload the non-segmented .ts files in your bucket.
  2. The extname of your encodings is now correctly set to .m3u8 instead of .ts. This means that from now on the Panda Gem and other client libraries will return the correct url for this profile.

Identify profiles by name instead of ID

We have found it very convenient to be able to specify which encoding to use when sending a video. Now you can do it even more easily by specifying the profile name instead of its ID.

For example:

Panda::Video.create(:file => ..., :profiles => 'h264,my-custom-name')
video.encodings.create(:profile_name => 'ogg')

We hope you find these features useful, and look forward to showing you what else we’ve got in development shortly.

We’re a 2010 European Readers’ Choice Awards nominee

Head over to the site and vote for your favourite transcoding solution (Panda, of course!). Go on, make some Pandas happy!

A new uploader powered by HTML5 technology

File uploads have traditionally had very bad usability on the web. The standard solution was uploading files as part of a form, leaving the user to just wait until the process was done. We could offer barely any feedback of what was going on.

Several options appeared to make the process more bearable for the user. Some alternatives were client-based, such as using some Flash-powered element like SWFUpload. Other alternatives laid more on the side of the server with a pinch of Ajax. However, there was still the question of why there was no solution that avoided proprietary technology, required minimal hassle OR was free of far-fetched hacks.

Our current Panda jQuery upload plugin makes use of SWFUpload to allow seamless integration with your existing site and provide a progress bar. As keen advocates of HTML5 video, this hasn’t always been the ideal solution.

We’re on the brink of a HTML5 revolution, and with some ingenuity we’ve managed to develop a HTML5 powered Ajax uploader with progress. In a similar fashion to new HTML5 video players, there is a seamless fallback to Flash if the user’s browser doesn’t support the necessary HTML5 features.

We think the end result is a great improvement. Please head over to the Github page to download the source of the beta version. We’d love to hear your feedback!

If you’d like to learn more about the new HTML5 File API and XMLHttpRequest Level 2. Head over to the New Bamboo blog to read the in depth technical breakdown.

Beta version of new Panda gem

We’ve just released a beta version of the new gem for Panda. The gem has an improved interface for accessing the Panda API and makes it much easier to find the right encodings for embedding.

To find the url of an encoding it’s now as simple as this:

video = Panda::Video.find("1234")
mp4_encoding = video.encodings.find_by_profile_name("h264")
mp4_encoding.url
=> "http://s3.amazonaws.com/my_panda_bucket/4567.mp4"

Read the docs and download the branch from Github and let us know what you think!

Once it’s out of beta we will make this the default version (the old Panda style of using API urls still works) and also update the Rails docs.

Panda mention by AWS CTO

Yesterday we were chuffed to hear that in his talk at Gigaom’s conference Werner Vogels, Amazon’s CTO gave a shout out to us as an example of an innovative service hosted in the cloud. Checkout the full video over at Gigaom.com (our mention is around 25:30).

Panda goes public today!

Over the past few months during our private beta we have had some incredibly useful feedback from you all. We’ve all been working hard to incorporate your feedback and polish the service to perfection.

The service has been successfully processing many thousands of videos a month and several of you have even launched publicly in this time!

Today we’re pleased to announce we are opening up the beta to the public.

If you haven’t yet tried it out, there’s no better time than now. We have several easy guides, client libraries and sample application for Rails, PHP and Django: http://pandastream.com/docs

That’s not all though!

HTML5 Video

There has been a lot of talk about HTML5 video over the past few months. Browsers have been embracing support for the video tag, and although there’s isn’t yet a consensus on the codec of choice, we are now at a stage where HTML5 video can be reliably delivered with a Flash fallback for older browsers. This year we are going to be seeing a lot more in this area and we have several great new things in the pipeline.

The great news today is that right now you can use Panda to encode video to serve up to your users via HTML5, Flash and iPhone/iPad devices. We’ve written a tutorial which is available here: http://pandastream.com/docs

Python library and Django example app

We’ve also released a Python library and Django example application: http://github.com/newbamboo/panda_client_python and http://github.com/newbamboo/panda_example_django

If you’ve written a client lib or example app yourself we’d love to hear about it.

Codecs

We’ve been paying close attention to the types of video that Panda has been processing and have identified several codecs and FFMpeg options which we will be adding in and deploying next week.

Private content

Amazon has recently released support for private content on CloudFront. Many of you have asked in the past how you can restrict content to certain users. We recently added a permission option to Encoding Clouds, which controls whether the files Panda saves to your S3 bucket are publicly readable or not. If you wish to use this new CloudFront feature, you should set your bucket to private, and then follow the instructions in the Amazon Documention to setup CloudFront correctly.

… and finally, thanks!

We’d like to thank all users part of the private beta for helping shape this superb product. We’ve got lots more to get on with now and will look forward to keeping you up to date with new features and updates.

Cheers, Damien and the whole Panda team.

PS: Don’t forget to checkout Pusherapp a new hosted realtime web service we launched into beta a few weeks ago.